jina.executors.indexers.keyvalue

class jina.executors.indexers.keyvalue.BinaryPbIndexer(*args, **kwargs)[source]

Bases: jina.executors.indexers.BaseKVIndexer

Parameters
  • index_filename – the name of the file for storing the index, when not given metas.name is used.

  • args

  • kwargs

class WriteHandler(path, mode)[source]

Bases: object

close()[source]
flush()[source]
class ReadHandler(path)[source]

Bases: object

close()[source]
get_add_handler()[source]

Get a writable index handler when the index_abspath already exist, need to be overrided

get_create_handler()[source]

Get a writable index handler when the index_abspath does not exist, need to be overrided

get_query_handler()[source]

Get a readable index handler when the index_abspath already exist, need to be overrided

add(keys, values, *args, **kwargs)[source]
query(key)[source]

Find the protobuf chunk/doc using id

Parameters

key (int) – id

Return type

Optional[bytes]

Returns

protobuf chunk or protobuf document

class jina.executors.indexers.keyvalue.DataURIPbIndexer(*args, **kwargs)[source]

Bases: jina.executors.indexers.keyvalue.BinaryPbIndexer

Shortcut for DocPbIndexer equipped with requests.on for storing doc-level protobuf and data uri info, differ with ChunkPbIndexer only in requests.on

Parameters
  • index_filename – the name of the file for storing the index, when not given metas.name is used.

  • args

  • kwargs

class jina.executors.indexers.keyvalue.UniquePbIndexer(routes=None, resolve_all=True, *args, **kwargs)[source]

Bases: jina.executors.compound.CompoundExecutor

A frequently used pattern for combining a BaseKVIndexer and a DocIDCache

Create a new CompoundExecutor object

Parameters
  • routes (Optional[Dict[str, Dict]]) –

    a map of function routes. The key is the function name, the value is a tuple of two pieces, where the first element is the name of the referred component (metas.name) and the second element is the name of the referred function.

    See also

    add_route()

  • resolve_all (bool) – universally add *_all() to all functions that have the identical name

Example:

We have two dummy executors as follows:

class dummyA(BaseExecutor):
    def say(self):
        return 'a'

    def sayA(self):
        print('A: im A')


class dummyB(BaseExecutor):
    def say(self):
        return 'b'

    def sayB(self):
        print('B: im B')

and we create a CompoundExecutor consisting of these two via

da, db = dummyA(), dummyB()
ce = CompoundExecutor()
ce.components = lambda: [da, db]

Now the new executor ce have two new methods, i.e ce.sayA() and ce.sayB(). They point to the original dummyA.sayA() and dummyB.sayB() respectively. One can say ce has inherited these two methods.

The interesting part is say(), as this function name is shared between dummyA and dummyB. It requires some resolution. When resolve_all=True, then a new function say_all() is add to ce. ce.say_all works as if you call dummyA.sayA() and dummyB.sayB() in a row. This makes sense in some cases such as training, saving. In other cases, it may require a more sophisticated resolution, where one can use add_route() to achieve that. For example,

ce.add_route('say', db.name, 'say')
assert b.say() == 'b'

Such resolution is what we call routes here, and it can be specified in advance with the arguments routes in __init__(), or using YAML.

!CompoundExecutor
components: ...
with:
  resolve_all: true
  routes:
    say:
    - dummyB-e3acc910
    - say