jina.executors.compound

class jina.executors.compound.CompoundExecutor(routes=None, resolve_all=True, *args, **kwargs)[source]

Bases: jina.executors.BaseExecutor

A CompoundExecutor is a set of multiple executors. The most common usage is chaining a pipeline of executors, where the input of the current is the output of the former.

A common use case of CompoundExecutor is to glue multiple BaseExecutor together, instead of breaking them into different Pods.

Example 1: a compound Chunk Indexer that does vector indexing and key-value index

!CompoundExecutor
components:
  - !NumpyIndexer
    with:
      index_filename: vec.gz
    metas:
      name: vecidx_exec  # a customized name
      workspace: $TEST_WORKDIR
  - !BinaryPbIndexer
    with:
      index_filename: chunk.gz
    metas:
      name: chunkidx_exec
      workspace: $TEST_WORKDIR
metas:
  name: chunk_compound_indexer
  workspace: $TEST_WORKDIR
requests:
  on:
    SearchRequest:
      - !VectorSearchDriver
        with:
          executor: vecidx_exec
    IndexRequest:
      - !VectorIndexDriver
        with:
          executor: vecidx_exec
    ControlRequest:
      - !ControlReqDriver {}

**Example 2: a compound crafter that first craft the doc and then segment **

!CompoundExecutor
components:
  - !GifNameRawSplit
    metas:
      name: name_split  # a customized name
      workspace: $TEST_WORKDIR
  - !GifPreprocessor
    with:
      every_k_frame: 2
      from_buffer: true
    metas:
      name: gif2chunk_preprocessor  # a customized name
metas:
  name: compound_crafter
  workspace: $TEST_WORKDIR
  py_modules: gif2chunk.py
requests:
  on:
    IndexRequest:
      - !DocCraftDriver
        with:
          executor: name_split
      - !SegmentDriver
        with:
          executor: gif2chunk_preprocessor
    ControlRequest:
      - !ControlReqDriver {}

One can access the component of a CompoundExecutor via index, e.g.

c = BaseExecutor.load_config('compound-example.yaml')
assertTrue(c[0] == c['dummyA-1ef90ea8'])
c[0].add(obj)

Note

All components workspace and replica_workspace are overrided by their CompoundExecutor counterparts.

Warning

When sub-component is external, py_modules must be given at root level metas not at the sub-level.

Create a new CompoundExecutor object

Parameters
  • routes (Optional[Dict[str, Dict]]) –

    a map of function routes. The key is the function name, the value is a tuple of two pieces, where the first element is the name of the referred component (metas.name) and the second element is the name of the referred function.

    See also

    add_route()

  • resolve_all (bool) – universally add *_all() to all functions that have the identical name

Example:

We have two dummy executors as follows:

class dummyA(BaseExecutor):
    def say(self):
        return 'a'

    def sayA(self):
        print('A: im A')


class dummyB(BaseExecutor):
    def say(self):
        return 'b'

    def sayB(self):
        print('B: im B')

and we create a CompoundExecutor consisting of these two via

da, db = dummyA(), dummyB()
ce = CompoundExecutor()
ce.components = lambda: [da, db]

Now the new executor ce have two new methods, i.e ce.sayA() and ce.sayB(). They point to the original dummyA.sayA() and dummyB.sayB() respectively. One can say ce has inherited these two methods.

The interesting part is say(), as this function name is shared between dummyA and dummyB. It requires some resolution. When resolve_all=True, then a new function say_all() is add to ce. ce.say_all works as if you call dummyA.sayA() and dummyB.sayB() in a row. This makes sense in some cases such as training, saving. In other cases, it may require a more sophisticated resolution, where one can use add_route() to achieve that. For example,

ce.add_route('say', db.name, 'say')
assert b.say() == 'b'

Such resolution is what we call routes here, and it can be specified in advance with the arguments routes in __init__(), or using YAML.

!CompoundExecutor
components: ...
with:
  resolve_all: true
  routes:
    say:
    - dummyB-e3acc910
    - say
property is_updated

Return True if any components is updated

Return type

bool

property is_trained

Return True only if all components are trained (i.e. is_trained=True)

Return type

bool

save(filename=None)[source]

Serialize this compound executor along with all components in it to binary files

Parameters

filename (Optional[str]) – file path of the serialized file, if not given then save_abspath is used

Return type

bool

Returns

successfully dumped or not

It uses pickle for dumping.

property components

Return all component executors as a list. The list follows the order as defined in the YAML config or the pre-given order when calling the setter.

Return type

List[~AnyExecutor]

add_route(fn_name, comp_name, comp_fn_name, is_stored=False)[source]

Create a new function for this executor which refers to the component’s function

This will create a new function fn_name() which actually refers to components[comp_name].comp_fn_name. It is useful when two components have a function with duplicated name and one wants to resolve this duplication.

Parameters
  • fn_name (str) – the name of the new function

  • comp_name (str) – the name of the referred component, defined in metas.name

  • comp_fn_name (str) – the name of the referred function of comp_name

  • is_stored (bool) – if True then this change will be stored in the config and affects future save() and save_config()

Return type

None

close()[source]

Close all components and release the resources

Return type

None

classmethod to_yaml(representer, data)[source]

Required by ruamel.yaml.constructor

classmethod from_yaml(constructor, node)[source]

Required by ruamel.yaml.constructor

train(*args, **kwargs)

Train this executor, need to be overrided

Return type

None