jina.executors

class jina.executors.ExecutorType[source]

Bases: type

static register_class(cls)[source]
class jina.executors.BaseExecutor(*args, **kwargs)[source]

Bases: object

The base class of the executor, can be used to build encoder, indexer, etc.

Any executor inherited from BaseExecutor always has the meta defined in jina.executors.metas.defaults.

All arguments in the __init__() can be specified with a with map in the YAML config. Example:

class MyAwesomeExecutor:
    def __init__(awesomeness = 5):
        pass

is equal to

!MyAwesomeExecutor
with:
    awesomeness: 5

To use an executor in a jina.peapods.pea.BasePea or jina.peapods.pod.BasePod, a proper jina.drivers.Driver is required. This is because the executor is NOT protobuf-aware and has no access to the key-values in the protobuf message.

Different executor may require different Driver with proper jina.drivers.handlers, jina.drivers.hooks installed.

See also

Methods of the BaseExecutor can be decorated via jina.executors.decorators.

See also

Meta fields jina.executors.metas.defaults.

store_args_kwargs = False

set this to True to save args (in a list) and kwargs (in a map) in YAML config

post_init()[source]

Initialize class attributes/members that can/should not be (de)serialized in standard way.

Examples:

  • deep learning models

  • index files

  • numpy arrays

Warning

All class members created here will NOT be serialized when calling save(). Therefore if you want to store them, please override the __getstate__().

classmethod pre_init()[source]

This function is called before the object initiating (i.e. __call__())

Packages and environment variables can be set and load here.

property save_abspath

Get the file path of the binary serialized object

The file name ends with .bin.

Return type

str

property config_abspath

Get the file path of the YAML config

The file name ends with .yml.

Return type

str

property current_workspace

Get the path of the current workspace.

Return type

str

Returns

if separated_workspace is set to False then metas.workspace is returned, otherwise the metas.replica_workspace is returned

get_file_from_workspace(name)[source]

Get a usable file path under the current workspace

Parameters

name (str) – the name of the file

:return depending on metas.separated_workspace the file could be located in metas.workspace or metas.replica_workspace

Return type

str

train(*args, **kwargs)[source]

Train this executor, need to be overrided

touch()[source]

Touch the executor and change is_updated to True so that one can call save().

save(filename=None)[source]

Persist data of this executor to the workspace (or replica_workspace). The data could be a file or collection of files produced/used during an executor run.

These are some of the common data that you might want to persist:

  • binary dump/pickle of the executor

  • the indexed files

  • (pre)trained models

Warning

All class members created here will NOT be serialized when calling save(). Therefore if you want to store them, please implement the __getstate__().

It uses pickle for dumping. For members/attributes that are not valid or not efficient for pickle, you need to implement their own persistence strategy in the __getstate__().

Parameters

filename (Optional[str]) – file path of the serialized file, if not given then save_abspath is used

Return type

bool

Returns

successfully persisted or not

save_config(filename=None)[source]

Serialize the object to a yaml file

Parameters

filename (Optional[str]) – file path of the yaml file, if not given then config_abspath is used

Return type

bool

Returns

successfully dumped or not

classmethod load_config(filename, separated_workspace=False, replica_id=0)[source]

Build an executor from a YAML file.

Parameters

  • filename (Union[str, TextIO]) – the file path of the YAML file or a TextIO stream to be loaded from

  • separated_workspace (bool) – the dump and data files associated to this executor will be stored separately for each replica, which will be indexed by the replica_id

  • replica_id (int) – the id of the storage of this replica, only effective when separated_workspace=True

Return type

~AnyExecutor

Returns

an executor object

static load(filename=None)[source]

Build an executor from a binary file

Parameters

filename (Optional[str]) – the file path of the binary serialized file

Return type

~AnyExecutor

Returns

an executor object

It uses pickle for loading.

close()[source]

Release the resources as executor is destroyed, need to be overrided

classmethod to_yaml(representer, data)[source]

Required by ruamel.yaml.constructor

classmethod from_yaml(constructor, node)[source]

Required by ruamel.yaml.constructor

add_driver(driver, req_type)[source]

Add a driver to this executor.

Warning

This has to be used before .attach(pea=self) to be effective

Parameters

  • driver (BaseDriver) – the driver to add

  • req_type (str) – the request type to handle by this driver

Returns

attach(*args, **kwargs)[source]

Attach this executor to a jina.peapods.pea.BasePea.

This is called inside the initializing of a jina.peapods.pea.BasePea.