jina.drivers.evaluate

class jina.drivers.evaluate.BaseEvaluateDriver(executor=None, method='evaluate', running_avg=False, *args, **kwargs)[source]

Bases: jina.drivers.BaseExecutableDriver

The Base Driver for evaluation operations.

Warning

When running_avg=True, then the running mean is returned. So far at Jina 0.8.10,

there is no way to reset the running statistics. If you have a query Flow running multiple queries, you may want to make sure the running statistics is meaningful across multiple runs.

Parameters

  • executor (Optional[str]) – the name of the sub-executor, only necessary when jina.executors.compound.CompoundExecutor is used

  • method (str) – the function name of the executor that the driver feeds to

  • running_avg (bool) – always return running average instead of value of the current run

  • args – additional positional arguments which are just used for the parent initialization

  • kwargs – additional key value arguments which are just used for the parent initialization

Initialize a BaseExecutableDriver

Parameters

  • executor (Optional[str]) – the name of the sub-executor, only necessary when jina.executors.compound.CompoundExecutor is used

  • method (str) – the function name of the executor that the driver feeds to

  • args – additional positional arguments which are just used for the parent initialization

  • kwargs – additional key value arguments which are just used for the parent initialization

extract(doc)[source]

Extract the to-be-evaluated field from the document.

Drivers inherit from BaseEvaluateDriver must implement this method. This function will be invoked two times in _apply_all(): once with actual doc, once with groundtruth doc.

Parameters

doc (Document) – the Document

Return type

Any

runtime: Optional[ZEDRuntime]
class jina.drivers.evaluate.FieldEvaluateDriver(field, *args, **kwargs)[source]

Bases: jina.drivers.evaluate.BaseEvaluateDriver

Evaluate on the values from certain field, the extraction is implemented with dunder_get().

Parameters

  • field (str) – the field name to be extracted from the Protobuf.

  • args – additional positional arguments which are just used for the parent initialization

  • kwargs – additional key value arguments which are just used for the parent initialization

Initialize a BaseExecutableDriver

Parameters

  • executor – the name of the sub-executor, only necessary when jina.executors.compound.CompoundExecutor is used

  • method – the function name of the executor that the driver feeds to

  • args – additional positional arguments which are just used for the parent initialization

  • kwargs – additional key value arguments which are just used for the parent initialization

extract(doc)[source]

Extract the field from the Document.

Parameters

doc (Document) – the Document

Return type

Any

Returns

the data in the field

runtime: Optional[ZEDRuntime]
class jina.drivers.evaluate.RankEvaluateDriver(fields=('id'), *args, **kwargs)[source]

Bases: jina.drivers.evaluate.BaseEvaluateDriver

Drivers used to pass matches from documents and groundtruths to an executor and add the evaluation value.

  • Example fields:

    [‘tags__id’, ‘score__value]

Parameters

  • fields (Union[str, Tuple[str]]) –

    the fields names to be extracted from the Protobuf. The differences with :class:FieldEvaluateDriver are:

    • More than one field is allowed. For instance, for NDCGComputation you may need to have both ID and Relevance information.

    • The fields are extracted from the matches of the Documents and the Groundtruth so it returns a sequence of values.

  • args – additional positional arguments which are just used for the parent initialization

  • kwargs – additional key value arguments which are just used for the parent initialization

Initialize a BaseExecutableDriver

Parameters

  • executor – the name of the sub-executor, only necessary when jina.executors.compound.CompoundExecutor is used

  • method – the function name of the executor that the driver feeds to

  • args – additional positional arguments which are just used for the parent initialization

  • kwargs – additional key value arguments which are just used for the parent initialization

property single_field

Get single field.

Property to guarantee compatibility when only one field is provided either as a string or as a unit length tuple.

Returns

a list of fields

extract(doc)[source]

Extract values of the matches from documents with fields as keys.

Parameters

doc (Document) – Documents to be extracted.

Returns

a list of tuples consisting of the values from the fields.

runtime: Optional[ZEDRuntime]
class jina.drivers.evaluate.NDArrayEvaluateDriver(field='embedding', *args, **kwargs)[source]

Bases: jina.drivers.evaluate.FieldEvaluateDriver

Drivers used to pass embedding from documents and groundtruths to an executor and add the evaluation value.

Note

  • Valid fields:

    [‘blob’, ‘embedding’]

Initialize a BaseExecutableDriver

Parameters

  • executor – the name of the sub-executor, only necessary when jina.executors.compound.CompoundExecutor is used

  • method – the function name of the executor that the driver feeds to

  • args – additional positional arguments which are just used for the parent initialization

  • kwargs – additional key value arguments which are just used for the parent initialization

runtime: Optional[ZEDRuntime]
class jina.drivers.evaluate.TextEvaluateDriver(field='text', *args, **kwargs)[source]

Bases: jina.drivers.evaluate.FieldEvaluateDriver

Drivers used to pass a content field from documents and groundtruths to an executor and add the evaluation value.

Note

  • Valid fields:

    [‘id’, ‘level_name’, ‘parent_id’, ‘text’, ‘mime_type’, ‘uri’, ‘modality’]

Initialize a BaseExecutableDriver

Parameters

  • executor – the name of the sub-executor, only necessary when jina.executors.compound.CompoundExecutor is used

  • method – the function name of the executor that the driver feeds to

  • args – additional positional arguments which are just used for the parent initialization

  • kwargs – additional key value arguments which are just used for the parent initialization

runtime: Optional[ZEDRuntime]
class jina.drivers.evaluate.LoadGroundTruthDriver(is_update=True, traversal_paths=('m'), *args, **kwargs)[source]

Bases: jina.drivers.search.KVSearchDriver

Driver used to search for the document key in a KVIndex to find the corresponding groundtruth.

(This driver does not use the recursive structure of jina Documents, and will not consider the traversal_path argument. It only retrieves groundtruth taking documents at root as key) This driver’s job is to fill the request groundtruth with the corresponding groundtruth for each document if found in the corresponding KVIndexer.

Warning

The documents that are not found to have an indexed groundtruth are removed from the request so that the Evaluator only works with documents which have groundtruth.

runtime: Optional[ZEDRuntime]