jina.drivers.evaluate

class jina.drivers.evaluate.BaseEvaluateDriver(executor=None, method='evaluate', running_avg=False, *args, **kwargs)

Bases: jina.drivers.BaseExecutableDriver

Parameters

• executor (Optional[str]) – the name of the sub-executor, only necessary when jina.executors.compound.CompoundExecutor is used

• method (str) – the function name of the executor that the driver feeds to

• running_avg (bool) – always return running average instead of value of the current run

• args –

• kwargs –

Warning

When running_avg=True, then the running mean is returned. So far at Jina 0.8.10,

there is no way to reset the running statistics. If you have a query Flow running multiple queries, you may want to make sure the running statistics is meaningful across multiple runs.

extract(doc)

Extracting the to-be-evaluated field from the document. Drivers inherit from BaseEvaluateDriver must implement this method.

This function will be invoked two times in _apply_all(): once with actual doc, once with groundtruth doc.

Return type

Any

class jina.drivers.evaluate.FieldEvaluateDriver(field, *args, **kwargs)

Bases: jina.drivers.evaluate.BaseEvaluateDriver

Evaluate on the values from certain field, the extraction is implemented with dunder_get()

Parameters

• field (str) – the field name to be extracted from the Protobuf

• args –

• kwargs –

extract(doc)

Extracting the to-be-evaluated field from the document. Drivers inherit from BaseEvaluateDriver must implement this method.

This function will be invoked two times in _apply_all(): once with actual doc, once with groundtruth doc.

Return type

Any

class jina.drivers.evaluate.RankEvaluateDriver(field='tags__id', *args, **kwargs)

Bases: jina.drivers.evaluate.FieldEvaluateDriver

Drivers used to pass matches from documents and groundtruths to an executor and add the evaluation value

• Example fields:

[‘tags__id’, ‘id’, ‘score__value]

Parameters

• args –

• kwargs –

extract(doc)

Extracting the to-be-evaluated field from the document. Drivers inherit from BaseEvaluateDriver must implement this method.

This function will be invoked two times in _apply_all(): once with actual doc, once with groundtruth doc.

class jina.drivers.evaluate.NDArrayEvaluateDriver(field='embedding', *args, **kwargs)

Bases: jina.drivers.evaluate.FieldEvaluateDriver

Drivers used to pass embedding from documents and groundtruths to an executor and add the evaluation value

• Valid fields:

  [‘blob’, ‘embedding’]

Parameters

• field (str) – the field name to be extracted from the Protobuf

• args –

• kwargs –

class jina.drivers.evaluate.TextEvaluateDriver(field='text', *args, **kwargs)

Bases: jina.drivers.evaluate.FieldEvaluateDriver

Drivers used to pass a content field from documents and groundtruths to an executor and add the evaluation value

• Valid fields:

  [‘id’,

  ‘level_name’, ‘parent_id’, ‘text’, ‘mime_type’, ‘uri’, ‘modality’]

Parameters

• field (str) – the field name to be extracted from the Protobuf

• args –

• kwargs –

class jina.drivers.evaluate.LoadGroundTruthDriver(is_merge=True, *args, **kwargs)

Bases: jina.drivers.search.KVSearchDriver

Driver used to search for the document key in a KVIndex to find the corresponding groundtruth.

(This driver does not use the recursive structure of jina Documents, and will not consider the traversal_path argument. It only retrieves groundtruth taking documents at root as key) This driver’s job is to fill the request groundtruth with the corresponding groundtruth for each document if found in the corresponding KVIndexer.

Warning

The documents that are not found to have an indexed groundtruth are removed from the request so that the Evaluator only works with documents which have groundtruth.

Parameters

is_merge (bool) – when set to true the retrieved docs are merged into current message using MergeFrom(), otherwise, it overrides the current message using CopyFrom()