jina.drivers.evaluate¶
-
class
jina.drivers.evaluate.
BaseEvaluateDriver
(executor=None, method='evaluate', running_avg=False, *args, **kwargs)[source]¶ Bases:
jina.drivers.BaseExecutableDriver
- Parameters
executor (
Optional
[str
]) – the name of the sub-executor, only necessary whenjina.executors.compound.CompoundExecutor
is usedmethod (
str
) – the function name of the executor that the driver feeds torunning_avg (
bool
) – always return running average instead of value of the current runargs –
kwargs –
Warning
- When
running_avg=True
, then the running mean is returned. So far at Jina 0.8.10, there is no way to reset the running statistics. If you have a query Flow running multiple queries, you may want to make sure the running statistics is meaningful across multiple runs.
-
extract
(doc)[source]¶ Extracting the to-be-evaluated field from the document. Drivers inherit from
BaseEvaluateDriver
must implement this method.This function will be invoked two times in
_apply_all()
: once with actual doc, once with groundtruth doc.- Return type
Any
-
class
jina.drivers.evaluate.
FieldEvaluateDriver
(field, *args, **kwargs)[source]¶ Bases:
jina.drivers.evaluate.BaseEvaluateDriver
Evaluate on the values from certain field, the extraction is implemented with
dunder_get()
- Parameters
field (
str
) – the field name to be extracted from the Protobufargs –
kwargs –
-
extract
(doc)[source]¶ Extracting the to-be-evaluated field from the document. Drivers inherit from
BaseEvaluateDriver
must implement this method.This function will be invoked two times in
_apply_all()
: once with actual doc, once with groundtruth doc.- Return type
Any
-
class
jina.drivers.evaluate.
RankEvaluateDriver
(field='tags__id', *args, **kwargs)[source]¶ Bases:
jina.drivers.evaluate.FieldEvaluateDriver
Drivers used to pass matches from documents and groundtruths to an executor and add the evaluation value
Example fields:
[‘tags__id’, ‘id’, ‘score__value]
- Parameters
args –
kwargs –
-
extract
(doc)[source]¶ Extracting the to-be-evaluated field from the document. Drivers inherit from
BaseEvaluateDriver
must implement this method.This function will be invoked two times in
_apply_all()
: once with actual doc, once with groundtruth doc.
-
class
jina.drivers.evaluate.
NDArrayEvaluateDriver
(field='embedding', *args, **kwargs)[source]¶ Bases:
jina.drivers.evaluate.FieldEvaluateDriver
Drivers used to pass embedding from documents and groundtruths to an executor and add the evaluation value
- Valid fields:
[‘blob’, ‘embedding’]
- Parameters
field (
str
) – the field name to be extracted from the Protobufargs –
kwargs –
-
class
jina.drivers.evaluate.
TextEvaluateDriver
(field='text', *args, **kwargs)[source]¶ Bases:
jina.drivers.evaluate.FieldEvaluateDriver
Drivers used to pass a content field from documents and groundtruths to an executor and add the evaluation value
- Valid fields:
- [‘id’,
‘level_name’, ‘parent_id’, ‘text’, ‘mime_type’, ‘uri’, ‘modality’]
- Parameters
field (
str
) – the field name to be extracted from the Protobufargs –
kwargs –
-
class
jina.drivers.evaluate.
LoadGroundTruthDriver
(is_merge=True, *args, **kwargs)[source]¶ Bases:
jina.drivers.search.KVSearchDriver
- Driver used to search for the document key in a KVIndex to find the corresponding groundtruth.
(This driver does not use the recursive structure of jina Documents, and will not consider the traversal_path argument. It only retrieves groundtruth taking documents at root as key) This driver’s job is to fill the request groundtruth with the corresponding groundtruth for each document if found in the corresponding KVIndexer.
Warning
The documents that are not found to have an indexed groundtruth are removed from the request so that the Evaluator only works with documents which have groundtruth.
- Parameters
is_merge (
bool
) – when set to true the retrieved docs are merged into current message usingMergeFrom()
, otherwise, it overrides the current message usingCopyFrom()