jina.drivers.evaluate¶
-
class
jina.drivers.evaluate.
BaseEvaluateDriver
(executor=None, method='evaluate', running_avg=False, *args, **kwargs)[source]¶ Bases:
jina.drivers.BaseExecutableDriver
The Base Driver for evaluation operations.
Warning
- When
running_avg=True
, then the running mean is returned. So far at Jina 0.8.10, there is no way to reset the running statistics. If you have a query Flow running multiple queries, you may want to make sure the running statistics is meaningful across multiple runs.
- Parameters
executor (
Optional
[str
]) – the name of the sub-executor, only necessary whenjina.executors.compound.CompoundExecutor
is usedmethod (
str
) – the function name of the executor that the driver feeds torunning_avg (
bool
) – always return running average instead of value of the current runargs – additional positional arguments which are just used for the parent initialization
kwargs – additional key value arguments which are just used for the parent initialization
Initialize a
BaseExecutableDriver
- Parameters
executor (
Optional
[str
]) – the name of the sub-executor, only necessary whenjina.executors.compound.CompoundExecutor
is usedmethod (
str
) – the function name of the executor that the driver feeds toargs – additional positional arguments which are just used for the parent initialization
kwargs – additional key value arguments which are just used for the parent initialization
-
extract
(doc)[source]¶ Extract the to-be-evaluated field from the document.
Drivers inherit from
BaseEvaluateDriver
must implement this method. This function will be invoked two times in_apply_all()
: once with actual doc, once with groundtruth doc.- Parameters
doc (
Document
) – the Document- Return type
Any
-
runtime
: Optional[ZEDRuntime]¶
- When
-
class
jina.drivers.evaluate.
FieldEvaluateDriver
(field, *args, **kwargs)[source]¶ Bases:
jina.drivers.evaluate.BaseEvaluateDriver
Evaluate on the values from certain field, the extraction is implemented with
dunder_get()
.- Parameters
field (
str
) – the field name to be extracted from the Protobuf.args – additional positional arguments which are just used for the parent initialization
kwargs – additional key value arguments which are just used for the parent initialization
Initialize a
BaseExecutableDriver
- Parameters
executor – the name of the sub-executor, only necessary when
jina.executors.compound.CompoundExecutor
is usedmethod – the function name of the executor that the driver feeds to
args – additional positional arguments which are just used for the parent initialization
kwargs – additional key value arguments which are just used for the parent initialization
-
extract
(doc)[source]¶ Extract the field from the Document.
- Parameters
doc (
Document
) – the Document- Return type
Any
- Returns
the data in the field
-
runtime
: Optional[ZEDRuntime]¶
-
class
jina.drivers.evaluate.
RankEvaluateDriver
(fields=('id'), *args, **kwargs)[source]¶ Bases:
jina.drivers.evaluate.BaseEvaluateDriver
Drivers used to pass matches from documents and groundtruths to an executor and add the evaluation value.
- Example fields:
[‘tags__id’, ‘score__value]
- Parameters
fields (
Union
[str
,Tuple
[str
]]) –the fields names to be extracted from the Protobuf. The differences with :class:FieldEvaluateDriver are:
More than one field is allowed. For instance, for NDCGComputation you may need to have both ID and Relevance information.
The fields are extracted from the matches of the Documents and the Groundtruth so it returns a sequence of values.
args – additional positional arguments which are just used for the parent initialization
kwargs – additional key value arguments which are just used for the parent initialization
Initialize a
BaseExecutableDriver
- Parameters
executor – the name of the sub-executor, only necessary when
jina.executors.compound.CompoundExecutor
is usedmethod – the function name of the executor that the driver feeds to
args – additional positional arguments which are just used for the parent initialization
kwargs – additional key value arguments which are just used for the parent initialization
-
property
single_field
¶ Get single field.
Property to guarantee compatibility when only one field is provided either as a string or as a unit length tuple.
- Returns
a list of fields
-
extract
(doc)[source]¶ Extract values of the matches from documents with fields as keys.
- Parameters
doc (
Document
) – Documents to be extracted.- Returns
a list of tuples consisting of the values from the fields.
-
runtime
: Optional[ZEDRuntime]¶
-
class
jina.drivers.evaluate.
NDArrayEvaluateDriver
(field='embedding', *args, **kwargs)[source]¶ Bases:
jina.drivers.evaluate.FieldEvaluateDriver
Drivers used to pass embedding from documents and groundtruths to an executor and add the evaluation value.
Note
- Valid fields:
[‘blob’, ‘embedding’]
Initialize a
BaseExecutableDriver
- Parameters
executor – the name of the sub-executor, only necessary when
jina.executors.compound.CompoundExecutor
is usedmethod – the function name of the executor that the driver feeds to
args – additional positional arguments which are just used for the parent initialization
kwargs – additional key value arguments which are just used for the parent initialization
-
runtime
: Optional[ZEDRuntime]¶
-
class
jina.drivers.evaluate.
TextEvaluateDriver
(field='text', *args, **kwargs)[source]¶ Bases:
jina.drivers.evaluate.FieldEvaluateDriver
Drivers used to pass a content field from documents and groundtruths to an executor and add the evaluation value.
Note
- Valid fields:
[‘id’, ‘level_name’, ‘parent_id’, ‘text’, ‘mime_type’, ‘uri’, ‘modality’]
Initialize a
BaseExecutableDriver
- Parameters
executor – the name of the sub-executor, only necessary when
jina.executors.compound.CompoundExecutor
is usedmethod – the function name of the executor that the driver feeds to
args – additional positional arguments which are just used for the parent initialization
kwargs – additional key value arguments which are just used for the parent initialization
-
runtime
: Optional[ZEDRuntime]¶
-
class
jina.drivers.evaluate.
LoadGroundTruthDriver
(is_update=True, traversal_paths=('m'), *args, **kwargs)[source]¶ Bases:
jina.drivers.search.KVSearchDriver
- Driver used to search for the document key in a KVIndex to find the corresponding groundtruth.
(This driver does not use the recursive structure of jina Documents, and will not consider the traversal_path argument. It only retrieves groundtruth taking documents at root as key) This driver’s job is to fill the request groundtruth with the corresponding groundtruth for each document if found in the corresponding KVIndexer.
Warning
The documents that are not found to have an indexed groundtruth are removed from the request so that the Evaluator only works with documents which have groundtruth.
-
runtime
: Optional[ZEDRuntime]¶