jina.executors.evaluators

class jina.executors.evaluators.BaseEvaluator(*args, **kwargs)[source]

Bases: jina.executors.BaseExecutor

A BaseEvaluator is used to evaluate different messages coming from any kind of executor

metric = ''

Get the name of the evaluation metric

post_init()[source]

Initialize running stats.

evaluate(actual, desired, *args, **kwargs)[source]

Evaluates difference between param:actual and param:desired, needs to be implemented in subclass.

Return type

float

property mean

Get the running mean.

Return type

float

property std

Get the running standard variance.

Return type

float

property variance

Get the running variance.

Return type

float

class jina.executors.evaluators.FileBasedEvaluator(routes=None, resolve_all=True, *args, **kwargs)[source]

Bases: jina.executors.compound.CompoundExecutor

A Frequently used pattern for combining A BinaryPbIndexer and BaseEvaluator.

It will be equipped with predefined requests.on behaviors:

  • At evaluation time(query or index)
      1. Checks for the incoming document, gets its value from the BinaryPbIndexer and fills the `groundtruth of the request

      1. Filter the documents that do not have a corresponding groundtruth

      1. The BaseEvaluator works as if the groundtruth had been provided by the client as it comes in the request.

Warning

The documents that are not found to have an indexed groundtruth are removed from the request so that the Evaluator only works with documents which have groundtruth.

One can use the FileBasedEvaluator via

!FileBasedEvaluator
components:
  - !BinaryPbIndexer
    with:
      index_filename: ground_truth.gz
    metas:
      name: groundtruth_index  # a customized name
      workspace: ${{TEST_WORKDIR}}
  - !BaseEvaluator

Without defining any requests.on logic. When load from this YAML, it will be auto equipped with

on:
  [SearchRequest, IndexRequest]:
    - !LoadGroundTruthDriver
      with:
        executor: BaseKVIndexer
    - !BaseEvaluateDriver
      with:
        executor: BaseEvaluator
  ControlRequest:
    - !ControlReqDriver {}