jina.executors.evaluators¶

jina.executors.evaluators.running_stats

class jina.executors.evaluators.BaseEvaluator(*args, **kwargs)[source]¶

Bases: jina.executors.BaseExecutor

A BaseEvaluator is used to evaluate different messages coming from any kind of executor

post_init()[source]¶

Initialize class attributes/members that can/should not be (de)serialized in standard way.

Examples:

deep learning models

index files

numpy arrays

Warning

All class members created here will NOT be serialized when calling save(). Therefore if you want to store them, please override the __getstate__().

property metric¶

Get the name of the evaluation metric

Return type: str

evaluate(actual, desired, *args, **kwargs)[source]¶

Return type: float

property mean¶

Return type: float

property std¶

Return type: float

property variance¶

Return type: float

class jina.executors.evaluators.FileBasedEvaluator(routes=None, resolve_all=True, *args, **kwargs)[source]¶

Bases: jina.executors.compound.CompoundExecutor

A Frequently used pattern for combining A BinaryPbIndexer and BaseEvaluator.

It will be equipped with predefined requests.on behaviors:

At evaluation time(query or index)

Checks for the incoming document, gets its value from the BinaryPbIndexer and fills the `groundtruth of the request

Filter the documents that do not have a corresponding groundtruth

The BaseEvaluator works as if the groundtruth had been provided by the client as it comes in the request.

Warning

The documents that are not found to have an indexed groundtruth are removed from the request so that the Evaluator only works with documents which have groundtruth.

One can use the FileBasedEvaluator via

!FileBasedEvaluator
components:
  - !BinaryPbIndexer
    with:
      index_filename: ground_truth.gz
    metas:
      name: groundtruth_index  # a customized name
      workspace: $TEST_WORKDIR
  - !BaseEvaluator

Without defining any requests.on logic. When load from this YAML, it will be auto equipped with

on:
  [SearchRequest, IndexRequest]:
    - !LoadGroundTruthDriver
      with:
        executor: BaseKVIndexer
    - !BaseEvaluateDriver
      with:
        executor: BaseEvaluator
  ControlRequest:
    - !ControlReqDriver {}

Create a new CompoundExecutor object

Parameters

routes (Optional[Dict[str, Dict]]) –
a map of function routes. The key is the function name, the value is a tuple of two pieces, where the first element is the name of the referred component (metas.name) and the second element is the name of the referred function.

See also

add_route()
resolve_all (bool) – universally add *_all() to all functions that have the identical name

Example:

We have two dummy executors as follows:

class dummyA(BaseExecutor):
    def say(self):
        return 'a'

    def sayA(self):
        print('A: im A')


class dummyB(BaseExecutor):
    def say(self):
        return 'b'

    def sayB(self):
        print('B: im B')

and we create a CompoundExecutor consisting of these two via

da, db = dummyA(), dummyB()
ce = CompoundExecutor()
ce.components = lambda: [da, db]

Now the new executor ce have two new methods, i.e ce.sayA() and ce.sayB(). They point to the original dummyA.sayA() and dummyB.sayB() respectively. One can say ce has inherited these two methods.

The interesting part is say(), as this function name is shared between dummyA and dummyB. It requires some resolution. When resolve_all=True, then a new function say_all() is add to ce. ce.say_all works as if you call dummyA.sayA() and dummyB.sayB() in a row. This makes sense in some cases such as training, saving. In other cases, it may require a more sophisticated resolution, where one can use add_route() to achieve that. For example,

ce.add_route('say', db.name, 'say')
assert b.say() == 'b'

Such resolution is what we call routes here, and it can be specified in advance with the arguments routes in __init__(), or using YAML.

!CompoundExecutor
components: ...
with:
  resolve_all: true
  routes:
    say:
    - dummyB-e3acc910
    - say