jina.executors.evaluators¶
-
class
jina.executors.evaluators.
BaseEvaluator
(*args, **kwargs)[source]¶ Bases:
jina.executors.BaseExecutor
A
BaseEvaluator
is used to evaluate different messages coming from any kind of executorConstructor.
-
metric
= ''¶ Get the name of the evaluation metric
-
evaluate
(actual, desired, *args, **kwargs)[source]¶ Evaluates difference between param:actual and param:desired, needs to be implemented in subclass.
- Return type
float
-
property
mean
¶ Get the running mean.
- Return type
float
-
property
std
¶ Get the running standard variance.
- Return type
float
-
property
variance
¶ Get the running variance.
- Return type
float
-
-
class
jina.executors.evaluators.
FileBasedEvaluator
(routes=None, resolve_all=True, *args, **kwargs)[source]¶ Bases:
jina.executors.compound.CompoundExecutor
- A Frequently used pattern for combining A
BinaryPbIndexer
andBaseEvaluator
. It will be equipped with predefined
requests.on
behaviors:- At evaluation time(query or index)
Checks for the incoming document, gets its value from the BinaryPbIndexer and fills the `groundtruth of the request
Filter the documents that do not have a corresponding groundtruth
The BaseEvaluator works as if the groundtruth had been provided by the client as it comes in the request.
Warning
The documents that are not found to have an indexed groundtruth are removed from the request so that the Evaluator only works with documents which have groundtruth.
One can use the
FileBasedEvaluator
via!FileBasedEvaluator components: - !BinaryPbIndexer with: index_filename: ground_truth.gz metas: name: groundtruth_index # a customized name workspace: ${{TEST_WORKDIR}} - !BaseEvaluator
Without defining any
requests.on
logic. When load from this YAML, it will be auto equipped withon: [SearchRequest, IndexRequest]: - !LoadGroundTruthDriver with: executor: BaseKVIndexer - !BaseEvaluateDriver with: executor: BaseEvaluator ControlRequest: - !ControlReqDriver {}
Create a new
CompoundExecutor
object- Parameters
routes (
Optional
[Dict
[str
,Dict
]]) –a map of function routes. The key is the function name, the value is a tuple of two pieces, where the first element is the name of the referred component (
metas.name
) and the second element is the name of the referred function.See also
add_route()
resolve_all (
bool
) – universally add*_all()
to all functions that have the identical name
Example:
We have two dummy executors as follows:
class dummyA(BaseExecutor): def say(self): return 'a' def sayA(self): print('A: im A') class dummyB(BaseExecutor): def say(self): return 'b' def sayB(self): print('B: im B')
and we create a
CompoundExecutor
consisting of these two viada, db = dummyA(), dummyB() ce = CompoundExecutor() ce.components = lambda: [da, db]
Now the new executor
ce
have two new methods, i.ece.sayA()
andce.sayB()
. They point to the originaldummyA.sayA()
anddummyB.sayB()
respectively. One can sayce
has inherited these two methods.The interesting part is
say()
, as this function name is shared betweendummyA
anddummyB
. It requires some resolution. When resolve_all=True, then a new functionsay_all()
is add toce
.ce.say_all
works as if you calldummyA.sayA()
anddummyB.sayB()
in a row. This makes sense in some cases such as training, saving. In other cases, it may require a more sophisticated resolution, where one can useadd_route()
to achieve that. For example,ce.add_route('say', db.name, 'say') assert b.say() == 'b'
Such resolution is what we call routes here, and it can be specified in advance with the arguments
routes
in__init__()
, or using YAML.!CompoundExecutor components: ... with: resolve_all: true routes: say: - dummyB-e3acc910 - say
Warning
When setting inner executors in components the workspace configuration will not be used and will be overriden by a workspace extracted considering the name of the CompoundExecutor, the name of each internal Component and the pea_id
- A Frequently used pattern for combining A