jina.math.evaluation module

jina.math.evaluation.r_precision(binary_relevance)[source]

R Precision after all relevant documents have been retrieved Relevance is binary (nonzero is relevant).

Parameters

binary_relevance (List[int]) – binary relevancy in rank order

Return type

float

Returns

precision

jina.math.evaluation.precision_at_k(binary_relevance, k=None)[source]

Precision @K.

Parameters
  • binary_relevance (List[int]) – binary relevancy in rank order

  • k (Optional[int]) – measured on top-k

Return type

float

Returns

precision @k

jina.math.evaluation.hit_at_k(binary_relevance, k=None)[source]

Score is percentage of first relevant item in list that occur

Parameters
  • binary_relevance (List[int]) – binary relevancy in rank order

  • k (Optional[int]) – measured on top-k

Return type

int

Returns

hit @k if hit return 1 else 0

jina.math.evaluation.average_precision(binary_relevance)[source]

Score is average precision (area under PR curve) Relevance is binary (nonzero is relevant).

Parameters

binary_relevance (List[int]) – binary relevancy in rank order

Return type

float

Returns

Average precision

jina.math.evaluation.reciprocal_rank(binary_relevance)[source]

Score is reciprocal of the rank of the first relevant item

Parameters

binary_relevance (List[int]) – binary relevancy in rank order

Return type

float

Returns

Average precision

jina.math.evaluation.recall_at_k(binary_relevance, max_rel, k=None)[source]

Score is recall after all relevant documents have been retrieved Relevance is binary (nonzero is relevant).

Parameters
  • binary_relevance (List[int]) – binary relevancy in rank order

  • k (Optional[int]) – measured on top-k

  • max_rel (int) – Maximum number of documents that can be relevant

Return type

float

Returns

Recall score

jina.math.evaluation.f1_score_at_k(binary_relevance, max_rel, k=None)[source]

Score is harmonic mean of precision and recall Relevance is binary (nonzero is relevant).

Parameters
  • binary_relevance (List[int]) – binary relevancy in rank order

  • k (Optional[int]) – measured on top-k

  • max_rel (int) – Maximum number of documents that can be relevant

Return type

float

Returns

F1 score @ k

jina.math.evaluation.dcg_at_k(relevance, method=0, k=None)[source]

Score is discounted cumulative gain (dcg) Relevance is positive real values. Can use binary as the previous methods.

Example from http://www.stanford.edu/class/cs276/handouts/EvaluationNew-handout-6-per.pdf

Parameters
  • relevance (List[float]) – Relevance scores (list or numpy) in rank order (first element is the first item)

  • k (Optional[int]) – measured on top-k

  • method (int) – If 0 then weights are [1.0, 1.0, 0.6309, 0.5, 0.4307, …] If 1 then weights are [1.0, 0.6309, 0.5, 0.4307, …]

Returns

Discounted cumulative gain

jina.math.evaluation.ndcg_at_k(relevance, method=0, k=None)[source]

Score is normalized discounted cumulative gain (ndcg) Relevance is positive real values. Can use binary as the previous methods.

Example from http://www.stanford.edu/class/cs276/handouts/EvaluationNew-handout-6-per.pdf

Parameters
  • relevance (List[float]) – Relevance scores (list or numpy) in rank order (first element is the first item)

  • k (Optional[int]) – measured on top-k

  • method (int) – If 0 then weights are [1.0, 1.0, 0.6309, 0.5, 0.4307, …] If 1 then weights are [1.0, 0.6309, 0.5, 0.4307, …]

Returns

Normalized discounted cumulative gain