ccobra.benchmark.comparators

CCOBRA response comparator functionality.

Classes

class ccobra.benchmark.comparators.AbsDiffComparator[source]

Absolute differences comparator.

compare(prediction, target, response_type, choices)[source]

Compares two response numbers based on their absolute difference.

Parameters

  • prediction (tuple) – Tuple containing number A.

  • target (tuple) – Tuple containing number B.

  • response_type (string) – The response type of the prediction and target.

  • choices (list(object)) – The choice options that were available for this comparison.

Returns

Absolute difference between the differences.

Return type

float

get_name()[source]

Returns the name of the comparator.

Returns

Comparator name.

Return type

string

class ccobra.benchmark.comparators.EqualityComparator[source]

Equality comparator. Checks if both responses are equal.

compare(prediction, target, response_type, choices)[source]

Compares two response objects based on equality. When using the multiple-choice response type, the predictions and targets are interpreted as mask-vectors for the choices. In this case, the score corresponds to the overlap of the vectors.

Parameters

  • prediction (tuple) – Response tuple A for comparison.

  • target (tuple) – Response tuple B for comparison.

  • response_type (string) – The response type of the prediction and target.

  • choices (list(object)) – The choice options that were available for this comparison.

Returns

True if both objects are equal, false otherwise.

Return type

bool

get_name()[source]

Returns the name of the comparator.

Returns

Comparator name.

Return type

string

class ccobra.benchmark.comparators.NVCComparator[source]

NVC response comparator. Performs the evaluation based on NVC and non-NVC classes.

compare(prediction, target, response_type, choices)[source]

Compares two response objects based on their NVCness. Only returns true if both responses are in agreement with either responding NVC or not NVC.

Parameters

  • prediction (tuple) – Response tuple A for comparison.

  • target (tuple) – Response tuple B for comparison.

  • response_type (string) – The response type of the prediction and target.

  • choices (list(object)) – The choice options that were available for this comparison.

Returns

True only if both objects agree on whether the response is NVC or not.

Return type

bool

get_name()[source]

Returns the name of the comparator.

Returns

Comparator name.

Return type

string

class ccobra.benchmark.comparators.SquaredDiffComparator[source]

Squared differences comparator.

compare(prediction, target, response_type, choices)[source]

Compares two response numbers based on their squared difference.

Parameters

  • prediction (tuple) – Tuple containing number A.

  • target (tuple) – Tuple containing number B.

  • response_type (string) – The response type of the prediction and target.

  • choices (list(object)) – The choice options that were available for this comparison.

Returns

Squared difference between the differences.

Return type

float

get_name()[source]

Returns the name of the comparator.

Returns

Comparator name.

Return type

string