Evaluation Strategy

class substrafl.evaluation_strategy.EvaluationStrategy(test_data_nodes: List[substrafl.nodes.test_data_node.TestDataNode], eval_frequency: Optional[int] = None, eval_rounds: Optional[List[int]] = None)

Bases: object

Creates an iterator which returns True or False depending on the defined strategy. At least one of eval_frequency or eval_rounds must be defined. If both are defined, the union of both selected indexes will be evaluated.

Parameters
  • test_data_nodes (List[TestDataNode]) – nodes on which the model is to be tested.

  • eval_frequency (Optional[int]) – The model will be tested every eval_frequency rounds. Set to None to activate eval_rounds only. Defaults to None.

  • eval_rounds (Optional[List[int]]) – If specified, the model will be tested on the index of a round given in the rounds list. Set to None to activate eval_frequency only. Defaults to None.

Raises
  • ValueError – test_data_nodes cannot be an empty list

  • TypeError – test_data_nodes must be filled with instances of TestDataNode

  • TypeError – rounds must be a list or an int

  • ValueError – both eval_rounds and eval_frequency cannot be None at the same time

Return type

None

Example

Evaluation strategy which returns True every 2 rounds

my_evaluation_strategy = EvaluationStrategy(
        test_data_nodes = list_of_test_data_nodes,
        eval_frequency = 2,
        eval_rounds=None,
    )

every next next(my_evaluation_strategy) will return:

True
False
True
True
StopIteration Error

Example

Evaluation strategy which returns True on rounds 1 and 2

my_evaluation_strategy = EvaluationStrategy(
        test_data_nodes = list_of_test_data_nodes,
        eval_frequency = None,
        eval_rounds = [1, 2],
    )

every next next(my_evaluation_strategy) will return

True
True
False
False
False
StopIteration Error
__iter__()

Required methods for iterables.

__next__()

returns True if this round matches the defined strategy and False otherwise. required for iterators

property num_rounds

Property to get the num_rounds.

Returns

Total number of rounds.

Return type

Union[int, None]

restart_rounds()

reinitializes current round to 0 (generator will start from the beginning)

property test_data_nodes_org_ids: Set

Property to get the ids or test data nodes organizations.

Returns

set of organization ids

Return type

Set