Evaluation Strategy¶

class substrafl.evaluation_strategy.EvaluationStrategy(test_data_nodes: List[substrafl.nodes.test_data_node.TestDataNode], eval_frequency: Optional[int] = None, eval_rounds: Optional[List[int]] = None)¶

Bases: object

Creates an iterator which returns True or False depending on the defined strategy. At least one of eval_frequency or eval_rounds must be defined. If both are defined, the union of both selected indexes will be evaluated.

Parameters

test_data_nodes (List[TestDataNode]) – nodes on which the model is to be tested.
eval_frequency (Optional[int]) – The model will be tested every eval_frequency rounds. Set to None to activate eval_rounds only. Defaults to None.
eval_rounds (Optional[List[int]]) – If specified, the model will be tested on the index of a round given in the rounds list. Set to None to activate eval_frequency only. Defaults to None.

Raises

ValueError – test_data_nodes cannot be an empty list
TypeError – test_data_nodes must be filled with instances of TestDataNode
TypeError – rounds must be a list or an int
ValueError – both eval_rounds and eval_frequency cannot be None at the same time

Return type

None

Example

Evaluation strategy which returns True every 2 rounds

my_evaluation_strategy = EvaluationStrategy(
        test_data_nodes = list_of_test_data_nodes,
        eval_frequency = 2,
        eval_rounds=None,
    )

every next next(my_evaluation_strategy) will return:

True
False
True
True
StopIteration Error

Example

Evaluation strategy which returns True on rounds 1 and 2

my_evaluation_strategy = EvaluationStrategy(
        test_data_nodes = list_of_test_data_nodes,
        eval_frequency = None,
        eval_rounds = [1, 2],
    )

every next next(my_evaluation_strategy) will return

True
True
False
False
False
StopIteration Error

__iter__()¶: Required methods for iterables.

__next__()¶: returns True if this round matches the defined strategy and False otherwise. required for iterators

property num_rounds¶

Property to get the num_rounds.

Returns: Total number of rounds.
Return type: Union[int, None]

restart_rounds()¶: reinitializes current round to 0 (generator will start from the beginning)

property test_data_nodes_org_ids: Set¶

Property to get the ids or test data nodes organizations.

Returns: set of organization ids
Return type: Set