Experiment

substrafl.experiment.execute_experiment(*, client: substra.sdk.client.Client, strategy: substrafl.strategies.strategy.Strategy, train_data_nodes: List[substrafl.nodes.train_data_node.TrainDataNode], num_rounds: int, experiment_folder: Union[str, pathlib.Path], aggregation_node: Optional[organizations.aggregation_node.AggregationNode] = None, evaluation_strategy: Optional[substrafl.evaluation_strategy.EvaluationStrategy] = None, dependencies: Optional[substrafl.dependency.Dependency] = None, clean_models: bool = True, name: Optional[str] = None, additional_metadata: Optional[Dict] = None, task_submission_batch_size: int = 500) substra.sdk.models.ComputePlan

Run a complete experiment. This will train (on the train_data_nodes) and test (on the test_data_nodes) the specified strategy n_rounds times and return the compute plan object from the Substra platform.

In SubstraFL, operations are linked to each other statically before being submitted to Substra.

The execution of:

  • the self.perform_round method from the passed strategy num_rounds times

  • the self.predict methods from the passed strategy

generate the static graph of operations.

Each element necessary for those operations (Tasks and Functions) is registered to the Substra platform thanks to the specified client.

Finally, the compute plan is sent and executed.

The experiment summary is saved in experiment_folder, with the name format {timestamp}_{compute_plan.key}.json

Parameters
  • client (substra.Client) – A substra client to interact with the Substra platform

  • strategy (Strategy) – The strategy that will be executed

  • train_data_nodes (List[TrainDataNode]) – List of the nodes where training on data occurs evaluation_strategy (EvaluationStrategy, Optional): If None performance will not be measured at all. Otherwise measuring of performance will follow the EvaluationStrategy. Defaults to None.

  • aggregation_node (Optional[AggregationNode]) – For centralized strategy, the aggregation node, where all the shared tasks occurs else None.

  • num_rounds (int) – The number of time your strategy will be executed

  • dependencies (Dependency, Optional) – Dependencies of the experiment. It must be defined from the SubstraFL Dependency class. Defaults None.

  • experiment_folder (Union[str, pathlib.Path]) – path to the folder where the experiment summary is saved.

  • clean_models (bool) – Clean the intermediary models on the Substra platform. Set it to False if you want to download or re-use intermediary models. This causes the disk space to fill quickly so should be set to True unless needed. Defaults to True.

  • name (str, Optional) – Optional name chosen by the user to identify the compute plan. If None, the compute plan name is set to the timestamp.

  • additional_metadata (dict, Optional) – Optional dictionary of metadata to be passed to the Substra WebApp.

  • task_submission_batch_size (int) – The compute plan tasks are submitted by batch. The higher the batch size, the faster the submission, a batch size that is too high makes the submission fail. Rule of thumb: batch_size = math.floor(120000 / number_of_samples_per_task)

  • evaluation_strategy (Optional[substrafl.evaluation_strategy.EvaluationStrategy]) –

Returns

The generated compute plan

Return type

ComputePlan