Remote

This is an internal SubstraFL module, the user should not use any functions here directly, apart from the remote_data and remote decorators. This modules defines how the user code is wrapped, transformed and registered as Substra algorithms.

Decorator

Decorators to wrap functions so that they are executed on the remote organizations.

substrafl.remote.decorators.remote(method: Callable)

Decorator for a remote function. With this decorator, when the function is called, it is not executed but it returns a RemoteOperation object containing all the informations needed to execute it later (see substrafl.remote.operations.RemoteOperation).

• The decorated function definition should have at least a shared_state argument

• If the decorated function is called without a _skip=True argument, the arguments required are the ones in remote_method_inner

• If the decorated function is called with a _skip=True argument, it should have the arguments of its original definition

• The decorated function should be within a class

• The __init__ of the class must be

  def __init__(self, *args, **kwargs):
      self.args = args
      self.kwargs = kwargs
  

• self.args and self.kwargs will be given to the init, any other init argument is ignored (not saved in the RemoteStruct)

Parameters:

method (Callable) – Method to wrap so that it is executed on the remote server

substrafl.remote.decorators.remote_data(method: Callable)

Decorator for a remote function containing a data_samples argument (e.g the Algo.train function) With this decorator, when the function is called, it is not executed but it returns a RemoteDataOperation object containing all the informations needed to execute it later (see substrafl.remote.operations.RemoteDataOperation).

• The decorated function definition should have at least a shared_state argument

• If the decorated function is called without a _skip=True argument, the arguments required are the ones in remote_method_inner, and it should have at least a data_samples argument

• If the decorated function is called with a _skip=True argument, it should have the arguments of its original definition

• The decorated function should be within a class

• The __init__ of the class must be

  def __init__(self, *args, **kwargs):
      self.args = args
      self.kwargs = kwargs
  

• self.args and self.kwargs will be given to the init, any other init argument is ignored (not saved in the RemoteStruct)

Parameters:

method (Callable) – Method to wrap so that it is executed on the remote server

Remote Struct

class substrafl.remote.remote_struct.RemoteStruct(cls: Type, cls_args: list, cls_kwargs: dict, remote_cls: Type[RemoteMethod], method_name: str, method_parameters: dict, algo_name: str | None)

Bases: object

Contains the wrapped user code and the necessary functions to transform it into a Substra asset to execute on the platform.

Parameters:

• cls (Type) – The remote struct type (e.g. Algorithm, dataset)

• cls_parameters (str) – The class parameters serialized into json string. E.g.: use json.dumps({"args": [], "kwargs": kwargs})

• remote_cls (Type[RemoteMethod]) – The name of the class used remotely

• remote_cls_parameters (str) – The remote class parameters serialized into json string. E.g.: use json.dumps({"args": [], "kwargs": kwargs})

• algo_name (str, Optional) – opportunity to set a custom algo name. If None, set to “{method_name}_{class_name}”

• cls – Locally defined class

• cls_args (list) – Arguments (args) to instantiate the class

• cls_kwargs (dict) – Arguments (kwargs) to instantiate the class

• remote_cls – Remote class to create from the user code

• method_name (str) – Name of the method from the local class to execute

• method_parameters (dict) – Parameters to pass to the method

• algo_name – opportunity to set a custom algo name. If None, set to “{method_name}_{class_name}”

get_instance() → Any

Get the class instance.

Returns:

Instance of the saved class

Return type:

Any

get_remote_instance() → RemoteMethod

Get the remote class (ie Substra algo) instance.

Returns:

instance of the remote Substra class

Return type:

RemoteMethod

classmethod load(src: Path) → RemoteStruct

Load the remote struct from the src directory.

Parameters:

src (pathlib.Path) – Path to the directory where the remote struct has been saved.

Return type:

RemoteStruct

save(dest: Path)

Save the instance to the dest directory using cloudpickle.

Parameters:

dest (pathlib.Path) – directory where to save the remote struct

summary() → Dict[str, str]

Get a summary of what the remote struct represents.

Returns:

description

Return type:

Dict[str, str]

Operations

Dataclasses describing the operations to execute on the remote.

class substrafl.remote.operations.RemoteDataOperation(remote_struct: RemoteStruct, data_samples: List[str], shared_state: Any)

Bases: object

Data operation

Parameters:

• remote_struct (RemoteStruct) –

• data_samples (List[str]) –

• shared_state (Any) –

class substrafl.remote.operations.RemoteOperation(remote_struct: RemoteStruct, shared_states: List | None)

Bases: object

Aggregation operation

Parameters:

• remote_struct (RemoteStruct) –

• shared_states (List | None) –

Substra tools methods

class substrafl.remote.substratools_methods.RemoteMethod(instance, method_name: str, method_parameters: ~typing.Dict, shared_state_serializer: ~typing.Type[~substrafl.remote.serializers.serializer.Serializer] = <class 'substrafl.remote.serializers.pickle_serializer.PickleSerializer'>)

Bases: object

Methods to register to Substra

Parameters:

• method_name (str) –

• method_parameters (Dict) –

• shared_state_serializer (Type[Serializer]) –

generic_function(inputs: TypedDict, outputs: TypedDict, task_properties: TypedDict) → None

Generic function to be registered and executed on the Substra platform using substra-tools.

Parameters:

• inputs (TypedDict) – dictionary containing the paths where to load the arguments for the method.

• outputs (TypedDict) – dictionary containing the paths where to save the output of the method.

• task_properties (TypedDict) – Unused.

Return type:

None

load_instance(path: str | PathLike) → Any

Load the instance from disk

Parameters:

path (Union[str, os.PathLike]) – path to the saved instance

Returns:

loaded instance

Return type:

Any

load_method_inputs(inputs: TypedDict, outputs: TypedDict)

Load the different parameters needed from the inputs and outputs dictionaries and increment a loaded_inputs dictionary depending on the InputIdentifiers or OutputIdentifiers of the parameter.

Parameters:

• inputs (TypedDict) – dictionary containing the paths where to load the arguments for the method.

• outputs (TypedDict) – dictionary containing the paths where to save the output for the method.

Returns:

dictionary containing the kwargs of the method to call.

Return type:

TypedDict

load_shared(path: str | PathLike) → Any

Load the shared state from disk

Parameters:

path (Union[str, os.PathLike]) – path to the saved shared state

Returns:

loaded shared state

Return type:

Any

register_substratools_function()

Register the function that can be accessed and executed by substratools.

save_instance(path: str | PathLike) → None

Save the instance

Parameters:

• model (Any) – Instance to save

• path (Union[str, os.PathLike]) – Path where to save the instance

Return type:

None

save_method_output(method_output: Any, outputs: TypedDict)

Save the method output on the path given in outputs, depending on the value of the OutputIdentifiers.

Parameters:

• method_output (Any) – return value from the called method.

• outputs (TypedDict) – dictionary containing the paths where to save the output for the method.

save_shared(shared_state, path: str | PathLike) → None

Save the shared state

Parameters:

• model (Any) – Shared state to save

• path (Union[str, os.PathLike]) – Path where to save the model

Return type:

None

Register

Create the Substra function assets and register them to the platform.

substrafl.remote.register.register.register_function(*, client: Client, remote_struct: RemoteStruct, permissions: Permissions, inputs: List[FunctionInputSpec], outputs: List[FunctionOutputSpec], dependencies: Dependency) → str

Automatically creates the needed files to register the function associated to the remote_struct.

Parameters:

• client (substra.Client) – The substra client.

• remote_struct (RemoteStruct) – The substra submittable function representation.

• permissions (substra.sdk.schemas.Permissions) – Permissions for the function.

• inputs (List[substra.sdk.schemas.FunctionInputSpec]) – List of function inputs to be used.

• outputs (List[substra.sdk.schemas.FunctionOutputSpec]) – List of function outputs to be used.

• dependencies (Dependency) – Function dependencies.

Returns:

Substra function key.

Return type:

str

substrafl.remote.register.register.register_metrics(*, client: Client, dependencies: Dependency, permissions: Permissions, metric_functions: Dict[str, Callable])

Adds a function to the Substra platform using the given metric functions as the function to register. Each metric function must be of type function, and their signature must ONLY contains datasamples and predictions_path as parameters. An error is raised otherwise.

Parameters:

• client (substra.Client) – The substra client.

• permissions (substra.sdk.schemas.Permissions) – Permissions for the metric function.

• dependencies (Dependency) – Metric function dependencies.

• metric_functions (Dict[str, Callable]) – functions to compute the score from the datasamples and the predictions. These functions are registered in substra as one function.

Returns:

Substra function containing all the given metric functions.

Return type:

str

Serializers

Serializers to save the user code and wrap it in the Substra algo code.

class substrafl.remote.serializers.PickleSerializer

Bases: Serializer

static load(path: Path) → Any

Load an object from a path using pickle.load

Parameters:

path (pathlib.Path) – path to the saved file

Returns:

loaded state

Return type:

Any

static save(state: Any, path: Path)

Pickle the state to path

Parameters:

• state (Any) – state to save

• path (pathlib.Path) – path where to save it

class substrafl.remote.serializers.Serializer

Bases: ABC

abstract static save(state: Any, path: Path)

Save the state to the path

Parameters:

• state (Any) – state to save

• path (pathlib.Path) – path where to save it