Remote¶

This is an internal SubstraFL module, the user should not use any functions here directly, apart from the remote_data and remote decorators. This modules defines how the user code is wrapped, transformed and registered as Substra algorithms.

Decorator¶

Decorators to wrap functions so that they are executed on the remote organizations.

substrafl.remote.decorators.remote(method: Callable)¶

Decorator for a remote function. With this decorator, when the function is called, it is not executed but it returns a RemoteOperation object containing all the informations needed to execute it later (see substrafl.remote.operations.RemoteOperation).

The decorated function definition should have at least a shared_state argument

If the decorated function is called without a _skip=True argument, the arguments required are the ones in remote_method_inner

If the decorated function is called with a _skip=True argument, it should have the arguments of its original definition

The decorated function should be within a class
The __init__ of the class must be
def __init__(self, *args, **kwargs):
    self.args = args
    self.kwargs = kwargs
self.args and self.kwargs will be given to the init, any other init argument is ignored (not saved in the RemoteStruct)

Parameters: method (Callable) – Method to wrap so that it is executed on the remote server

substrafl.remote.decorators.remote_data(method: Callable)¶

Decorator for a remote function containing a data_samples argument (e.g the Algo.train function) With this decorator, when the function is called, it is not executed but it returns a RemoteDataOperation object containing all the informations needed to execute it later (see substrafl.remote.operations.RemoteDataOperation).

The decorated function definition should have at least a shared_state argument

If the decorated function is called without a _skip=True argument, the arguments required are the ones in remote_method_inner, and it should have at least a data_samples argument

If the decorated function is called with a _skip=True argument, it should have the arguments of its original definition

The decorated function should be within a class
The __init__ of the class must be
def __init__(self, *args, **kwargs):
    self.args = args
    self.kwargs = kwargs
self.args and self.kwargs will be given to the init, any other init argument is ignored (not saved in the RemoteStruct)

Parameters: method (Callable) – Method to wrap so that it is executed on the remote server

Remote Struct¶

class substrafl.remote.remote_struct.RemoteStruct(cls: Type, cls_args: list, cls_kwargs: dict, remote_cls: Type[substrafl.remote.substratools_methods.RemoteMethod], method_name: str, method_parameters: dict, algo_name: Optional[str])¶

Bases: object

Contains the wrapped user code and the necessary functions to transform it into a Substra asset to execute on the platform.

Parameters

cls (Type) – The remote struct type (e.g. Algorithm, dataset)
cls_parameters (str) – The class parameters serialized into json string. E.g.: use json.dumps({"args": [], "kwargs": kwargs})
remote_cls (Type[RemoteMethod]) – The name of the class used remotely
remote_cls_parameters (str) – The remote class parameters serialized into json string. E.g.: use json.dumps({"args": [], "kwargs": kwargs})
algo_name (str, Optional) – opportunity to set a custom algo name. If None, set to “{method_name}_{class_name}”
cls – Locally defined class
cls_args (list) – Arguments (args) to instantiate the class
cls_kwargs (dict) – Arguments (kwargs) to instantiate the class
remote_cls – Remote class to create from the user code
method_name (str) – Name of the method from the local class to execute
method_parameters (dict) – Parameters to pass to the method
algo_name – opportunity to set a custom algo name. If None, set to “{method_name}_{class_name}”

get_instance() → Any¶

Get the class instance.

Returns: Instance of the saved class
Return type: Any

get_remote_instance() → substrafl.remote.substratools_methods.RemoteMethod¶

Get the remote class (ie Substra algo) instance.

Returns: instance of the remote Substra class
Return type: RemoteMethod

classmethod load(src: pathlib.Path) → substrafl.remote.remote_struct.RemoteStruct¶

Load the remote struct from the src directory.

Parameters: src (pathlib.Path) – Path to the directory where the remote struct has been saved.
Return type: substrafl.remote.remote_struct.RemoteStruct

save(dest: pathlib.Path)¶

Save the instance to the dest directory using cloudpickle.

Parameters: dest (pathlib.Path) – directory where to save the remote struct

summary() → Dict[str, str]¶

Get a summary of what the remote struct represents.

Returns: description
Return type: Dict[str, str]

Operations¶

Dataclasses describing the operations to execute on the remote.

class substrafl.remote.operations.RemoteDataOperation(remote_struct: substrafl.remote.remote_struct.RemoteStruct, data_samples: List[str], shared_state: Any)¶

Bases: object

Data operation

Parameters

remote_struct (substrafl.remote.remote_struct.RemoteStruct) –
data_samples (List[str]) –
shared_state (Any) –

Return type

None

class substrafl.remote.operations.RemoteOperation(remote_struct: substrafl.remote.remote_struct.RemoteStruct, shared_states: Optional[List])¶

Bases: object

Aggregation operation

Parameters

remote_struct (substrafl.remote.remote_struct.RemoteStruct) –
shared_states (Optional[List]) –

Return type

None

Substra tools methods¶

class substrafl.remote.substratools_methods.RemoteMethod(instance, method_name: str, method_parameters: typing.Dict, shared_state_serializer: typing.Type[substrafl.remote.serializers.serializer.Serializer] = <class 'substrafl.remote.serializers.pickle_serializer.PickleSerializer'>)¶

Bases: object

Methods to register to Substra

Parameters

method_name (str) –
method_parameters (Dict) –
shared_state_serializer (Type[substrafl.remote.serializers.serializer.Serializer]) –

generic_function(inputs: TypedDict, outputs: TypedDict, task_properties: TypedDict) → None¶

Generic function to be registered and executed on the Substra platform using substra-tools.

Parameters

inputs (TypedDict) – dictionary containing the paths where to load the arguments for the method.
outputs (TypedDict) – dictionary containing the paths where to save the output of the method.
task_properties (TypedDict) – Unused.

Return type

None

load_instance(path: str) → Any¶

Load the instance from disk

Parameters: path (str) – path to the saved instance
Returns: loaded instance
Return type: Any

load_method_inputs(inputs: TypedDict, outputs: TypedDict)¶

Load the different parameters needed from the inputs and outputs dictionaries and increment a loaded_inputs dictionary depending on the InputIdentifiers or OutputIdentifiers of the parameter.

Parameters

inputs (TypedDict) – dictionary containing the paths where to load the arguments for the method.
outputs (TypedDict) – dictionary containing the paths where to save the output for the method.

Returns

dictionary containing the kwargs of the method to call.

Return type

TypedDict

load_model(path: str) → Any¶

Load the model from disk

Parameters: path (str) – path to the saved model
Returns: loaded model
Return type: Any

register_substratools_function()¶: Register the function that can be accessed and executed by substratools.

save_instance(path: str) → None¶

Save the instance

Parameters

model (Any) – Instance to save
path (str) – Path where to save the instance

Return type

None

save_method_output(method_output: Any, outputs: TypedDict)¶

Save the method output on the path given in outputs, depending on the value of the OutputIdentifiers.

Parameters

method_output (Any) – return value from the called method.
outputs (TypedDict) – dictionary containing the paths where to save the output for the method.

save_model(model, path: str) → None¶

Save the model

Parameters

model (Any) – Model to save
path (str) – Path where to save the model

Return type

None

Register¶

Create the Substra function assets and register them to the platform.

substrafl.remote.register.register.add_metric(client: substra.sdk.client.Client, permissions: substra.sdk.schemas.Permissions, dependencies: substrafl.dependency.Dependency, metric_function: Callable, metric_name: Optional[str] = None) → str¶

Adds a metric to the Substra platform using the given metric function as the function to register. The metric function must be of type function, and its signature must ONLY contains datasamples and predictions_path as parameters. An error is raised otherwise.

Parameters

client (substra.Client) – The substra client.
permissions (substra.sdk.schemas.Permissions) – Permissions for the metric function.
dependencies (Dependency) – Metric function dependencies.
metric_function (Callable) – function to compute the score from the datasamples and the predictions. This function is registered in substra as a metric.
metric_name (str, Optional) – Optional name chosen by the user to identify the metric. If None, the metric name is set to the ‘metric_{metric_function.__name__}’.

Returns

The key of the function created from the metric function.

Return type

str

substrafl.remote.register.register.register_function(client: substra.sdk.client.Client, remote_struct: substrafl.remote.remote_struct.RemoteStruct, permissions: substra.sdk.schemas.Permissions, inputs: List[substra.sdk.schemas.FunctionInputSpec], outputs: List[substra.sdk.schemas.FunctionOutputSpec], dependencies: substrafl.dependency.Dependency) → str¶

Automatically creates the needed files to register the function associated to the remote_struct.

Parameters

client (substra.Client) – The substra client.
remote_struct (RemoteStruct) – The substra submittable function representation.
permissions (substra.sdk.schemas.Permissions) – Permissions for the function.
inputs (List[substra.sdk.schemas.FunctionInputSpec]) – List of function inputs to be used.
outputs (List[substra.sdk.schemas.FunctionOutputSpec]) – List of function outputs to be used.
dependencies (Dependency) – Function dependencies.

Returns

Substra function key.

Return type

str

Generate wheels for the Substra algo.

substrafl.remote.register.generate_wheel.local_lib_wheels(lib_modules: List, operation_dir: pathlib.Path, python_major_minor: str, dest_dir: str) → str¶

Prepares the private modules from lib_modules list to be installed in a Docker image and generates the appropriated install command for a dockerfile. It first creates the wheel for each library. Each of the libraries must be already installed in the correct version locally. Use command: pip install -e library-name in the directory of each library.

This allows one user to use custom version of the passed modules.

Parameters

lib_modules (list) – list of modules to be installed.
operation_dir (pathlib.Path) – PosixPath to the operation directory
python_major_minor (str) – version which is to be used in the dockerfile. Eg: ‘3.8’
dest_dir (str) – relative directory where the wheels are saved

Returns

dockerfile command for installing the given modules

Return type

str

substrafl.remote.register.generate_wheel.pypi_lib_wheels(lib_modules: List, operation_dir: pathlib.Path, python_major_minor: str, dest_dir: str) → str¶

Retrieves lib_modules’ wheels to be installed in a Docker image and generates the appropriated install command for a dockerfile.

Parameters

lib_modules (list) – list of modules to be installed.
operation_dir (pathlib.Path) – PosixPath to the operation directory
python_major_minor (str) – version which is to be used in the dockerfile. Eg: ‘3.8’
dest_dir (str) – relative directory where the wheels are saved

Returns

dockerfile command for installing the given modules

Return type

str

Serializers¶

Serializers to save the user code and wrap it in the Substra algo code.

class substrafl.remote.serializers.PickleSerializer¶

Bases: substrafl.remote.serializers.serializer.Serializer

static load(path: pathlib.Path) → Any¶

Load an object from a path using pickle.load

Parameters: path (pathlib.Path) – path to the saved file
Returns: loaded state
Return type: Any

static save(state: Any, path: pathlib.Path)¶

Pickle the state to path

Parameters

state (Any) – state to save
path (pathlib.Path) – path where to save it

class substrafl.remote.serializers.Serializer¶

Bases: abc.ABC

abstract static save(state: Any, path: pathlib.Path)¶

Save the state to the path

Parameters

state (Any) – state to save
path (pathlib.Path) – path where to save it