Remote¶
This is an internal SubstraFL module, the user should not use
any functions here directly, apart from the remote_data
and remote
decorators.
This modules defines how the user code is wrapped, transformed and registered as
Substra algorithms.
Decorator¶
Decorators to wrap functions so that they are executed on the remote organizations.
- substrafl.remote.decorators.remote(method: Callable)¶
Decorator for a remote function. With this decorator, when the function is called, it is not executed but it returns a
RemoteOperation
object containing all the informations needed to execute it later (seesubstrafl.remote.operations.RemoteOperation
).The decorated function definition should have at least a shared_state argument
If the decorated function is called without a
_skip=True
argument, the arguments required are the ones inremote_method_inner
If the decorated function is called with a
_skip=True
argument, it should have the arguments of its original definitionThe decorated function should be within a class
The
__init__
of the class must bedef __init__(self, *args, **kwargs): self.args = args self.kwargs = kwargs
self.args
andself.kwargs
will be given to the init, any other init argument is ignored (not saved in the RemoteStruct)
- Parameters
method (Callable) – Method to wrap so that it is executed on the remote server
- substrafl.remote.decorators.remote_data(method: Callable)¶
Decorator for a remote function containing a
data_samples
argument (e.g theAlgo.train
function) With this decorator, when the function is called, it is not executed but it returns aRemoteDataOperation
object containing all the informations needed to execute it later (seesubstrafl.remote.operations.RemoteDataOperation
).The decorated function definition should have at least a shared_state argument
If the decorated function is called without a
_skip=True
argument, the arguments required are the ones inremote_method_inner
, and it should have at least adata_samples
argumentIf the decorated function is called with a
_skip=True
argument, it should have the arguments of its original definitionThe decorated function should be within a class
The
__init__
of the class must bedef __init__(self, *args, **kwargs): self.args = args self.kwargs = kwargs
self.args
andself.kwargs
will be given to the init, any other init argument is ignored (not saved in the RemoteStruct)
- Parameters
method (Callable) – Method to wrap so that it is executed on the remote server
Remote Struct¶
- class substrafl.remote.remote_struct.RemoteStruct(cls: Type, cls_args: list, cls_kwargs: dict, remote_cls: Type[substrafl.remote.substratools_methods.RemoteMethod], method_name: str, method_parameters: dict, algo_name: Optional[str])¶
Bases:
object
Contains the wrapped user code and the necessary functions to transform it into a Substra asset to execute on the platform.
- Parameters
cls (Type) – The remote struct type (e.g. Algorithm, dataset)
cls_parameters (str) – The class parameters serialized into json string. E.g.: use
json.dumps({"args": [], "kwargs": kwargs})
remote_cls (Type[RemoteMethod]) – The name of the class used remotely
remote_cls_parameters (str) – The remote class parameters serialized into json string. E.g.: use
json.dumps({"args": [], "kwargs": kwargs})
algo_name (str, Optional) – opportunity to set a custom algo name. If None, set to “{method_name}_{class_name}”
cls – Locally defined class
cls_args (list) – Arguments (args) to instantiate the class
cls_kwargs (dict) – Arguments (kwargs) to instantiate the class
remote_cls – Remote class to create from the user code
method_name (str) – Name of the method from the local class to execute
method_parameters (dict) – Parameters to pass to the method
algo_name – opportunity to set a custom algo name. If None, set to “{method_name}_{class_name}”
- get_remote_instance() substrafl.remote.substratools_methods.RemoteMethod ¶
Get the remote class (ie Substra algo) instance.
- Returns
instance of the remote Substra class
- Return type
- classmethod load(src: pathlib.Path) substrafl.remote.remote_struct.RemoteStruct ¶
Load the remote struct from the src directory.
- Parameters
src (pathlib.Path) – Path to the directory where the remote struct has been saved.
- Return type
- save(dest: pathlib.Path)¶
Save the instance to the dest directory using cloudpickle.
- Parameters
dest (pathlib.Path) – directory where to save the remote struct
Operations¶
Dataclasses describing the operations to execute on the remote.
- class substrafl.remote.operations.RemoteDataOperation(remote_struct: substrafl.remote.remote_struct.RemoteStruct, data_samples: List[str], shared_state: Any)¶
Bases:
object
Data operation
- Parameters
remote_struct (substrafl.remote.remote_struct.RemoteStruct) –
shared_state (Any) –
- Return type
None
- class substrafl.remote.operations.RemoteOperation(remote_struct: substrafl.remote.remote_struct.RemoteStruct, shared_states: Optional[List])¶
Bases:
object
Aggregation operation
- Parameters
remote_struct (substrafl.remote.remote_struct.RemoteStruct) –
- Return type
None
Substra tools methods¶
- class substrafl.remote.substratools_methods.RemoteMethod(instance, method_name: str, method_parameters: typing.Dict, shared_state_serializer: typing.Type[substrafl.remote.serializers.serializer.Serializer] = <class 'substrafl.remote.serializers.pickle_serializer.PickleSerializer'>)¶
Bases:
object
Methods to register to Substra
- Parameters
method_name (str) –
method_parameters (Dict) –
shared_state_serializer (Type[substrafl.remote.serializers.serializer.Serializer]) –
- generic_function(inputs: TypedDict, outputs: TypedDict, task_properties: TypedDict) None ¶
Generic function to be registered and executed on the Substra platform using substra-tools.
- Parameters
inputs (TypedDict) – dictionary containing the paths where to load the arguments for the method.
outputs (TypedDict) – dictionary containing the paths where to save the output of the method.
task_properties (TypedDict) – Unused.
- Return type
None
- load_instance(path: Union[str, os.PathLike]) Any ¶
Load the instance from disk
- Parameters
path (Union[str, os.PathLike]) – path to the saved instance
- Returns
loaded instance
- Return type
Any
- load_method_inputs(inputs: TypedDict, outputs: TypedDict)¶
Load the different parameters needed from the inputs and outputs dictionaries and increment a loaded_inputs dictionary depending on the InputIdentifiers or OutputIdentifiers of the parameter.
- Parameters
inputs (TypedDict) – dictionary containing the paths where to load the arguments for the method.
outputs (TypedDict) – dictionary containing the paths where to save the output for the method.
- Returns
dictionary containing the kwargs of the method to call.
- Return type
TypedDict
Load the shared state from disk
- Parameters
path (Union[str, os.PathLike]) – path to the saved shared state
- Returns
loaded shared state
- Return type
Any
- register_substratools_function()¶
Register the function that can be accessed and executed by substratools.
- save_instance(path: Union[str, os.PathLike]) None ¶
Save the instance
- Parameters
model (Any) – Instance to save
path (Union[str, os.PathLike]) – Path where to save the instance
- Return type
None
- save_method_output(method_output: Any, outputs: TypedDict)¶
Save the method output on the path given in outputs, depending on the value of the OutputIdentifiers.
- Parameters
method_output (Any) – return value from the called method.
outputs (TypedDict) – dictionary containing the paths where to save the output for the method.
Save the shared state
- Parameters
model (Any) – Shared state to save
path (Union[str, os.PathLike]) – Path where to save the model
- Return type
None
Register¶
Create the Substra function assets and register them to the platform.
- substrafl.remote.register.register.register_function(*, client: substra.sdk.client.Client, remote_struct: substrafl.remote.remote_struct.RemoteStruct, permissions: substra.sdk.schemas.Permissions, inputs: List[substra.sdk.schemas.FunctionInputSpec], outputs: List[substra.sdk.schemas.FunctionOutputSpec], dependencies: substrafl.dependency.schemas.Dependency) str ¶
Automatically creates the needed files to register the function associated to the remote_struct.
- Parameters
client (substra.Client) – The substra client.
remote_struct (RemoteStruct) – The substra submittable function representation.
permissions (substra.sdk.schemas.Permissions) – Permissions for the function.
inputs (List[substra.sdk.schemas.FunctionInputSpec]) – List of function inputs to be used.
outputs (List[substra.sdk.schemas.FunctionOutputSpec]) – List of function outputs to be used.
dependencies (Dependency) – Function dependencies.
- Returns
Substra function key.
- Return type
- substrafl.remote.register.register.register_metrics(*, client: substra.sdk.client.Client, dependencies: substrafl.dependency.schemas.Dependency, permissions: substra.sdk.schemas.Permissions, metric_functions: Dict[str, Callable])¶
Adds a function to the Substra platform using the given metric functions as the function to register. Each metric function must be of type function, and their signature must ONLY contains datasamples and predictions_path as parameters. An error is raised otherwise.
- Parameters
client (substra.Client) – The substra client.
permissions (substra.sdk.schemas.Permissions) – Permissions for the metric function.
dependencies (Dependency) – Metric function dependencies.
metric_functions (Dict[str, Callable]) – functions to compute the score from the datasamples and the predictions. These functions are registered in substra as one function.
- Returns
Substra function containing all the given metric functions.
- Return type
Serializers¶
Serializers to save the user code and wrap it in the Substra algo code.
- class substrafl.remote.serializers.PickleSerializer¶
Bases:
substrafl.remote.serializers.serializer.Serializer
- static load(path: pathlib.Path) Any ¶
Load an object from a path using pickle.load
- Parameters
path (pathlib.Path) – path to the saved file
- Returns
loaded state
- Return type
Any
- static save(state: Any, path: pathlib.Path)¶
Pickle the state to path
- Parameters
state (Any) – state to save
path (pathlib.Path) – path where to save it
- class substrafl.remote.serializers.Serializer¶
Bases:
abc.ABC
- abstract static save(state: Any, path: pathlib.Path)¶
Save the state to the path
- Parameters
state (Any) – state to save
path (pathlib.Path) – path where to save it