Dependency

Schemas

class substrafl.dependency.schemas.Dependency(*args, editable_mode: bool = False, compile: bool = False, pypi_dependencies: List[str] = None, local_installable_dependencies: List[pathlib.Path] = None, local_code: List[pathlib.Path] = None, excluded_paths: List[pathlib.Path] = None, excluded_regex: List[str] = None, force_included_paths: List[pathlib.Path] = None)

Bases: pydantic.main.BaseModel

Dependency pydantic class.

Note

If you are using your current package as a local dependencies, be aware that folders named local-worker or with tmp_substrafl as prefix are ignored during the installation.

Parameters

• editable_mode (bool) – If set to False, substra, substrafl and substratools used in the Dockerfiles submitted to Substra platform will be taken from pypi. If set to True, it will be the one installed in editable mode from your python environment. Defaults to False.

• compile (bool) – If set to True, dependencies will be resolved only once (using pip compile) and the set of compatible versions for each dependencies (and indirect dependencies) will be reused. Default to False.

• pypi_dependencies (List[str]) – Python packages installable from PyPI.

• local_installable_dependencies (List[pathlib.Path]) – Local installable packages. Each one can either be a wheel or a local folder. If it’s a local folder, the command python -m pip wheel . will be run, so each folder needs to be a valid Python module (containing a valid setup.py or pyproject.toml). See the documentation of pip wheel for more details.

• local_code (List[pathlib.Path]) – Local relative imports used by your script. All files / folders will be pasted to the level of the running script.

• excluded_paths (List[pathlib.Path]) – Local paths excluded from local_installable_dependencies / local_code. Default to [].

• excluded_regex (List[pathlib.Path]) – Regex used to exclude files from local_installable_dependencies / local_code. Default to []. Always excludes common data formats (see substrafl.dependency.path_management.EXCLUDED_PATHS_REGEX_DEFAULT).

• force_included_paths (List[pathlib.Path]) – Force include files otherwise excluded by excluded_paths and excluded_regex Default to []

Return type

None

Dependencies are computed at object initialization. The computation is stored in a cache directory, that will be cleaned up at the object deletion.

property cache_directory: pathlib.Path

Getter method to retrieve the path to the cache directory where the dependencies are computed.

Raises

exceptions.DependencyCacheNotFoundError – If no cache directory is found, raise an exception.

Returns

return the path to the cache directory of the dependency.

Return type

Path

classmethod check_setup(v)

Check the presence of a setup.py file or a pyproject.toml in the provided paths.

classmethod resolve_path(v)

Resolve list of local code paths and check if they exist.

Manage Dependencies

Utility functions to manage dependencies (building wheels, compiling requirement…)

substrafl.dependency.manage_dependencies.build_user_dependency_wheel(lib_path: pathlib.Path, dest_dir: pathlib.Path) → str

Build the wheel for user dependencies passed as a local module. Delete the local module when the build is done.

Parameters

• lib_path (Path) – where the module is located.

• dest_dir (Path) – where the wheel needs to be copied.

Returns

the filename of the wheel

Return type

str

substrafl.dependency.manage_dependencies.compile_requirements(dependency_list: List[Union[str, pathlib.Path]], *, dest_dir: pathlib.Path) → None

Compile a list of requirements using pip-compile to generate a set of fully pinned third parties requirements

Writes down a requirements.in file with the list of explicit dependencies, then generates a requirements.txt file using pip-compile. The requirements.txt file contains a set of fully pinned dependencies, including indirect dependencies.

Parameters

• dependency_list (List[Union[str, pathlib.Path]]) – list of dependencies to install; acceptable formats are library names (eg “substrafl”), any constraint expression accepted by pip(“substrafl==0.36.0” or “substrafl < 1.0”) or wheel names (“substrafl-0.36.0-py3-none-any.whl”)

• dest_dir (pathlib.Path) – path to the directory where to write the requirements.in and requirements.txt.

Raises

InvalidDependenciesError – if pip-compile does not find a set of compatible dependencies

Return type

None

substrafl.dependency.manage_dependencies.get_pypi_dependencies_versions(lib_modules: List) → List[str]

Retrieve the version of the PyPI libraries installed to generate the dependency list

Parameters

lib_modules (list) – list of modules to be installed.

Returns

list of dependencies to install in the Docker container

Return type

list(str)

substrafl.dependency.manage_dependencies.local_lib_wheels(lib_modules: List[module], *, dest_dir: pathlib.Path) → List[str]

Generate wheels for the private modules from lib_modules list and returns the list of names for each wheel.

It first creates the wheel for each library. Each of the libraries must be already installed in the correct version locally. Use command: pip install -e library-name in the directory of each library. Then it copies the wheels to the given directory.

This allows one user to use custom version of the passed modules.

Parameters

• lib_modules (list) – list of modules to be installed.

• dest_dir (str) – relative directory where the wheels are saved

Returns

wheel names for the given modules

Return type

List[str]

substrafl.dependency.manage_dependencies.write_requirements(dependency_list: List[Union[str, pathlib.Path]], *, dest_dir: pathlib.Path) → None

Writes down a requirements.txt file with the list of explicit dependencies.

Parameters

• dependency_list (List[Union[str, pathlib.Path]]) – list of dependencies to install; acceptable formats are library names (eg “substrafl”), any constraint expression accepted by pip(“substrafl==0.36.0” or “substrafl < 1.0”) or wheel names (“substrafl-0.36.0-py3-none-any.whl”)

• dest_dir (pathlib.Path) – path to the directory where to write the requirements.txt.

Return type

None