Kubernetes deployment reference
For a step-by-step guide on how to deploy the application on Kubernetes, please refer to the the deployment how-to.
This section provides a reference of how various Kubernetes resources should look like once deployed with the default settings. Substra is distributed as Helm charts, and most values can be customized to accommodate the requirements of your environment.
Pod Security Standards
All pods in a Substra deployment are compliant with the baseline policy of the Pod Security Standards.
All pods can run as non-root, with two exceptions:
If the builder feature is enabled (at least one backend per network must have the ability to build images), Kaniko pods used for building images run as root.
If the private CA feature is used, the initContainer add-cert runs as root.
We are working on ensuring that all pods except the two listed above are compliant with the restricted policy.
Network Policies
Substra uses Network Policies to restrict network traffic between pods. A general overview of the network connections between pods (inside a same organisation, and across organisations) can be found below.
Pod |
Incoming |
Outgoing |
|---|---|---|
orchestrator-server |
(from other organisations, over the internet) backend-api-events, backend-worker-events, backend-scheduler, backend-scheduler-worker, backend-server, backend-builder, backend-worker |
orchestrator-database |
orchestrator-database |
orchestrator-server, orchestrator-migrations |
NONE |
frontend |
internet |
NONE |
backend-database |
backend-server, backend-worker, backend-api-events, backend-worker-events, backend-scheduler, backend-scheduler-worker, job-migrations |
NONE |
backend-redis |
backend-server, backend-worker, backend-builder, backend-api-events, backend-worker-events, backend-scheduler, backend-scheduler-worker |
NONE |
backend-object-storage |
backend-server, backend-builder, backend-worker |
NONE |
backend-docker-registry |
backend-task-build, backend-builder, backend-worker, backend-scheduler, backend-scheduler-worker, registry-prepopulate |
NONE |
backend-server |
backend-worker, backend-builder, backend-server (other orgs) |
orchestrator-server, backend-server (other orgs), backend-database, backend-redis, backend-object-storage, backend-docker-registry |
backend-worker |
NONE |
k8s-api, orchestrator-server, backend-server, backend-database, backend-redis, backend-registry, backend-object-storage |
backend-builder |
NONE |
k8s-api, orchestrator-server, backend-server, backend-database, backend-redis, backend-registry, backend-object-storage |
backend-api-events |
NONE |
orchestrator-server, backend-redis, backend-database |
backend-worker-events |
NONE |
k8s-api, orchestrator-server, backend-redis, backend-database |
backend-scheduler |
NONE |
orchestrator-server, backend-redis, backend-database backend-docker-registry |
backend-scheduler-worker |
NONE |
orchestrator-server, backend-redis, backend-database backend-docker-registry |
backend-task-execution |
NONE |
NONE |
backend-task-build |
NONE |
backend-docker-registry, internet (external registries) |
JOBS |
||
registry-prepopulate |
internet |
backend-docker-registry |
job-delete-compute-task |
NONE |
k8s-api |
job-delete-stateful-pvc |
NONE |
k8s-api |
job-migrations (backend) |
NONE |
backend-database |
job-migrations (orc) |
NONE |
orchestrator-database |
Note
All pods are also given access to kube-dns on port 53.
The implementation chosen by Substra does not rely on any external network plugin, in order to maximize compatibility. It defines a set of roles (minimal network policies that block or allow a given connection, or IP ranges) and relies on label selectors to apply these roles to appropriate pods. You can adapt each of those roles to your own network configuration, by adjusting the templates networkpolicy-*.yaml in Helm charts.
Broadly speaking, we can distinguish 3 kinds of pods:
the pods that execute the compute functions are fully isolated (no incoming nor outgoing connections);
“storage” pods (database, redis, object storage, docker registry) are only accessible from inside the cluster, and have no outgoing connections;
other pods requires a connection to the orchestrator, or other backend pods, and are typically communicating over the internet, if you do not have a network plugin that allows for finer filtering.
Volumes
This section lists the different persistent volume claims that are created during a standard deployment of Substra in a cluster.
Orchestrator
The orchestrator claims are only linked with its database. This database (and the underlying volume) is important has it is where we store all the events that happen in the network.
PVC name |
Component |
Pod |
access mode |
Volume Size default (Gi) |
storage class |
reclaim policy |
Can be reused |
How to re use |
|---|---|---|---|---|---|---|---|---|
Postgres PVC |
orchestrator |
postgres |
ReadWriteOnce (can be modified) |
8 |
<empty> |
default |
yes |
postgresql.primary.persistence.existingClaim |
Backend
The backend is the most complex component and it requires different volumes for functioning. Volumes that should be persisted on the long term can be created outside of the deployment of the Substra stack. It is not currently possible to re-use existing volumes for the other ones (acting as cache).
PVC name |
Component |
Pod |
access mode |
Volume Size default (Gi) |
storage class |
reclaim policy |
Can be reused |
How to re use |
Comment |
|---|---|---|---|---|---|---|---|---|---|
Postgres PVC |
backend |
postgres |
ReadWriteOnce (can be modified) |
8 |
<empty> |
default |
yes |
postgresql.primary.persistence.existingClaim |
|
Localstack PVC |
backend |
localstack |
ReadWriteOnce (can be modified) |
5 |
<empty> |
default |
yes |
localstack.persistence.existingClaim |
Only created when localstack.enabled = true |
Minio PVC |
backend |
minio |
ReadWriteOnce (can be modified) |
8 |
<empty> |
default |
yes |
minio.persistence.existingClaim |
Only created when minio.enabled = true |
Redis PVC |
backend |
redis |
ReadWriteOnce (can be modified) |
8 |
<empty> |
default |
yes |
redis.master.persistence.existingClaim |
|
Kaniko cache warmer |
backend |
builder |
ReadWriteOnce |
10 |
<empty> |
default |
no |
||
Builder PVC |
backend |
builder |
ReadWriteOnce |
10 |
<empty> |
default |
no |
||
Worker subtuple |
backend |
worker |
ReadWriteOnce (can be modified) |
10 |
<empty> |
default |
no |
||
Docker registry |
backend |
registry |
ReadWriteOnce (can be modified) |
10 |
<empty> |
default |
yes |
docker-registry.persistence.existingClaim |
Only created when docker-registry.enabled = true |
Frontend
The frontend does not need any persistent volume claim.