Backend

The backend exposes the REST API for an Organization and executes compute tasks (in a subsystem we call compute engine).

Kubernetes pods

docker-registry

We use this service to store images built from user-provided Functions. Make sure to assign a large enough volume to avoid rebuilding images over and over due to eviction.

registry-prepopulate

This Pod is managed by a Job running on chart installation or update. It uploads container Images to the docker-registry to make them available for future use in Functions.

minio

MinIO is an object storage service and stores all assets registered on the Organization. You should back up the data of this Pod.

postgresql

This is the database supporting the backend. You should back up the data of this Pod.

redis

This is an organization-specific message broker to support Celery tasks.

backend-events

This component will consume events from the orchestrator. It should be able to access the orchestrator over gRPC. It handles events and triggers appropriate responses such as starting compute tasks. On startup, it will also register the Organization on the orchestrator.

migrations

This Pod is managed by a Job running on chart installation or update to deal with database schema changes. This Pod also performs user creation.

scheduler, scheduler-worker

Those are Celery components, handling scheduled tasks.

server

This is a Django application exposing the REST API through which users interact with Substra.

worker

This is the service processing Celery tasks. It handles Function images builds and running compute tasks. This is where you will find logs related to task processing.

Communication

The backend should be able to reach its orchestrator. If Organizations share Models, involved backends must be able to communicate with each other.

Helm chart

We use Helm charts as a way to package our application deployments. If you want to deploy the backend you can use the Helm chart substra-backend.