If you are running on AWS ECS or another container-based orchestration system, you'll likely want to package the Dagster webserver using a Docker image.
A minimal skeleton Dockerfile that will run the webserver is shown below:
FROM python:3.10-slim RUN mkdir -p /opt/dagster/dagster_home /opt/dagster/app RUN pip install dagster-webserver dagster-postgres dagster-aws # Copy your code and workspace to /opt/dagster/app COPY repo.py workspace.yaml /opt/dagster/app/ ENV DAGSTER_HOME=/opt/dagster/dagster_home/ # Copy dagster instance YAML to $DAGSTER_HOME COPY dagster.yaml /opt/dagster/dagster_home/ WORKDIR /opt/dagster/app EXPOSE 3000 ENTRYPOINT ["dagster-webserver", "-h", "0.0.0.0", "-p", "3000"]
You'll also need to include a workspace file (workspace.yaml) file in the same directory as the Dockerfile to configure your code location server:
load_from: # References the file copied into your Dockerfile - python_file: repo.py
As well as a dagster.yaml file to configure your Dagster instance:
storage: postgres: postgres_db: username: env: DAGSTER_PG_USERNAME password: env: DAGSTER_PG_PASSWORD hostname: env: DAGSTER_PG_HOST db_name: env: DAGSTER_PG_DB port: 5432 compute_logs: module: dagster_aws.s3.compute_log_manager class: S3ComputeLogManager config: bucket: "mycorp-dagster-compute-logs" prefix: "dagster-test-" local_artifact_storage: module: dagster.core.storage.root class: LocalArtifactStorage config: base_dir: "/opt/dagster/local/"
In cases where you're using environment variables to configure the instance, you should ensure these environment variables are exposed in the container running dagster-webserver.
The webserver exposes a health check endpoint at /server_info, which returns a JSON response like:
{ "dagster_webserver_version": "0.12.0", "dagster_graphql_version": "0.12.0", "dagster_version": "0.12.0" }
More advanced Dagster deployments will require deploying more than one container. For example, if you are using the Dagster daemon to run schedules and sensors or manage a queue of runs, you'll likely want a separate container running the dagster-daemon service.
This service must have access to your dagster.yaml and workspace.yaml files, just like the webserver container. You can also configure your code location server (gRPC server) so that your code can be updated and deployed separately in its own container, without needing to redeploy the other Dagster services. To enable this setup, include a container exposing a gRPC server at a port and add that port in your workspace.yaml file.
For example, your user code container might have the following Dockerfile:
FROM python:3.10-slim # Checkout and install dagster libraries needed to run the gRPC server by exposing # your code location to dagster-webserver and dagster-daemon, and loading the # DagsterInstance. RUN pip install \ dagster \ dagster-postgres \ dagster-docker # Set $DAGSTER_HOME and copy dagster instance there ENV DAGSTER_HOME=/opt/dagster/dagster_home RUN mkdir -p $DAGSTER_HOME COPY dagster.yaml $DAGSTER_HOME # Add repository code WORKDIR /opt/dagster/app COPY repo.py /opt/dagster/app # Run dagster gRPC server on port 4000 EXPOSE 4000 # Using CMD rather than ENTRYPOINT allows the command to be overridden in # run launchers or executors to run other commands using this image CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "repo.py"]
And your code location server might look like:
load_from: # Each entry here corresponds to a container that exposes a gRPC server. - grpc_server: host: docker_example_user_code port: 4000 location_name: "example_user_code"
When you update your code, you can rebuild and restart your user code container without needing to redeploy other parts of the system. The Dagster UI will automatically notice that a new server has been redeployed and prompt you to refresh.
When you add or remove a user code container, you can also remove the corresponding entry from your workspace.yaml file. If this file is mounted on the webserver Dagster daemon containers as a volume, you can pick up the changes in the UI by reloading from the Deployment tab. The Dagster daemon container will automatically pick up the changes by periodically reloading the workspace.yaml file.
To launch each run in its own container, you can add the DockerRunLauncher to your dagster.yaml file:
run_launcher: module: dagster_docker class: DockerRunLauncher config: env_vars: - DAGSTER_POSTGRES_USER - DAGSTER_POSTGRES_PASSWORD - DAGSTER_POSTGRES_DB
This launcher will start each run in a new container, using whatever image set in the DAGSTER_CURRENT_IMAGE environment variable in your user code container. This will usually be the same image as the user code container itself.
Any container that launches runs - usually the dagster-daemon container if you're maintaining a run queue or launching runs from schedules or sensors - must have permissions to create Docker containers in order to use this run launcher. Mounting /var/run/docker.sock as a volume is one way to give it these permissions.
You can mount your code in your user code container so that you don't have to rebuild your container whenever your code changes. Even if you're using volume mounts, you still need to restart the container whenever your code changes.
If you're mounting your code as a volume in your user code container and using DockerRunLauncher to launch each run in a new container, you must specify your volume mounts in the DockerRunLauncher config as well. For example:
run_launcher: module: dagster_docker class: DockerRunLauncher config: env_vars: - DAGSTER_POSTGRES_USER - DAGSTER_POSTGRES_PASSWORD - DAGSTER_POSTGRES_DB container_kwargs: volumes: - /absolute/path/to/local/repo.py:/opt/dagster/app/
This example demonstrates a Dagster deployment using docker-compose that includes:
dagster-daemon container for managing a run queue and submitting runs from schedules and sensorsDockerRunLauncher to launch each run in its own container.To start the deployment, run:
docker-compose up