Skip to main content

Serverless runtime environment

By default, Dagster+ Serverless will package your code as PEX files and deploy them on Docker images. Using PEX files significantly reduces the time to deploy since it does not require building a new Docker image and provisioning a new container for every code change. However, you are able to customize the Serverless runtime environment in various ways:

Add dependencies​

You can add dependencies by including the corresponding Python libraries in your Dagster project's package configuration file.

If you are using uv with a pyproject.toml file, you should follow the pyproject.toml specification.

Example pyproject.toml
[project]
name = "quickstart_etl"
requires-python = ">=3.9,<3.14"
version = "0.1.0"
dependencies = [
    "dagster",
    "dagster-cloud",
    "boto3",
    "matplotlib",
    "pandas",
    "textblob",
    "tweepy",
    "wordcloud",
]

[project.optional-dependencies]
dev = [
    "dagster-dg-cli",
    "dagster-webserver",
    "pytest",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.dg]
directory_type = "project"

[tool.dg.project]
root_module = "quickstart_etl"
registry_modules = [
    "quickstart_etl.components.*",
]

Add dependencies from tarballs​

Example pyproject.toml with tarball links
[project]
name = "quickstart_etl"
requires-python = ">=3.9,<3.14"
version = "0.1.0"
dependencies = [
    "dagster",
    "dagster-cloud",
    "boto3",
    "matplotlib",
    "pandas",
    "textblob",
    "tweepy",
    "wordcloud",
    "soda @ https://pypi.cloud.soda.io/packages/soda-1.6.2.tar.gz",
    "soda-snowflake @ https://pypi.cloud.soda.io/packages/soda_snowflake-1.6.2.tar.gz",
]
tip

To add a package from a private GitHub repository, see Use private Python packages.

Use a different Python version​

Python versions 3.9 through 3.13 are all supported for Serverless deployments. You can specify the Python version you want to use in your GitHub or GitLab workflow, or by using the dagster-cloud CLI.

In your .github/workflows/deploy.yml file, update the PYTHON_VERSION environment variable with your desired Python version:

Updating the Python version in deploy.yml
env:
  DAGSTER_CLOUD_URL: "http://jamie-test-1.canary.dagster.cloud"
  DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
  ENABLE_FAST_DEPLOYS: 'true'
  PYTHON_VERSION: '3.11'
  DAGSTER_CLOUD_FILE: 'dagster_cloud.yaml'

Use a different base image​

Dagster+ runs your code on a Docker image that we build as follows:

  • The standard Python "slim" Docker image, such as python:3.10-slim is used as the base
  • The dagster-cloud[serverless] module installed in the image

You can add dependencies in your pyproject.toml or setup.py file, but when that is not possible, you can build and upload a custom base image that will be used to run your Python code:

note

Setting a custom base image isn't supported for GitLab CI/CD workflows out of the box, but you can write a custom GitLab CI/CD YAML file that implements the manual steps noted.

  1. Include dagster-cloud and dagster-cloud[serverless] as dependencies in your Docker image by adding the following lines to your Dockerfile:

    RUN pip install "dagster-cloud"
    RUN pip install "dagster-cloud[serverless]"

  2. Build your Docker image, using your usual Docker toolchain.

    note

    If you try to deploy a Docker image to Dagster+ Serverless that was locally built on an M1/M2 Mac it will use ARM architecture and be incompatible with AWS ECS. In this case you must specify Docker builds the image using AMD architecture:

    docker build -t dagster-project:latest --platform=linux/amd64 .

  3. Upload your Docker image to Dagster+ using the upload-base-image command. This command will print out the tag used in Dagster+ to identify your image:

    dagster-cloud serverless upload-base-image local-image:tag

    To use the uploaded image run:

    dagster-cloud serverless deploy-python-executable ... --base-image-tag=sha256_518ad2f92b078c63c60e89f0310f13f19d3a1c7ea9e1976d67d59fcb7040d0d6

  4. Specify this base image tag in you GitHub workflow, or using the dagster-cloud CLI:

In your .github/workflows/deploy.yml file, add the SERVERLESS_BASE_IMAGE_TAG environment variable and set it to the tag printed out in the previous step:

Setting a custom base image in deploy.yml
env:
  DAGSTER_CLOUD_URL: "http://jamie-test-1.canary.dagster.cloud"
  DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
  SERVERLESS_BASE_IMAGE_TAG: "sha256_518ad2f92b078c63c60e89f0310f13f19d3a1c7ea9e1976d67d59fcb7040d0d6"

Include data files​

To add data files to your deployment, create a data directory under the src/{PACKAGE NAME} directory:

quickstart-etl
├── dagster_cloud.yaml
├── pyproject.toml
├── tests
│   ├── __init__.py
│   └── test_defs.py
├── README.md
├── src
│   └── quickstart_etl
│       ├── __init__.py
│       ├── data
│       │   ├── file1.txt
│       │   └── file2.csv
│       ├── definitions.py
│       └── defs
│           └── assets
│               ├── __init__.py
│               ├── hackernews.py
│               └── schedules.py
└── workspace.yaml

Disable PEX deploys​

You have the option to disable PEX-based deploys and deploy using a Docker image instead of PEX. You can disable PEX in your GitHub or GitLab workflow, or by using the dagster-cloud CLI.

In your .github/workflows/deploy.yml file, update the ENABLE_FAST_DEPLOYS environment variable to false:

Disable PEX deploys in deploy.yml
env:
  DAGSTER_CLOUD_URL: "http://jamie-test-1.canary.dagster.cloud"
  DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
  ENABLE_FAST_DEPLOYS: 'false'

You can customize the Docker image using lifecycle hooks or by customizing the base image:

This method is the easiest to set up, and doesn't require setting up any additional infrastructure.

In the root of your repo, you can provide two optional shell scripts: dagster_cloud_pre_install.sh and dagster_cloud_post_install.sh. These will run before and after Python dependencies are installed. They're useful for installing any non-Python dependencies or otherwise configuring your environment.

Use private Python packages​

If you use PEX deploys in your workflow (ENABLE_FAST_DEPLOYS: 'true'), the following steps can install a package from a private GitHub repository, e.g. my-org/private-repo, as a dependency:

  1. In your deploy.yml file, add the following to the top of steps: section in the dagster-cloud-default-deploy job.

    - name: Checkout internal repository
      uses: actions/checkout@v3
      with:
        token: ${{ secrets.GH_PAT }}
        repository: my-org/private-repo
        path: deps/private-repo
        ref: some-branch # optional to check out a specific branch

    - name: Build a wheel
      # adjust the `cd` command to cd into the directory with setup.py
      run: >
        cd deps/private-repo &&
        python setup.py bdist_wheel &&
        mkdir -p $GITHUB_WORKSPACE/deps &&
        cp dist/*whl $GITHUB_WORKSPACE/deps

    # If you have multiple private packages, the above two steps should be repeated for each but the following step is only
    # needed once
    - name: Configure dependency resolution to use the wheel built above
      run: >
        echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&
        echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&
        echo "    file://$GITHUB_WORKSPACE/deps/" >> $GITHUB_WORKSPACE/deps/pip.conf &&
        echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENV

  2. Create a GitHub personal access token and set it as the GH_PAT secret for your Actions.

  3. Add your package to your Dagster project's package configuration file:

In your pyproject.toml file, add your package name to the dependencies section:

Adding private packages to pyproject.toml
dependencies = [
    "dagster",
    "dagster-cloud",
    "private-package", # add this line - must match your private Python package name
]

Once the deploy.yml is updated and changes pushed to your repo, then any subsequent code deploy should checkout your private repository, build the package and install it as a dependency in your Dagster+ project. Repeat the above steps for your branch_deployments.yml if needed.