Skip to main content

Set up CI/CD with GitHub Actions

With agents running in Kubernetes, the final piece is a CI/CD pipeline that builds your Docker image and tells Dagster+ to use it. All of the sections below are part of a single workflow file: .github/workflows/dagster-cloud-deploy.yml.

The workflow handles three scenarios:

  • Push to main, staging, or dev: Deploy to the corresponding long-lived environment
  • Pull request opened or updated: Create or update a branch deployment
  • Pull request closed: Tear down the branch deployment

Step 1: Configure required secrets and variables

In your GitHub repository settings, add:

Secrets (Settings → Secrets and variables → Actions → Secrets):

NameValue
DAGSTER_CLOUD_API_TOKENCI token from dg plus create ci-api-token
REGISTRY_USERNAMEContainer registry username
REGISTRY_PASSWORDContainer registry password or token

Variables (Settings → Secrets and variables → Actions → Variables):

NameValue
DAGSTER_CLOUD_ORGANIZATIONYour Dagster+ org name (the subdomain in <org>.dagster.cloud)
CONTAINER_REGISTRYRegistry prefix, e.g. ghcr.io/your-org or us-docker.pkg.dev/project/repo

Step 2: Create the workflow

Triggers and concurrency

The workflow fires on pushes to the three long-lived branches and on all pull request lifecycle events. The concurrency block ensures only one deploy runs per branch or PR at a time — if a new push arrives while a deploy is in progress, GitHub cancels the older run:

.github/workflows/dagster-cloud-deploy.yml
on:
push:
branches:
- main
- staging
- dev
pull_request:
types: [opened, synchronize, reopened, closed]

concurrency:
group: dagster-deploy-${{ github.event.pull_request.number || github.ref_name }}
cancel-in-progress: true

env:
DAGSTER_CLOUD_URL: "https://${{ vars.DAGSTER_CLOUD_ORGANIZATION }}.dagster.cloud"

Step 2.1: Add configure job

This job runs first and determines where to deploy. It maps branch names to Dagster+ deployment names and computes an immutable image tag from the commit SHA. All downstream jobs read from its outputs, so the branch-to-deployment mapping lives in one place:

.github/workflows/dagster-cloud-deploy.yml
  configure:
runs-on: ubuntu-latest
outputs:
deployment: ${{ steps.target.outputs.deployment }}
is_branch_deploy: ${{ steps.target.outputs.is_branch_deploy }}
image_tag: ${{ steps.tag.outputs.image_tag }}
steps:
- name: Determine deployment target
id: target
run: |
if [ "${{ github.event_name }}" = "pull_request" ]; then
echo "deployment=branch" >> "$GITHUB_OUTPUT"
echo "is_branch_deploy=true" >> "$GITHUB_OUTPUT"
else
case "${{ github.ref_name }}" in
main) echo "deployment=prod" >> "$GITHUB_OUTPUT" ;;
staging) echo "deployment=staging" >> "$GITHUB_OUTPUT" ;;
dev) echo "deployment=dev" >> "$GITHUB_OUTPUT" ;;
esac
echo "is_branch_deploy=false" >> "$GITHUB_OUTPUT"
fi

- name: Compute image tag
id: tag
run: |
echo "image_tag=${{ github.sha }}" >> "$GITHUB_OUTPUT"

Step 2.2: Add build-and-push job

This job builds the Docker image and pushes two tags: the commit SHA (immutable, used by the deploy jobs), and the branch name (mutable convenience pointer). GitHub Actions layer caching makes rebuilds fast when only source files change. This job is skipped entirely when a PR is closed, since there's nothing to build:

.github/workflows/dagster-cloud-deploy.yml
  build-and-push:
runs-on: ubuntu-latest
needs: configure
if: github.event.action != 'closed'
steps:
- uses: actions/checkout@v4

- name: Log in to container registry
uses: docker/login-action@v3
with:
registry: ${{ vars.CONTAINER_REGISTRY }}
username: ${{ secrets.REGISTRY_USERNAME }}
password: ${{ secrets.REGISTRY_PASSWORD }}

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Build and push user code image
uses: docker/build-push-action@v6
with:
context: .
push: true
tags: |
${{ vars.CONTAINER_REGISTRY }}/dagster-deploy-demo:${{ needs.configure.outputs.image_tag }}
${{ vars.CONTAINER_REGISTRY }}/dagster-deploy-demo:${{ github.ref_name }}
cache-from: type=gha
cache-to: type=gha,mode=max

Step 2.3: Add deploy-full job

This job runs on pushes to main, staging, or dev. It calls dagster-cloud deployment code-location update to point the target deployment at the newly built image, recording the commit SHA and Git URL for traceability in the Dagster+ UI:

.github/workflows/dagster-cloud-deploy.yml
  deploy-full:
runs-on: ubuntu-latest
needs: [configure, build-and-push]
if: needs.configure.outputs.is_branch_deploy == 'false'
steps:
- uses: actions/checkout@v4

- name: Install dagster-cloud CLI
run: pip install dagster-cloud-cli

- name: Update code location image in Dagster+
run: |
dagster-cloud deployment code-location update \
--url "${{ env.DAGSTER_CLOUD_URL }}" \
--api-token "${{ secrets.DAGSTER_CLOUD_API_TOKEN }}" \
--deployment "${{ needs.configure.outputs.deployment }}" \
--location-name dagster_deploy_demo \
--image "${{ vars.CONTAINER_REGISTRY }}/dagster-deploy-demo:${{ needs.configure.outputs.image_tag }}" \
--location-file dagster_cloud.yaml \
--git-url "${{ github.server_url }}/${{ github.repository }}/tree/${{ github.sha }}" \
--commit-hash "${{ github.sha }}"

Step 2.4: Add deploy-branch job

This job handles pull requests. On open or sync, it creates or updates a branch deployment named pr-<number>, then points its code location at the new image. On PR close, it deletes the branch deployment. The always() condition ensures cleanup runs even when build-and-push was skipped:

.github/workflows/dagster-cloud-deploy.yml
  deploy-branch:
runs-on: ubuntu-latest
needs: [configure, build-and-push]
if: |
always()
&& needs.configure.outputs.is_branch_deploy == 'true'
steps:
- uses: actions/checkout@v4

- name: Install dagster-cloud CLI
run: pip install dagster-cloud-cli

- name: Close branch deployment
if: github.event.action == 'closed'
run: |
dagster-cloud deployment branch-deployment delete \
--url "${{ env.DAGSTER_CLOUD_URL }}" \
--api-token "${{ secrets.DAGSTER_CLOUD_API_TOKEN }}" \
--branch-deployment-name "pr-${{ github.event.pull_request.number }}"

- name: Create or update branch deployment
if: github.event.action != 'closed'
run: |
dagster-cloud deployment branch-deployment create-or-update \
--url "${{ env.DAGSTER_CLOUD_URL }}" \
--api-token "${{ secrets.DAGSTER_CLOUD_API_TOKEN }}" \
--branch-deployment-name "pr-${{ github.event.pull_request.number }}" \
--branch-name "${{ github.head_ref }}" \
--commit-hash "${{ github.sha }}" \
--base-deployment-name "dev"

dagster-cloud deployment code-location update \
--url "${{ env.DAGSTER_CLOUD_URL }}" \
--api-token "${{ secrets.DAGSTER_CLOUD_API_TOKEN }}" \
--deployment "pr-${{ github.event.pull_request.number }}" \
--location-name dagster_deploy_demo \
--image "${{ vars.CONTAINER_REGISTRY }}/dagster-deploy-demo:${{ needs.configure.outputs.image_tag }}" \
--location-file dagster_cloud.yaml \
--git-url "${{ github.server_url }}/${{ github.repository }}/tree/${{ github.sha }}" \
--commit-hash "${{ github.sha }}"

Step 3: Verify the pipeline

Push a commit to your dev branch and watch the workflow run in the Actions tab. After it completes:

  1. Open the Dagster+ UI and navigate to your dev deployment.
  2. Check Deployment → Code Locations — the image tag should match the commit SHA.
  3. Open a pull request and confirm a new branch deployment appears in Dagster+.
  4. Close the PR and confirm the branch deployment is removed.

Summary

You now have a complete Dagster+ deployment on Kubernetes:

  • Assets that read DAGSTER_CLOUD_DEPLOYMENT_NAME to adapt to their environment
  • A multi-stage Docker image built with uv and cached in GitHub Actions
  • Three Helm-managed agents — one per environment, isolated in their own namespaces, with resource limits and TTLs appropriate for each tier
  • A GitHub Actions workflow that maps branches to deployments and manages branch deploy lifecycle automatically

Typical developer workflow

Once CI/CD is set up, the day-to-day flow for shipping a change looks like this:

  1. Create a feature branch off dev:

    git checkout dev
    git checkout -b feature/my-change
  2. Develop locally with dg dev, write tests, iterate.

  3. Open a PR against dev — GitHub Actions builds the image and creates a branch deployment named pr-<number>. You can test your pipeline changes in the Dagster+ UI under that branch deployment before merging.

  4. Merge to dev — CI deploys to the dev deployment. Run integration tests against dev data.

  5. Promote to staging: merge devstaging. CI deploys to staging for pre-production validation.

  6. Promote to prod: merge stagingmain. CI deploys to prod.