Set up CI/CD with GitHub Actions
With agents running in Kubernetes, the final piece is a CI/CD pipeline that builds your Docker image and tells Dagster+ to use it. All of the sections below are part of a single workflow file: .github/workflows/dagster-cloud-deploy.yml.
The workflow handles three scenarios:
- Push to
main,staging, ordev: Deploy to the corresponding long-lived environment - Pull request opened or updated: Create or update a branch deployment
- Pull request closed: Tear down the branch deployment
Step 1: Configure required secrets and variables
In your GitHub repository settings, add:
Secrets (Settings → Secrets and variables → Actions → Secrets):
| Name | Value |
|---|---|
DAGSTER_CLOUD_API_TOKEN | CI token from dg plus create ci-api-token |
REGISTRY_USERNAME | Container registry username |
REGISTRY_PASSWORD | Container registry password or token |
Variables (Settings → Secrets and variables → Actions → Variables):
| Name | Value |
|---|---|
DAGSTER_CLOUD_ORGANIZATION | Your Dagster+ org name (the subdomain in <org>.dagster.cloud) |
CONTAINER_REGISTRY | Registry prefix, e.g. ghcr.io/your-org or us-docker.pkg.dev/project/repo |
Step 2: Create the workflow
Triggers and concurrency
The workflow fires on pushes to the three long-lived branches and on all pull request lifecycle events. The concurrency block ensures only one deploy runs per branch or PR at a time — if a new push arrives while a deploy is in progress, GitHub cancels the older run:
on:
push:
branches:
- main
- staging
- dev
pull_request:
types: [opened, synchronize, reopened, closed]
concurrency:
group: dagster-deploy-${{ github.event.pull_request.number || github.ref_name }}
cancel-in-progress: true
env:
DAGSTER_CLOUD_URL: "https://${{ vars.DAGSTER_CLOUD_ORGANIZATION }}.dagster.cloud"
Step 2.1: Add configure job
This job runs first and determines where to deploy. It maps branch names to Dagster+ deployment names and computes an immutable image tag from the commit SHA. All downstream jobs read from its outputs, so the branch-to-deployment mapping lives in one place:
configure:
runs-on: ubuntu-latest
outputs:
deployment: ${{ steps.target.outputs.deployment }}
is_branch_deploy: ${{ steps.target.outputs.is_branch_deploy }}
image_tag: ${{ steps.tag.outputs.image_tag }}
steps:
- name: Determine deployment target
id: target
run: |
if [ "${{ github.event_name }}" = "pull_request" ]; then
echo "deployment=branch" >> "$GITHUB_OUTPUT"
echo "is_branch_deploy=true" >> "$GITHUB_OUTPUT"
else
case "${{ github.ref_name }}" in
main) echo "deployment=prod" >> "$GITHUB_OUTPUT" ;;
staging) echo "deployment=staging" >> "$GITHUB_OUTPUT" ;;
dev) echo "deployment=dev" >> "$GITHUB_OUTPUT" ;;
esac
echo "is_branch_deploy=false" >> "$GITHUB_OUTPUT"
fi
- name: Compute image tag
id: tag
run: |
echo "image_tag=${{ github.sha }}" >> "$GITHUB_OUTPUT"
Step 2.2: Add build-and-push job
This job builds the Docker image and pushes two tags: the commit SHA (immutable, used by the deploy jobs), and the branch name (mutable convenience pointer). GitHub Actions layer caching makes rebuilds fast when only source files change. This job is skipped entirely when a PR is closed, since there's nothing to build:
build-and-push:
runs-on: ubuntu-latest
needs: configure
if: github.event.action != 'closed'
steps:
- uses: actions/checkout@v4
- name: Log in to container registry
uses: docker/login-action@v3
with:
registry: ${{ vars.CONTAINER_REGISTRY }}
username: ${{ secrets.REGISTRY_USERNAME }}
password: ${{ secrets.REGISTRY_PASSWORD }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build and push user code image
uses: docker/build-push-action@v6
with:
context: .
push: true
tags: |
${{ vars.CONTAINER_REGISTRY }}/dagster-deploy-demo:${{ needs.configure.outputs.image_tag }}
${{ vars.CONTAINER_REGISTRY }}/dagster-deploy-demo:${{ github.ref_name }}
cache-from: type=gha
cache-to: type=gha,mode=max
Step 2.3: Add deploy-full job
This job runs on pushes to main, staging, or dev. It calls dagster-cloud deployment code-location update to point the target deployment at the newly built image, recording the commit SHA and Git URL for traceability in the Dagster+ UI:
deploy-full:
runs-on: ubuntu-latest
needs: [configure, build-and-push]
if: needs.configure.outputs.is_branch_deploy == 'false'
steps:
- uses: actions/checkout@v4
- name: Install dagster-cloud CLI
run: pip install dagster-cloud-cli
- name: Update code location image in Dagster+
run: |
dagster-cloud deployment code-location update \
--url "${{ env.DAGSTER_CLOUD_URL }}" \
--api-token "${{ secrets.DAGSTER_CLOUD_API_TOKEN }}" \
--deployment "${{ needs.configure.outputs.deployment }}" \
--location-name dagster_deploy_demo \
--image "${{ vars.CONTAINER_REGISTRY }}/dagster-deploy-demo:${{ needs.configure.outputs.image_tag }}" \
--location-file dagster_cloud.yaml \
--git-url "${{ github.server_url }}/${{ github.repository }}/tree/${{ github.sha }}" \
--commit-hash "${{ github.sha }}"
Step 2.4: Add deploy-branch job
This job handles pull requests. On open or sync, it creates or updates a branch deployment named pr-<number>, then points its code location at the new image. On PR close, it deletes the branch deployment. The always() condition ensures cleanup runs even when build-and-push was skipped:
deploy-branch:
runs-on: ubuntu-latest
needs: [configure, build-and-push]
if: |
always()
&& needs.configure.outputs.is_branch_deploy == 'true'
steps:
- uses: actions/checkout@v4
- name: Install dagster-cloud CLI
run: pip install dagster-cloud-cli
- name: Close branch deployment
if: github.event.action == 'closed'
run: |
dagster-cloud deployment branch-deployment delete \
--url "${{ env.DAGSTER_CLOUD_URL }}" \
--api-token "${{ secrets.DAGSTER_CLOUD_API_TOKEN }}" \
--branch-deployment-name "pr-${{ github.event.pull_request.number }}"
- name: Create or update branch deployment
if: github.event.action != 'closed'
run: |
dagster-cloud deployment branch-deployment create-or-update \
--url "${{ env.DAGSTER_CLOUD_URL }}" \
--api-token "${{ secrets.DAGSTER_CLOUD_API_TOKEN }}" \
--branch-deployment-name "pr-${{ github.event.pull_request.number }}" \
--branch-name "${{ github.head_ref }}" \
--commit-hash "${{ github.sha }}" \
--base-deployment-name "dev"
dagster-cloud deployment code-location update \
--url "${{ env.DAGSTER_CLOUD_URL }}" \
--api-token "${{ secrets.DAGSTER_CLOUD_API_TOKEN }}" \
--deployment "pr-${{ github.event.pull_request.number }}" \
--location-name dagster_deploy_demo \
--image "${{ vars.CONTAINER_REGISTRY }}/dagster-deploy-demo:${{ needs.configure.outputs.image_tag }}" \
--location-file dagster_cloud.yaml \
--git-url "${{ github.server_url }}/${{ github.repository }}/tree/${{ github.sha }}" \
--commit-hash "${{ github.sha }}"
Step 3: Verify the pipeline
Push a commit to your dev branch and watch the workflow run in the Actions tab. After it completes:
- Open the Dagster+ UI and navigate to your
devdeployment. - Check Deployment → Code Locations — the image tag should match the commit SHA.
- Open a pull request and confirm a new branch deployment appears in Dagster+.
- Close the PR and confirm the branch deployment is removed.
Summary
You now have a complete Dagster+ deployment on Kubernetes:
- Assets that read
DAGSTER_CLOUD_DEPLOYMENT_NAMEto adapt to their environment - A multi-stage Docker image built with
uvand cached in GitHub Actions - Three Helm-managed agents — one per environment, isolated in their own namespaces, with resource limits and TTLs appropriate for each tier
- A GitHub Actions workflow that maps branches to deployments and manages branch deploy lifecycle automatically
Typical developer workflow
Once CI/CD is set up, the day-to-day flow for shipping a change looks like this:
-
Create a feature branch off
dev:git checkout dev
git checkout -b feature/my-change -
Develop locally with
dg dev, write tests, iterate. -
Open a PR against
dev— GitHub Actions builds the image and creates a branch deployment namedpr-<number>. You can test your pipeline changes in the Dagster+ UI under that branch deployment before merging. -
Merge to
dev— CI deploys to thedevdeployment. Run integration tests against dev data. -
Promote to staging: merge
dev→staging. CI deploys tostagingfor pre-production validation. -
Promote to prod: merge
staging→main. CI deploys toprod.