Setting up branch deployments

Dagster+ feature

This feature is only available in Dagster+.

This guide covers setting up branch deployments for a code location using GitHub, GitLab, or an alternative CI platform using the dagster-cloud CLI. Once you've set up branch deployments, any time you create or update a pull request (or merge request) in the repository for your code location, it will automatically create or update an associated branch deployment in Dagster+.

info

Output created from a branch deployment -- such as a database, table, etc. -- won't be automatically removed from storage once a branch is merged or closed. For more information on handling this output, see the best practices section.

Prerequisites

To follow the steps in this guide, you'll need:

• Organization Admin permissions in Dagster+.
• If using a Hybrid deployment: The ability to run a new agent in your infrastructure.

Step 1: Generate a Dagster+ agent token

The first step is to generate a token for the Dagster+ agent. The Dagster+ agent will use this token to authenticate to the agent API.

1. Sign in to your Dagster+ instance.
2. Click the user menu (your profile icon) > Organization Settings.
3. On the Organization Settings page, click the Tokens tab.
4. Click the Create agent token button.
5. After the token has been created, click Reveal token and copy the token.

Keep the token somewhere handy, as you'll need it to complete the setup.

Step 2: Create and configure a branch deployment agent (Hybrid only)

info

If you are using Dagster+ Serverless, you can skip this step.

While you can use your existing production agent for branch deployment on Dagster+ Hybrid, we recommend creating a dedicated branch deployment agent. This ensures that your production instance isn't negatively impacted by the workload associated with branch deployments.

Amazon ECS

1. Deploy an ECS agent to serve your branch deployments. Follow the ECS agent setup guide, making sure to set the Enable branch deployments parameter if using the CloudFormation template. If you are running an existing agent, follow the upgrade guide to ensure your template is up-to-date. Then, turn on the Enable branch deployments parameter.

2. Create a private Amazon Elastic Registry (ECR) repository. Refer to the AWS ECR documentation for instructions.

After the repository has been created, navigate back to the list of ECR repositories.

In the list, locate the repository and its URI:

Keep this around, as you'll need it in a later step.

3. Create an IAM user. This user must:

   • Have push access to the ECR repository, and
   • Have programmatic access to AWS using an access key

   After the user is created, save the Access key ID and Secret access key values shown on the confirmation page:

   

Docker

1. Set up a new Docker agent. For instructions, see the Docker agent setup guide.

2. After the agent is set up, modify the dagster.yaml file as follows:

   • Set the dagster_cloud_api.branch_deployments field to true
   • Remove any deployment field(s)

   For example:

dagster.yaml

instance_class:
  module: dagster_cloud.instance
  class: DagsterCloudAgentInstance

dagster_cloud_api:
  agent_token:
    env: DAGSTER_AGENT_TOKEN
  branch_deployments: true ## true enables branch deployments

user_code_launcher:
  module: dagster_cloud.workspace.docker
  class: DockerUserCodeLauncher
  config:
    networks:
      - dagster_cloud_agent
    server_ttl:
      enabled: true
      ttl_seconds: 7200 #2 hours

Kubernetes

1. Set up a new Kubernetes agent. For instructions, see the Kubernetes agent setup guide.

2. After the agent is set up, modify your Helm values file to include the following:

dagsterCloud:
  branchDeployments: true
  workspace:
    serverTTL:
      enabled: true
      ttlSeconds: 7200

Step 3: Automate branch deployment creation

GitHub

You can set up GitHub to automatically create branch deployments for new pull requests, using GitHub Actions.

This approach may be a good fit if:

• You use GitHub for version control
• You want Dagster to fully automate branch deployments

Step 3.1: Add GitHub CI/CD script to your project

info

If you used the GitHub app in Dagster+ Serverless to configure your repository, you can skip ahead to Step 3.5.

Copy the following files to your project, and replace all references to quickstart_etl with the name of your project:

Dagster+ Serverless

• dagster_cloud.yaml
• .github/workflows/dagster-plus-deploy.yml

Dagster+ Hybrid

• dagster_cloud.yaml
• .github/workflows/dagster-cloud-deploy.yml

Step 3.2: Add the agent registry to dagster_cloud.yaml

info

If you used the GitHub app in Dagster+ Serverless to configure your repository, you can skip ahead to Step 3.5.

In the dagster_cloud.yaml file, replace build.registry with the registry used by the agent you created in step 1.

For example:

dagster_cloud.yaml

locations:
  - location_name: example_location
    code_source:
      python_file: repo.py
    build:
      directory: ./example_location
      registry: 764506304434.dkr.ecr.us-east-1.amazonaws.com/branch-deployment-agent

Step 3.3: Configure GitHub Action secrets

info

If you used the GitHub app in Dagster+ Serverless to configure your repository, you can skip ahead to Step 3.5.

1. In your GitHub repository, click the Settings tab.
2. In the Security section of the sidebar, click Secrets > Actions.
3. Click New repository secret.
4. In the Name field, enter the name of the secret. For example, DAGSTER_CLOUD_API_TOKEN
5. In the Value field, paste the value of the secret.
6. Click Add secret.

Repeat steps 3-6 for each of the secrets required for the registry used by the agent you created in step 1. See below for more details:

Docker

• DAGSTER_CLOUD_API_TOKEN - The Dagster+ agent token you created in step 1
• DAGSTER_CLOUD_URL - Your Dagster+ base URL (https://my_org.dagster.cloud)
• DOCKERHUB_USERNAME - Your DockerHub username
• DOCKERHUB_TOKEN - A DockerHub access token

Amazon ECR

• DAGSTER_CLOUD_API_TOKEN - The Dagster+ agent token you created in step 1
• DAGSTER_CLOUD_URL - Your Dagster+ base URL (https://my_org.dagster.cloud)
• AWS_ACCESS_KEY - The Access key ID of the AWS IAM user you created in step 3
• AWS_SECRET_ACCESS_KEY - The Secret access key of the AWS IAM user you created in step 3
• AWS_REGION - The AWS region where your ECR registry is located

Google Container Registry (GCR)

• DAGSTER_CLOUD_API_TOKEN - The Dagster+ agent token you created in step 1
• DAGSTER_CLOUD_URL - Your Dagster+ base URL (https://my_org.dagster.cloud)
• GCR_JSON_KEY - Your GCR JSON credentials

Step 3.4: Configure GitHub Action

info

If you used the GitHub app in Dagster+ Serverless to configure your repository, you can skip ahead to Step 3.5.

In this step, you'll update the GitHub workflow file in your repository to set up Docker registry access.

Dagster+ Serverless

In the .github/workflows/dagster-plus-deploy.yml file, un-comment the step associated with your registry. For example, for an Amazon ECR registry, you'd un-comment the following portion of the workflow file:

# dagster-plus-deploy.yml
jobs:
  dagster-cloud-deploy:
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}

Save and commit the file to your repository.

Dagster+ Hybrid

In the .github/workflows/dagster-cloud-deploy.yml file, un-comment the step associated with your registry. For example, for an Amazon ECR registry, you'd un-comment the following portion of the workflow file:

# dagster-cloud-deploy.yml
jobs:
  dagster-cloud-deploy:
    steps:
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}

Save and commit the file to your repository.

Step 3.5: Verify GitHub action runs

The last step is to verify that the GitHub Action runs successfully.

1. In the repository, click the Actions tab.
2. In the list of workflows, locate the latest branch deployment run. For example:

GitLab

Step 3.1: add GitLab CI/CD script to your project

You can set up GitLab to automatically create branch deployments for new merge request, using GitLab's CI/CD workflow.

Using this approach to branch deployments may be a good fit if:

• You use GitLab for version control
• You want Dagster to fully automate branch deployments

note

If you used the GitLab app to configure your repository, this step isn't required, you can skip ahead to Step 3.5.

Copy the following files to your project, and replace all references to quickstart-etl with the name of your project:

• dagster_cloud.yaml
• .gitlab-ci.yml (for Hybrid deployments)
• .gitlab-ci.yml (for Serverless deployments)

In the next step, you'll modify these files to work with your Dagster+ setup.

Step 3.2: add the agent registry to dagster_cloud.yaml

note

If you used the GitLab app to configure your repository, this step isn't required, you can skip ahead to Step 3.5.

In the dagster_cloud.yaml file, replace build.registry with the registry used by the agent you created in step 1.

For example:

dagster_cloud.yaml

locations:
  - location_name: example_location
    code_source:
      python_file: repo.py
    build:
      directory: ./example_location
      registry: 764506304434.dkr.ecr.us-east-1.amazonaws.com/branch-deployment-agent

Step 3.3: configure GitLab CI/CD variables

note

If you used the GitLab app to configure your repository, this step isn't required, you can skip ahead to Step 3.5.

1. In your project, click the Settings tab.
2. In the CI/CD section of the sidebar, expand Variables.
3. Click Add variable.
4. In the Key field, enter the name of the variable. For example, DAGSTER_CLOUD_API_TOKEN
5. In the Value field, paste the value of the variable.
6. Click Add variable.

Repeat steps 3-6 for each of the secrets required for your registry type:

GitLab Container Registry

• DAGSTER_CLOUD_API_TOKEN - The Dagster+ agent token you created in step 2
• DAGSTER_CLOUD_URL - Your Dagster+ base URL (https://my_org.dagster.cloud)

Docker

• DAGSTER_CLOUD_API_TOKEN - The Dagster+ agent token you created in step 2
• DAGSTER_CLOUD_URL - Your Dagster+ base URL (https://my_org.dagster.cloud)
• DOCKERHUB_USERNAME - Your DockerHub username
• DOCKERHUB_TOKEN - A DockerHub access token

Amazon ECR

• DAGSTER_CLOUD_API_TOKEN - The Dagster+ agent token you created in step 2
• DAGSTER_CLOUD_URL - Your Dagster+ base URL (https://my_org.dagster.cloud)
• AWS_ACCESS_KEY - The Access key ID of the AWS IAM user you created in step 3
• AWS_SECRET_ACCESS_KEY - The Secret access key of the AWS IAM user you created in step 3
• AWS_REGION - The AWS region where your ECR registry is located

Google Container Registry (GCR)

• DAGSTER_CLOUD_API_TOKEN - The Dagster+ agent token you created in step 2
• DAGSTER_CLOUD_URL - Your Dagster+ base URL (https://my_org.dagster.cloud)
• GCR_JSON_KEY - Your GCR JSON credentials

Step 3.4: configure GitLab CI/CD script

note

If you used the GitLab app to configure your repository, this step isn't required, you can skip ahead to Step 3.5.

In this step, you'll update the GitLab CI/CD config to set up Docker registry access.

In the .gitlab-ci.yml file, un-comment the step associated with your registry. For example, for the GitLab container registry, you'd un-comment the following portion of the .gitlab-ci.yml file:

build-image:
  ...
  before_script:
    # For GitLab Container Registry
    - echo $CI_JOB_TOKEN | docker login --username $CI_REGISTRY_USER --password-stdin $REGISTRY_URL

Save and commit the files to the project.

Step 3.5: verify GitLab pipeline runs

The last step is to verify that the GitLab pipeline runs successfully.

1. On the project page, click the CI/CD tab.
2. In the list of pipelines, locate the latest branch deployment run. For example:

dagster-cloud CLI

You can manually execute dagster-cloud CLI commands to deploy and manage branch deployments. This is a more advanced option than the other methods.

This approach may be a good fit if:

• You don't use GitHub or GitLab for version control
• You use an alternative CI platform
• You want full control over branch deployment configuration

Whenever the state of your branch is updated, Dagster+ expects the following steps to occur:

1. A new image containing your code and requirements is built on the branch. Refer to Managing code locations to learn more.

2. The new image is pushed to a Docker registry accessible to your agent.

The details of how this is accomplished depend on your specific CI/CD solution.

note

The following examples assume the registry URL and image tag are stored in the LOCATION_REGISTRY_URL and IMAGE_TAG environment variables.

Step 3.1 Create a branch deployment associated with the branch

Execute the following command within your CI/CD process:

BRANCH_DEPLOYMENT_NAME=$(
    dagster-cloud branch-deployment create-or-update \
        --organization $ORGANIZATION_NAME \
        --api-token $DAGSTER_CLOUD_API_TOKEN \ # Agent token from step 1
        --git-repo-name $REPOSITORY_NAME \ # Git repository name
        --branch-name $BRANCH_NAME \ # Git branch name
        --commit-hash $COMMIT_SHA \ # Latest commit SHA on the branch
        --timestamp $TIMESTAMP # UTC unixtime timestamp of the latest commit
)

One or more additional parameters can optionally be supplied to the create-or-update command to enhance the branch deployments UI in Dagster+:

BRANCH_DEPLOYMENT_NAME=$(
    dagster-cloud branch-deployment create-or-update \
        --organization $ORGANIZATION_NAME \
        --api-token $DAGSTER_CLOUD_API_TOKEN \
        --git-repo-name $REPOSITORY_NAME \
        --branch-name $BRANCH_NAME \
        --commit-hash $COMMIT_SHA \
        --timestamp $TIMESTAMP
        --code-review-url $PR_URL \ # URL to review the given changes, e.g.
            # Pull Request or Merge Request
        --code-review-id $INPUT_PR \ # Alphanumeric ID for the given set of changes
        --pull-request-status $PR_STATUS \ # A status, one of `OPEN`, `CLOSED`,
            # or `MERGED`, that describes the set of changes
        --commit-message $MESSAGE \ # The message associated with the latest commit
        --author-name $NAME \ # A display name for the latest commit's author
        --author-email $EMAIL \ # An email for the latest commit's author
        --author-avatar-url $AVATAR_URL # An avatar URL for the latest commit's author
        --base-deployment-name $BASE_DEPLOYMENT_NAME # The main deployment that will be compared against. Default is 'prod'
)

If the command is being executed from the context of the git repository, you can alternatively pull this metadata from the repository itself:

BRANCH_DEPLOYMENT_NAME=$(
    dagster-cloud branch-deployment create-or-update \
        --organization $ORGANIZATION_NAME \
        --api-token $DAGSTER_CLOUD_API_TOKEN \
        --git-repo-name $REPOSITORY_NAME \
        --branch-name $BRANCH_NAME \
        --read-git-state # Equivalent to passing --commit-hash, --timestamp
                        # --commit-message, --author-name, --author-email
)

Step 3.2 Deploy your code to the branch deployment

Execute the following command within your CI/CD process:

dagster-cloud deployment add-location \
    --organization $ORGANIZATION_NAME \
    --deployment $BRANCH_DEPLOYMENT_NAME \
    --api-token $DAGSTER_CLOUD_API_TOKEN \
    --location-file $LOCATION_FILE \
    --location-name $LOCATION_NAME \
    --image "${LOCATION_REGISTRY_URL}:${IMAGE_TAG}" \
    --commit-hash "${COMMIT_SHA}" \
    --git-url "${GIT_URL}"

Accessing branch deployments

Once configured, branch deployments can be accessed:

From a GitHub pull request

Every pull request in the repository contains a View in Cloud link, which will open a branch deployment - or a preview of the changes - in Dagster+.

In Dagster+

note

To access a branch deployment in Dagster+, you need permissions that grant you access to branch deployments and the code location associated with the branch deployment.

You can also access branch deployments directly in Dagster+ from the deployment switcher:

Changing the base deployment

The base deployment has two main purposes:

• It sets which full deployment is used to propagate Dagster+ managed environment variables that are scoped for branch deployments.
• It is used in the UI to track changes to the branch deployment from its parent full deployment.

The default base for branch deployments is prod. To configure a different full deployment as the base, create a branch deployment using the dagster-cloud CLI (see step 3.1 above) and specify the deployment with the optional --base-deployment-name parameter.

Best practices

To ensure the best experience when using branch deployments, we recommend:

• Configuring jobs based on environment. Dagster automatically sets environment variables containing deployment metadata, allowing you to parameterize jobs based on the executing environment. Use these variables in your jobs to configure things like connection credentials, databases, and so on. This practice will allow you to use branch deployments without impacting production data.
• Creating jobs to automate output cleanup. As branch deployments don't automatically remove the output they create, you may want to create an additional Dagster job to perform the cleanup.

Next steps

• Learn more about branch deployments
• Learn how to track changes on a branch deployment