Configuring branch deployments with GitHub or GitLab

Dagster+ feature

This feature is only available in Dagster+.

This guide covers setting up branch deployments for a Dagster project in Dagster+ Serverless or Hybrid with GitHub or GitLab. Once you've set up branch deployments, any time you create or update a pull request (or merge request) in your project repository, an associated branch deployment will automatically be created or updated in Dagster+.

note

Output created from a branch deployment -- such as a database, table, etc. -- won't be automatically removed from storage once a branch is merged or closed. For more information on handling this output, see the best practices section.

Dagster+ Serverless

Branch deployments are automatically configured for Serverless deployments when you configure CI/CD. For more information, see the CI/CD configuration guide.

Dagster+ Hybrid

Prerequisites

To follow the steps in this section, you'll need:

• Organization Admin permissions in Dagster+.
• The ability to run a new agent in your infrastructure.

Step 1: Configure CI/CD

Follow the CI/CD configuration guide to set up CI/CD for Dagster+ Hybrid with your preferred Git provider.

Step 2: Create a branch deployment agent or update your production agent config

While you can use your existing production agent for branch deployments on Dagster+ Hybrid, we strongly recommend creating a dedicated branch deployment agent. This ensures that your production instance isn't negatively impacted by the workload associated with branch deployments.

Amazon ECS

Dedicated branch deployment agent (recommended)

1. Deploy an ECS agent to serve your branch deployments.

   • To create a new agent, follow the ECS agent setup guide, making sure to set the Enable branch deployments parameter if using the CloudFormation template.
   • To upgrade an existing agent, follow the upgrade guide to ensure your template is up-to-date. Then, turn on the Enable branch deployments parameter.

2. Create a private Amazon Elastic Registry (ECR) repository. For instructions, see the AWS ECR documentation.

3. After you have created a new ECR repository, navigate back to the list of ECR repositories. In the list, locate the repository and its URI. Copy the URI to a separate text file, as you'll need it in a later step.

4. Create an IAM user. This user must:

   • Have push access to the ECR repository, and
   • Have programmatic access to AWS using an access key

5. After the user is created, save the Access key ID and Secret access key values shown on the confirmation page:

Existing production agent

To update an existing production ECS agent, sign in to the AWS Management Console and modify the Dagster+ CloudFormation stack template with a text editor or AWS Infrastructure Composer. You will need to set EnableBranchDeployments to true:

EnableBranchDeployments: true

Docker

Dedicated branch deployment agent (recommended)

1. Set up a new Docker agent. For instructions, see the Docker agent setup guide.
2. After the agent is set up, modify the dagster.yaml file as follows:
   • Set the dagster_cloud_api.branch_deployments field to true
   • Remove any deployment field(s)

For example:

dagster.yaml

instance_class:
  module: dagster_cloud.instance
  class: DagsterCloudAgentInstance

dagster_cloud_api:
  agent_token:
    env: DAGSTER_AGENT_TOKEN
  branch_deployments: true ## true enables branch deployments

user_code_launcher:
  module: dagster_cloud.workspace.docker
  class: DockerUserCodeLauncher
  config:
    networks:
      - dagster_cloud_agent
    server_ttl:
      enabled: true
      ttl_seconds: 7200 #2 hours

Existing production agent

Modify your production agent's dagster.yaml file to set the dagster_cloud_api.branch_deployments field to true. For example:

dagster.yaml

instance_class:
  module: dagster_cloud.instance
  class: DagsterCloudAgentInstance

dagster_cloud_api:
  agent_token:
    env: DAGSTER_AGENT_TOKEN
  branch_deployments: true ## true enables branch deployments
  deployment: prod

user_code_launcher:
  module: dagster_cloud.workspace.docker
  class: DockerUserCodeLauncher
  config:
    networks:
      - dagster_cloud_agent
    server_ttl:
      enabled: true
      ttl_seconds: 7200 #2 hours

Kubernetes

Dedicated branch deployment agent (recommended)

1. Set up a new Kubernetes agent. For instructions, see the Kubernetes agent setup guide.

2. After the agent is set up, modify your Helm values file to include the following:

dagsterCloud:
  branchDeployments: true
  workspace:
    serverTTL:
      enabled: true
      ttlSeconds: 7200

Existing production agent

Modify your production agent's Helm values file to enable branch deployments:

dagsterCloud:
  branchDeployments: true
  deployments:
    - prod
  workspace:
    serverTTL:
      enabled: true
      ttlSeconds: 7200

Step 3: Update build.yaml with the branch deployment agent

In the build.yaml file, replace build.registry with the registry used by the agent you created in step 2.

For example:

build.yaml

locations:
  - location_name: example_location
    code_source:
      python_file: repo.py
    build:
      directory: ./example_location
      registry: 764506304434.dkr.ecr.us-east-1.amazonaws.com/branch-deployment-agent

note

In older deployments, you may have a dagster_cloud.yaml file instead of a build.yaml file.

Step 4: Add secrets to your Git provider

GitHub

1. In your GitHub repository, click the Settings tab.
2. In the Security section of the sidebar, click Secrets > Actions.
3. Click New repository secret.
4. In the Name field, enter the name of the secret. For example, DAGSTER_CLOUD_URL
5. In the Value field, paste the value of the secret.
6. Click Add secret.

Repeat steps 3-6 for each of the secrets required for the registry used by the agent you created in step 1. See below for more details:

Docker

• DAGSTER_CLOUD_URL - Your Dagster+ base URL (https://my_org.dagster.cloud)
• DOCKERHUB_USERNAME - Your DockerHub username
• DOCKERHUB_TOKEN - A DockerHub access token

Amazon ECR

• DAGSTER_CLOUD_URL - Your Dagster+ base URL (https://my_org.dagster.cloud)
• AWS_ACCESS_KEY - The Access key ID of the AWS IAM user you created in step 3
• AWS_SECRET_ACCESS_KEY - The Secret access key of the AWS IAM user you created in step 3
• AWS_REGION - The AWS region where your ECR registry is located

Google Container Registry (GCR)

• DAGSTER_CLOUD_URL - Your Dagster+ base URL (https://my_org.dagster.cloud)
• GCR_JSON_KEY - Your GCR JSON credentials

GitLab

1. In your GitLab repository, click Settings > CI/CD.
2. On the settings page, click Variables.
3. Under Project variables, click Add variable.
4. In the Key field, enter the name of the secret. For example, DAGSTER_CLOUD_URL.
5. In the Value field, paste the value of the secret.
6. Update the type, environments, visibility, flags, and description fields as needed.
7. Click Add variable.

Repeat steps 3-6 for each of the secrets required for the registry used by the agent you created in step 1. See below for more details:

Docker

• DAGSTER_CLOUD_URL - Your Dagster+ base URL (https://my_org.dagster.cloud)
• DOCKERHUB_USERNAME - Your DockerHub username
• DOCKERHUB_TOKEN - A DockerHub access token

Amazon ECR

• DAGSTER_CLOUD_URL - Your Dagster+ base URL (https://my_org.dagster.cloud)
• AWS_ACCESS_KEY - The Access key ID of the AWS IAM user you created in step 3
• AWS_SECRET_ACCESS_KEY - The Secret access key of the AWS IAM user you created in step 3
• AWS_REGION - The AWS region where your ECR registry is located

Google Container Registry (GCR)

• DAGSTER_CLOUD_URL - Your Dagster+ base URL (https://my_org.dagster.cloud)
• GCR_JSON_KEY - Your GCR JSON credentials

Accessing branch deployments

Once configured, branch deployments can be accessed:

From a GitHub pull request

Every pull request in the repository contains a View in Cloud link, which will open a branch deployment - or a preview of the changes - in Dagster+.

In Dagster+

note

To access a branch deployment in Dagster+, you need permissions that grant you access to branch deployments and the code location associated with the branch deployment.

You can also access branch deployments directly in Dagster+ from the deployment switcher:

Changing the base deployment

The base deployment has two main purposes:

• It sets which full deployment is used to propagate Dagster+ managed environment variables that are scoped for branch deployments.
• It is used in the UI to track changes to the branch deployment from its parent full deployment.

The default base for branch deployments is prod. To configure a different full deployment as the base, follow the steps in Configuring branch deployments with the dagster-cloud CLI to create a branch deployment using the dagster-cloud CLI, and specify the deployment with the optional --base-deployment-name parameter.

Best practices

To ensure the best experience when using branch deployments, we recommend:

• Configuring jobs based on environment. Dagster automatically sets environment variables containing deployment metadata, allowing you to parameterize jobs based on the executing environment. Use these variables in your jobs to configure things like connection credentials, databases, and so on. This practice will allow you to use branch deployments without impacting production data.
• Creating jobs to automate output cleanup. As branch deployments don't automatically remove the output they create, you may want to create an additional Dagster job to perform the cleanup.