Ask AI

Using Branch Deployments (CI) with GitHub Actions#

This guide is applicable to Dagster Cloud.

In this guide, we'll walk you through setting up Continuous Integration (CI) using Branch Deployments with GitHub Actions.

Using this approach to branch deployments may be a good fit if:

  • You use GitHub for version control
  • You want Dagster to fully automate Branch Deployments

If you don't use GitHub for version control or want full control over your setup, check out the Branch deployments with the dagster-cloud CLI guide.


Prerequisites#

To complete the steps in this guide, you'll need:

  • Organization Admin permissions in Dagster Cloud
  • The ability to run a new agent in your infrastructure. This isn't required if you're using Serverless deployment.
  • The ability to configure GitHub Actions for your repository. This isn't required if you used the Dagster Cloud GitHub app to connect your repository as a code location.

Step 1: Generate a Dagster Cloud agent token#

In this step, you'll generate a token for the Dagster Cloud agent. The Dagster Cloud agent will use this to authenticate to the agent API.

  1. Sign in to your Dagster Cloud instance.
  2. Click the user menu (your icon) > Organization Settings.
  3. In the Organization Settings page, click the Tokens tab.
  4. Click the + Create agent token button.
  5. After the token has been created, click Reveal token.

Keep the token somewhere handy - you'll need it to complete the setup.


Step 2: Create and configure an agent#

If using Serverless deployment, this step isn't required.

While you can use your existing production agent, we recommend creating a dedicated branch deployment agent. This ensures that your production instance isn't negatively impacted by the workload associated with branch deployments.

Using the tabs, select your agent type to view instructions.


Step 3: Add GitHub workflow files to your repository#

If you used the GitHub app to add the repository as a code location, this step isn't required. Skip to the next step.

In this step, you'll add the required GitHub workflow files to your GitHub repository. The files can be found in our Hybrid quickstart repository or Serverless quickstart repository, depending on the agent you are using.

Copy the following files to your repository (the linked files are shown in the Hybrid repo, there are equivalents in the Serverless repo).

In the next step, you'll modify these files to work with your Dagster Cloud setup.


Step 4: Configure the GitHub repository#

In this section, you'll:

  1. Add the agent registry to dagster_cloud.yaml
  2. Configure GitHub Action secrets
  3. Update GitHub Workflow files
  4. Verify Action runs

Step 4.1: Add the agent registry to dagster_cloud.yaml#

If you're using Serverless deployment, this step isn't required. Skip to the next step.

In the dagster_cloud.yaml file, replace build.registry with the registry used by the agent you created in Step 2.

For example:

# dagster_cloud.yaml

locations:
  - location_name: example_location
    code_source:
      python_file: repo.py
    build:
      directory: ./example_location
      registry: 764506304434.dkr.ecr.us-east-1.amazonaws.com/branch-deployment-agent

Step 4.2: Configure GitHub Action secrets#

If you used the GitHub app to add the repository as a code location, this step isn't required.

Want to use secrets in your Dagster code? Check out the Dagster Cloud environment variables and secrets guide for more info.
  1. In the repository, click the Settings tab.
  2. In the Security section of the sidebar, click Secrets > Actions.
  3. Click New repository secret.
  4. In the Name field, enter the name of the secret. For example, API_TOKEN
  5. In the Value field, paste the value of the secret.
  6. Click Add secret.

For Hybrid deployments, repeat steps 3-6 for each of the secrets required for your registry type:

Select the tab for your registry type to view instructions.

Step 4.3: Update GitHub Workflow files#

If you're using Serverless deployment, this step isn't required. Skip to the next step.

In this step, you'll update the GitHub Workflow files in the repository to set up Docker registry access.

In the .github/workflows/deploy.yml and .github/workflows/branch_deployments.yml files, uncomment the step associated with your registry. For example, for an Amazon ECR registry, you'd uncomment the following portion of the workflow files:

steps:
  - name: Configure AWS credentials
    uses: aws-actions/configure-aws-credentials@v1
    with:
      aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
      aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      aws-region: ${{ secrets.AWS_REGION }}

Save and commit the files to the repository.

Step 4.4: Verify Action runs#

The last step is to verify that the Action runs successfully.

  1. In the repository, click the Actions tab.

  2. In the list of workflows, locate the latest branch deployment run. For example:

    A successful run for a Branch Deployment Action

If the run finished successfully, that's it!


Step 5: Access the branch deployment#

Now that Branch Deployments are configured, you can check out the preview in Dagster Cloud. There are two ways to do this:

From a pull request#

Every pull request in the repository contains a View in Cloud link:

View in Cloud preview link highlighted in a GitHub pull request

Clicking the link will open a branch deployment - or a preview of the changes - in Dagster Cloud.

Directly in Dagster Cloud#

You can also access branch deployments directly in Dagster Cloud from the deployment switcher:

Highlighted branch deployment in the Dagster Cloud deployment switcher