Getting started with Dagster Cloud#

Welcome to Dagster Cloud!

In this guide, we'll give you everything you need to get up and running with Dagster Cloud, including:

  • Creating your Dagster Cloud account
  • Selecting a deployment type
  • Deploying your code
  • Setting up CI/CD
  • Setting up environment variables and secrets

Let's get started!

Step 1: Create a Dagster Cloud account and organization#

If you don't already have a Dagster Cloud account, sign up for one before continuing.

You'll also be asked to create a Dagster Cloud organization. When your organization is created, a single deployment, named prod, will also be created. Refer to the Managing deployments guide for more information about deployments in Dagster Cloud.

Step 2: Select a deployment type#

In this step, you'll select the type of deployment you want to use: Serverless or Hybrid.

TypeHow it worksMay be a good fit if...
ServerlessIn a Serverless deployment, Dagster manages the infrastructure needed to run Dagster jobs. This means you don't need to spin up or manage any infrastructure. Refer to the Serverless deployment documentation for more info.
  • You want to try Dagster Cloud without a lot of time investment
  • You don't want to focus on infrastructure management
  • Your Dagster jobs aren't computationally expensive or memory intensive
HybridIn a Hybrid deployment, your Dagster code is executed in your infrastructure. The agent runs in your infrastructure, executes your code, and streams metadata about code execution and assets and jobs (for visualization) over HTTPS back to Dagster Cloud. Refer to the Hybrid deployment documentation for more info.

Dagster Cloud supports a variety of agents, each with its own prerequisites for setup. Refer to the Agent documentation for more info.

Select the deployment type you want to use and proceed to the next step. Note: If you find your deployment type isn't right for you, it can be changed at any time.

Step 3: Deploy your code#

Now that you've selected a deployment type, the next step is to tell Dagster Cloud the location of your Dagster code. In addition to this, you'll set up Continuous Integration (CI) and Continuous Deployment (CD). Once completed, Branch Deployments and continuous code location deployments will be enabled for your Dagster project.

The steps for accomplishing this vary depending on the deployment type you selected in the previous step:

Click the tab for your deployment type - Serverless or Hybrid - to view what's next.

Step 4: Configure CI/CD for your project#

Skip to the next step if using Serverless. This step is only required for Hybrid deployments.

To finish setting up your Hybrid deployment, you'll configure CI/CD for your Dagster project. How this is accomplished depends on your CI/CD provider:

  • GitHub - If using GitHub, you can use our GitHub Action workflow to set up CI/CD for your project.
  • Another CI/CD provider - If you're not using GitHub, you can configure CI/CD using the dagster-cloud CLI.

GitHub Actions#

To set up continuous integration using GitHub Actions, you can use your own Dagster code or the Dagster Cloud Hybrid Quickstart. The quickstart is a template with everything you need to get started using Hybrid deployment in Dagster Cloud.

  • If using the template, clone the repository.

  • If not using the template, copy the GitHub workflow files (.github/workflows) from the Hybrid quickstart repository and add them to your repository. This is already done for you if using the quickstart.

Configure the GitHub workflow YAML file as described below. The GitHub workflow deploys your code to Dagster Cloud using these steps:

  1. Initialize - Check out your code and validate dagster_cloud.yaml.

  2. Docker image push - Build a Docker image from your code and upload it to your container registry.

  3. Deploy to Dagster Cloud - Update code locations in Dagster Cloud to use the new Docker image.

To configure the workflow, follow these steps:

  1. In the repository, set the DAGSTER_CLOUD_API_TOKEN GitHub action secret. This is the Dagster Cloud agent token from the previous section. Refer to the agent tokens documentation for more info.

    Refer to the GitHub docs for more info about GitHub Action secrets.

  2. In your dagster-cloud-deploy.yml, set the DAGSTER_CLOUD_ORGANIZATION environment variable to your Dagster Cloud organization name.

  3. In your dagster-cloud-deploy.yml, uncomment the step that is relevant to your Docker container registry. For example, if using DockerHub, you'd uncomment the DockerHub step in these files. Ensure you have set up the relevant secrets for building and uploading your Docker images.

After making the above changes and commiting the workflow file, the CI process should be triggered to deploy your GitHub repository to Dagster Cloud. During the deployment, the agent will attempt to load your code and update the metadata in Dagster Cloud. Once finished, you should see the GitHub Action complete successfully and also be able to see the code location under the Deployment tag in Dagster Cloud.

When finished, continue to the next step.

Step 5: Set up environment variables and secrets#

Congrats! At this point, your Dagster Cloud deployment should be up and running. To ensure the external services you use in your Dagster project work correctly, start setting up your environment variables. Using environment variables, you can securely pass in sensitive info like passwords, API tokens, etc.

Next steps#

From here, you can: