Welcome to Dagster Cloud!
In this guide, we'll give you everything you need to get up and running with Dagster Cloud, including:
Let's get started!
If you don't already have a Dagster Cloud account, sign up for one before continuing.
You'll also be asked to create a Dagster Cloud organization. When your organization is created, a single deployment, named
prod, will also be created. Refer to the Managing deployments guide for more information about deployments in Dagster Cloud.
In this step, you'll select the type of deployment you want to use: Serverless or Hybrid.
|How it works
|May be a good fit if...
|In a Serverless deployment, Dagster manages the infrastructure needed to run Dagster jobs. This means you don't need to spin up or manage any infrastructure. Refer to the Serverless deployment documentation for more info.
|In a Hybrid deployment, your Dagster code is executed in your infrastructure. The agent runs in your infrastructure, executes your code, and streams metadata about code execution and assets and jobs (for visualization) over HTTPS back to Dagster Cloud. Refer to the Hybrid deployment documentation for more info.
Dagster Cloud supports a variety of agents, each with its own prerequisites for setup. Refer to the Agent documentation for more info.
Select the deployment type you want to use and proceed to the next step. Note: If you find your deployment type isn't right for you, it can be changed at any time.
Now that you've selected a deployment type, the next step is to tell Dagster Cloud the location of your Dagster code. In addition to this, you'll set up Continuous Integration (CI) and Continuous Deployment (CD). Once completed, Branch Deployments and continuous code location deployments will be enabled for your Dagster project.
The steps for accomplishing this vary depending on the deployment type you selected in the previous step:
To finish setting up your Hybrid deployment, you'll configure CI/CD for your Dagster project. How this is accomplished depends on your CI/CD provider:
To set up continuous integration using GitHub Actions, you can use your own Dagster code or the Dagster Cloud Hybrid Quickstart. The quickstart is a template with everything you need to get started using Hybrid deployment in Dagster Cloud.
If using the template, clone the repository.
If not using the template, copy the GitHub workflow files (
.github/workflows) from the Hybrid quickstart repository and add them to your repository. This is already done for you if using the quickstart.
Configure the GitHub workflow YAML file as described below. The GitHub workflow deploys your code to Dagster Cloud using these steps:
Initialize - Check out your code and validate
Docker image push - Build a Docker image from your code and upload it to your container registry.
Deploy to Dagster Cloud - Update code locations in Dagster Cloud to use the new Docker image.
To configure the workflow, follow these steps:
In the repository, set the
DAGSTER_CLOUD_API_TOKEN GitHub action secret. This is the Dagster Cloud agent token from the previous section. Refer to the agent tokens documentation for more info.
Refer to the GitHub docs for more info about GitHub Action secrets.
dagster-cloud-deploy.yml, set the
DAGSTER_CLOUD_ORGANIZATION environment variable to your Dagster Cloud organization name.
dagster-cloud-deploy.yml, uncomment the step that is relevant to your Docker container registry. For example, if using DockerHub, you'd uncomment the
DockerHub step in these files. Ensure you have set up the relevant secrets for building and uploading your Docker images.
After making the above changes and commiting the workflow file, the CI process should be triggered to deploy your GitHub repository to Dagster Cloud. During the deployment, the agent will attempt to load your code and update the metadata in Dagster Cloud. Once finished, you should see the GitHub Action complete successfully and also be able to see the code location under the Deployment tag in Dagster Cloud.
When finished, continue to the next step.
Congrats! At this point, your Dagster Cloud deployment should be up and running. To ensure the external services you use in your Dagster project work correctly, start setting up your environment variables. Using environment variables, you can securely pass in sensitive info like passwords, API tokens, etc.
From here, you can: