In this guide, we'll give you everything you need to get up and running with Dagster+, including:
Creating your Dagster+ account
Selecting a deployment type
Deploying your code
Setting up CI/CD
Setting up environment variables and secrets
Let's get started!
Step 1: Create a Dagster+ account and organization#
If you don't already have a Dagster+ account, sign up for one before continuing.
You'll also be asked to create a Dagster+ organization. When your organization is created, a single deployment, named prod, will also be created. Refer to the Managing deployments guide for more information about deployments in Dagster+.
In this step, you'll select the type of deployment you want to use: Serverless or Hybrid.
Type
How it works
May be a good fit if...
Serverless
In a Serverless deployment, Dagster manages the infrastructure needed to run Dagster jobs. This means you don't need to spin up or manage any infrastructure. Refer to the Serverless deployment documentation for more info.
You want to try Dagster+ without a lot of time investment
You don't want to focus on infrastructure management
Your Dagster jobs aren't computationally expensive or memory intensive
Hybrid
In a Hybrid deployment, your Dagster code is executed in your infrastructure. The agent runs in your infrastructure, executes your code, and streams metadata about code execution and assets and jobs (for visualization) over HTTPS back to Dagster+. Refer to the Hybrid deployment documentation for more info.
Dagster+ supports a variety of agents, each with its own prerequisites for setup. Refer to the Agent documentation for more info.
You want to use your own infrastructure
You can create and configure an agent in your infrastructure
Select the deployment type you want to use and proceed to the next step. Note: If you find your deployment type isn't right for you, it can be changed at any time.
Now that you've selected a deployment type, the next step is to tell Dagster+ the location of your Dagster code. In addition to this, you'll set up Continuous Integration (CI) and Continuous Deployment (CD). Once completed, Branch Deployments and continuous code location deployments will be enabled for your Dagster project.
The steps for accomplishing this vary depending on the deployment type you selected in the previous step:
Click the tab for your deployment type - Serverless or Hybrid - to view what's next.
For Serverless deployments, there are two ways to deploy your code to Dagster+:
Start from a template - Use one of our quickstart templates to get up and running. All templates come with CI/CD already configured and will be cloned to a new GitHub repository.
Import an existing project - Import an existing GitHub repository using our GitHub integration or the dagster-cloud CLI. Note: If using the GitHub integration, Dagster+ will automatically set up CI/CD for you.
Using the GitHub integration to import an existing GitHub repository also sets up CI/CD for you.
Before you get started, note that the repository must have a dagster_cloud.yaml file in order for Dagster+ to deploy it. This file defines the code locations in your Dagster code.
If this file doesn't currently exist, create it in the root of your repository with the following code:
# dagster_cloud.yamllocations:-location_name: my_location_name # the name of the code locationcode_source:package_name: hackernews # the name of the python package associated with the code location
After you've committed the file to the repository, come back to Dagster+ to complete the import process:
Click the Import an existing project tab.
Sign in to your GitHub account, if prompted.
In Dagster+, define the following:
Git scope - Select the organization or account that the repository is in.
Using the Gitlab integration to import an existing Gitlab project also sets up CI/CD for you.
Before you get started, note that the project must have a dagster_cloud.yaml file in order for Dagster+ to deploy it. This file defines the code locations in your Dagster code.
If this file doesn't currently exist, create it in the root of your repository with the following code:
# dagster_cloud.yamllocations:-location_name: my_location_name # the name of the code locationcode_source:package_name: hackernews # the name of the python package associated with the code location
After you've committed the file to the project, come back to Dagster+ to complete the import process:
Click the Import an existing project tab.
Sign in to your Gitlab account, if prompted.
In Dagster+, define the following:
Namespace - Select the group or account that the project is in.
Heads up! Using the dagster-cloud CLI requires a recent version of Python 3 and Docker.
To complete this step using the CLI, you can use your own Dagster code or the Dagster starter kit. The starter kit is a template with everything you need to get started using Serverless in Dagster+, including CI/CD configuration and the required dagster_cloud.yaml file.
If not using the template, add dagster-cloud as a dependency in setup.py. Click here for an example. This is already done for you if using the starter kit.
After you've finished setting up your local project, move on to deploying using the CLI:
To install the dagster-cloud CLI, run:
pip install dagster-cloud
Next, you'll need to authenticate to Dagster+. Run the following command and follow the prompts to log in:
dagster-cloud config setup
After you've successfully authenticated, run the following to deploy your code to Dagster+:
dagster-cloud serverless deploy \
--location-name "<LOCATION_NAME>"\# name of the code loation
--package-name "<PACKAGE_NAME>"\# name of the Python package associated with the code location
To set up Hybrid deployment and deploy your code, you'll need to:
Set up an agent
Configure CI/CD for your project. We'll walk you through this in Step 4.
For most Hybrid deployments - with the exception of those using a local agent - you'll need to create a Docker image containing your Dagster code and then add a code location to Dagster+ that references the image.
The dialog that displays will contain a pre-generated agent token and details about the agents currently supported by Dagster+.
Note: Keep this token handy - you'll need it to set up CI/CD in the next step.
Follow the steps for setting up and deploying your agent:
To set up continuous integration using GitHub Actions, you can use your own Dagster code or the Dagster+ Hybrid Quickstart. The quickstart is a template with everything you need to get started using Hybrid deployment in Dagster+.
If not using the template, copy the GitHub workflow file (.github/workflows) from the Hybrid quickstart repository and add it to your repository. This is already done for you if using the quickstart.
Configure the GitHub workflow YAML file as described below. The GitHub workflow deploys your code to Dagster+ using these steps:
Initialize - Check out your code and validate dagster_cloud.yaml.
Docker image push - Build a Docker image from your code and upload it to your container registry.
Deploy to Dagster+ - Update code locations in Dagster+ to use the new Docker image.
To configure the workflow, follow these steps:
In the repository, set the DAGSTER_CLOUD_API_TOKEN GitHub action secret. This is the Dagster+ agent token from the previous section. Refer to the agent tokens documentation for more info.
Refer to the GitHub docs for more info about GitHub Action secrets.
In your dagster-cloud-deploy.yml, set the DAGSTER_CLOUD_ORGANIZATION environment variable to your Dagster+ organization name.
In your dagster-cloud-deploy.yml, uncomment the step that is relevant to your Docker container registry. For example, if using DockerHub, you'd uncomment the DockerHub step in these files. Ensure you have set up the relevant secrets for building and uploading your Docker images.
After making the above changes and commiting the workflow file, the CI process should be triggered to deploy your GitHub repository to Dagster+. During the deployment, the agent will attempt to load your code and update the metadata in Dagster+. Once finished, you should see the GitHub Action complete successfully and also be able to see the code location under the Deployment tag in Dagster+.
For continuous integration using a CI/CD provider other than GitHub, your system should use the dagster-cloud ci subcommand to deploy code locations to Dagster+.
Ensure that you have created a dagster_cloud.yaml file as described in the quickstart. For detailed documentation about this file see the dagster_cloud.yaml reference. The following steps are typically implemented in the CI/CD workflow:
Set the build environment variables. Note that all variables are required:
DAGSTER_CLOUD_ORGANIZATION - The name of your organization in Dagster+.
DAGSTER_CLOUD_API_TOKEN - A Dagster+ API token. Note: This is a sensitive value and should be stored as a CI/CD secret, if possible.
DAGSTER_BUILD_STATEDIR - A path to a blank or non-existent temporary directory on the build machine. This directory is used to store local state during the build.
Run the configuration check:
dagster-cloud ci check --project-dir=.
This is an optional step but useful to validate the contents of your dagster_cloud.yaml and connection to Dagster+.
Initialize the build session:
dagster-cloud ci init --project-dir=.
This reads the dagster_cloud.yaml configuration and initializes the DAGSTER_BUILD_STATEDIR.
Build and upload Docker images for your code locations.
The Docker image should contain a Python environment with dagster, dagster-cloud, and your code. For reference, see the example Dockerfile in our quickstart repository. The example uses pip install . to install the code including the dependencies specified in setup.py.
It is a good idea to use a unique image tag for each Docker build. You can build one image per code location or a shared image for multiple code locations. As an example image tag, you can use the git commit SHA:
exportIMAGE_TAG=`git log --format=format:%H -n 1`
Use this tag to build and upload your Docker image, for example:
The upload step is specific to your Docker container registry and will require authentication. The only requirement is that the registry you upload to must match the registry specified in dagster_cloud.yaml.
Update the build session with the Docker image tag. For each code location you want to deploy, run the following command passing the IMAGE_TAG used in the previous step:
dagster-cloud ci set-build-output --location-name=code-location-a --image-tag=IMAGE_TAG
This command does not deploy the code location but just updates the local state in DAGSTER_BUILD_STATEDIR.
Deploy to Dagster+:
dagster-cloud ci deploy
This command updates the code locations in Dagster+. Once this finishes successfully, you should be able to see the code locations under the Deployments tab in Dagster+.
Congrats! At this point, your Dagster+ deployment should be up and running. To ensure the external services you use in your Dagster project work correctly, start setting up your environment variables. Using environment variables, you can securely pass in sensitive info like passwords, API tokens, etc.