Ask AI

Getting started with Dagster+#

Welcome to Dagster+!

In this guide, we'll give you everything you need to get up and running with Dagster+, including:

  • Creating your Dagster+ account
  • Selecting a deployment type
  • Deploying your code
  • Setting up CI/CD
  • Setting up environment variables and secrets

Let's get started!

Step 1: Create a Dagster+ account and organization#

If you don't already have a Dagster+ account, sign up for one before continuing.

You'll also be asked to create a Dagster+ organization. When your organization is created, a single deployment, named prod, will also be created. Refer to the Managing deployments guide for more information about deployments in Dagster+.

Step 2: Select a deployment type#

In this step, you'll select the type of deployment you want to use: Serverless or Hybrid.

TypeHow it worksMay be a good fit if...
ServerlessIn a Serverless deployment, Dagster manages the infrastructure needed to run Dagster jobs. This means you don't need to spin up or manage any infrastructure. Refer to the Serverless deployment documentation for more info.
  • You want to try Dagster+ without a lot of time investment
  • You don't want to focus on infrastructure management
  • Your Dagster jobs aren't computationally expensive or memory intensive
HybridIn a Hybrid deployment, your Dagster code is executed in your infrastructure. The agent runs in your infrastructure, executes your code, and streams metadata about code execution and assets and jobs (for visualization) over HTTPS back to Dagster+. Refer to the Hybrid deployment documentation for more info.

Dagster+ supports a variety of agents, each with its own prerequisites for setup. Refer to the Agent documentation for more info.

Select the deployment type you want to use and proceed to the next step. Note: If you find your deployment type isn't right for you, it can be changed at any time.

Step 3: Deploy your code#

Now that you've selected a deployment type, the next step is to tell Dagster+ the location of your Dagster code. In addition to this, you'll set up Continuous Integration (CI) and Continuous Deployment (CD). Once completed, Branch Deployments and continuous code location deployments will be enabled for your Dagster project.

The steps for accomplishing this vary depending on the deployment type you selected in the previous step:

Click the tab for your deployment type - Serverless or Hybrid - to view what's next.

Step 4: Configure CI/CD for your project#

Skip to the next step if using Serverless. This step is only required for Hybrid deployments.

To finish setting up your Hybrid deployment, you'll configure CI/CD for your Dagster project. How this is accomplished depends on your CI/CD provider:

  • GitHub - If using GitHub, you can use our GitHub Action workflow to set up CI/CD for your project.
  • Another CI/CD provider - If you're not using GitHub, you can configure CI/CD using the dagster-cloud CLI.

GitHub Actions#

To set up continuous integration using GitHub Actions, you can use your own Dagster code or the Dagster+ Hybrid Quickstart. The quickstart is a template with everything you need to get started using Hybrid deployment in Dagster+.

  • If using the template, clone the repository.

  • If not using the template, copy the GitHub workflow file (.github/workflows) from the Hybrid quickstart repository and add it to your repository. This is already done for you if using the quickstart.

Configure the GitHub workflow YAML file as described below. The GitHub workflow deploys your code to Dagster+ using these steps:

  1. Initialize - Check out your code and validate dagster_cloud.yaml.

  2. Docker image push - Build a Docker image from your code and upload it to your container registry.

  3. Deploy to Dagster+ - Update code locations in Dagster+ to use the new Docker image.

To configure the workflow, follow these steps:

  1. In the repository, set the DAGSTER_CLOUD_API_TOKEN GitHub action secret. This is the Dagster+ agent token from the previous section. Refer to the agent tokens documentation for more info.

    Refer to the GitHub docs for more info about GitHub Action secrets.

  2. In your dagster-cloud-deploy.yml, set the DAGSTER_CLOUD_ORGANIZATION environment variable to your Dagster+ organization name.

  3. In your dagster-cloud-deploy.yml, uncomment the step that is relevant to your Docker container registry. For example, if using DockerHub, you'd uncomment the DockerHub step in these files. Ensure you have set up the relevant secrets for building and uploading your Docker images.

After making the above changes and commiting the workflow file, the CI process should be triggered to deploy your GitHub repository to Dagster+. During the deployment, the agent will attempt to load your code and update the metadata in Dagster+. Once finished, you should see the GitHub Action complete successfully and also be able to see the code location under the Deployment tag in Dagster+.

When finished, continue to the next step.

Step 5: Set up environment variables and secrets#

Congrats! At this point, your Dagster+ deployment should be up and running. To ensure the external services you use in your Dagster project work correctly, start setting up your environment variables. Using environment variables, you can securely pass in sensitive info like passwords, API tokens, etc.

Next steps#

From here, you can: