This guide is applicable to Dagster Cloud.
Dagster Cloud Serverless is a fully managed version of Dagster Cloud, and is the easiest way to get started with Dagster. With Serverless, you can run your Dagster jobs without spinning up any infrastructure.
Serverless works best with workloads that primarily orchestrate other services or perform light computation. Most workloads fit into this category, especially those that orchestrate third-party SaaS products like cloud data warehouses and ETL tools.
If any of the following are applicable, you should select Hybrid deployment:
Serverless is currently in early access and is subject to the following limitations:
debian:bullseye-slim. Custom Dockerfiles are not currently supported.
Enterprise customers may request a quota increase by contacting Sales.
If you are a GitHub user, our GitHub integration is the fastest way to get started. It uses a GitHub app and GitHub Actions to set up a repo containing skeleton code and configuration consistent with Dagster Cloud's best practices with a single click.
When you create a new Dagster Cloud organization, you'll be prompted to choose Serverless or Hybrid deployment. Once activated, our GitHub integration will scaffold a new git repo for you with Serverless and Branch Deployments already configured. Pushing to the
main branch will deploy to your
prod Serverless deployment. Pull requests will spin up ephemeral branch deployments using the Serverless agent.
If you don't want to use our GitHub integration, we offer a powerful CLI that you can use in another CI environment or on your local laptop.
First, create a new project with the Dagster open-source CLI.
pip install dagster dagster project from-example \ --name my-dagster-project \ --example assets_modern_data_stack
Once scaffolded, add
dagster-cloud as a dependency in your
Next, install the dagster-cloud CLI and log in to your org. Note: The CLI requires a recent version of Python 3 and Docker.
pip install dagster-cloud dagster-cloud configure
You can also configure the
dagster-cloud tool noninteractively; see the CLI docs for more information.
Finally, deploy your project with Dagster Cloud Serverless:
dagster-cloud serverless deploy \ --location-name example \ --package-name assets_modern_data_stack
Often you'll need to securely access secrets from your jobs. With Dagster Cloud Serverless, secrets are added as environment variables that can be read using APIs like StringSource.
KEY=VALUEpairs using the
dagster-cloud serverless deploy --helpto learn more.
If your organization begins to hit the limitations of Serverless, you should transition to a Hybrid deployment. Hybrid deployments allow you to run an agent in your own infrastructure and give you substantially more flexibility and control over the Dagster environment.
To switch to Hybrid, navigate to Status > Agents in your Dagster Cloud account. On this page, you can disable the Serverless agent on and view instructions for enabling Hybrid.
Unlike Hybrid, Serverless Deployments on Dagster Cloud require direct access to your data, secrets and source code.