Skip to main content

Migrate from Dagster+ Serverless to Hybrid

After using a Dagster+ Serverless deployment, you may decide to use your own infrastructure to execute your code. Transitioning to a Hybrid deployment only requires a few steps, and can be done without any loss of execution history or metadata, allowing you to maintain continuity and control over your operations.

warning

Transitioning from Serverless to Hybrid requires some downtime, as your Dagster+ deployment will temporarily not have an agent to execute user code.

Prerequisites​

To follow the steps in this guide, you'll need Organization Admin permissions in your Dagster+ account.

1. Deactivate your Serverless agent​

  1. In the Dagster+ UI, navigate to the Deployment > Agents page.
  2. On the right side of the page, click the dropdown arrow and select Switch to Hybrid.

Switch agent to Hybrid dropdown arrow

It may take a few minutes for the agent to deactivate and be removed from the list of agents.

2. Create a Hybrid agent​

Next, you'll need to create a Hybrid agent to execute your code. Follow the setup instructions for the agent of your choice:

  • Amazon Web Services (AWS), which launches user code as Amazon Elastic Container Service (ECS) tasks.
  • Docker, which launches user code in Docker containers on your machine.
  • Microsoft Azure, which launches user code to Azure infrastructure.
  • Kubernetes, which launches user code on a Kubernetes cluster.
  • Local, which launches user code in operating system subprocesses on your machine.

3. Update your code locations' configuration in dagster_cloud.yaml​

See the documentation for the agent of your choice:

4. Confirm successful setup​

Once you've set up a Hybrid agent, navigate to the Deployment > Agents page in the UI. The new agent should display in the list with a RUNNING status:

Running Hybrid agent displayed in Dagster+ UI

5. Update your build process​

Update your build process to publish a new container image and configuration for each code location. To use Dagster's CI/CD process, see the CI/CD in Dagster+ Hybrid guide.

6. Replace Serverless-only features with their Hybrid equivalents​

Serverless-only featureHybrid equivalent
Disabling PEX-based deploys and customizing the Docker image with lifecycle hooksTo customize a code location's runtime environment, you can customize the code location's Dockerfile to build its image.
Enabling non-isolated runsWhile this feature doesn't have a direct Hybrid equivalent, you can experiment with the in_process_executor or multiprocess_executor for specific jobs or entire code locations to reduce overhead.

Next steps​