Ask AI

Managing code locations in Dagster+#

This guide is applicable to Dagster+.

A code location specifies a single Python package or file that defines your Dagster code. When you add a code location in Dagster+, you're instructing your deployment's agent where to find your code. This applies to both Hybrid and Serverless deployments.

Dagster+ agents use this location configuration to load your code and upload metadata about your jobs. Each full deployment - for example, prod - can include code from one or more code locations.

By the end of this guide, you'll understand:

  • The requirements for Dagster code to work with Dagster+
  • How to interact with code locations using the Dagster UI
  • How to interact with code locations using the dagster-cloud CLI

Dagster+ code requirements#

Learn by example? Check out the example repo, which is set up to run in Dagster+.

To work with Dagster+, your Dagster code:

  • Must be loaded from a single entry point, either a Python file or package. This entry point can load repositories from other files or packages.

  • Must run in an environment where the dagster and dagster-cloud 0.13.2 or later Python packages are installed.

  • If using Hybrid Deployment:

    • And you're using an Amazon Elastic Container Service (ECS), Kubernetes, or Docker agent, your code must be packaged into a Docker image and pushed to a registry your agent can access. Dagster+ doesn't need access to your image - your agent only needs to be able to pull it.

      Additionally, the Dockerfile for your image doesn't need to specify an entry point or command. These will be supplied by the agent when it runs your code using your supplied image.

    • And you're using a local agent, your code must be in a Python environment that can be accessed on the same machine as your agent.

Additionally, note that:

  • Your code doesn't need to use the same version of Dagster as your agent
  • Different code locations can use different versions of Dagster
  • Dagster+ doesn't require a workspace.yaml file. You can still create a workspace.yaml file to load your code in an open source Dagster webserver instance, but doing so won't affect how your code is loaded in Dagster+.

Interacting with code locations#

Editor, Admin, or Organization Admin permissions are required to manage code locations in Dagster+.

If you're an Editor or Admin, you can only manage code locations in deployments where you're an Editor or Admin.

Interacting with code locations can be done using the Dagster+ UI or the dagster-cloud CLI. The CLI commands perform the same underlying operations as those in the Deployment tab of the UI.

Adding code locations#

For Serverless deployments, there are two ways to deploy your code to Dagster+:

  • Start from a template- Use one of our quickstart templates to get up and running. All templates come with CI/CD already configured and will be cloned to a new git repository.

  • Import an existing project - Import an existing git repository using our GitHub or GitLab integrations or the dagster-cloud CLI. Note: If using the GitHub or GitLab integrations, Dagster+ will:

Select an option in the UI and follow the prompts.

Modifying code locations#

To modify a code location, click the dropdown menu to the right of the location. In the menu, click Modify:

Highlighted Modify option in the dropdown menu next to a code location in Dagster+

After a code location is updated, the agent will perform a rolling update of your code and jobs will update in Dagster+. Note: Updating code won't interrupt any currently launched runs.

Redeploying code locations#

To reload your code and upload job metadata to Dagster+ without modifying the code location, click the Redeploy button:

Highlighted Redeploy option in the dropdown menu next to a code location in Dagster+

For example, if the agent was unable to pull your image due to a permissions issue that's since been addressed, clicking Redeploy will tell the agent to try again.

Deleting code locations#

To delete a code location, click the dropdown menu to the right of the location. In the menu, click Remove:

Highlighted Remove option in the dropdown menu next to a code location in Dagster+

When prompted, confirm the deletion.

Syncing the workspace with the CLI#

The YAML configuration for all code locations in a workspace can be kept in a dagster_cloud.yaml file and reconciled with the workspace config in Dagster+ using the deployment sync-locations command.

For example, if you have the following dagster_cloud.yaml file:

locations:
  - location_name: machine-learning
    image: myregistry/dagster-machine-learning:mytag
    code_source:
      package_name: dagster_cloud_machine_learning
    executable_path: /my/folder/python_executable
    attribute: my_repo
  - location_name: data-eng
    image: myregistry/dagster-data-eng:myothertag
    code_source:
      python_file: repo.py
    working_directory: /my/folder/working_dir/

Run the following to reconcile the file with Dagster+'s remote workspace:

dagster-cloud deployment sync-locations -w dagster_cloud.yaml

To see all available options, run:

dagster-cloud deployment sync-locations -h