This guide is applicable to Dagster Cloud.
In this guide, we'll cover the requirements for Dagster code, how to add code in Dagster Cloud, and how to add code using the
A code location specifies a single Python package or file that defines your Dagster code. When you add a code location in Dagster Code, you're instructing the agent where to find your code.
When you add or update a code location, the agent uses the location configuration to load your code and upload metadata about your jobs to Dagster Cloud. Each full deployment - for example,
prod - can include code from one or more code locations.
Note that, unlike Dagster Open Source, Dagster Cloud doesn't require a
workspace.yaml file. Instead, you use the Dagster Cloud API to configure your workspace. You can still create a
workspace.yaml file if you want to load your code in an open-source Dagit instance, but doing so won't affect how your code is loaded in Dagster Cloud.
To work with Dagster Cloud, your Dagster code:
Must be loaded from a single entry point, either a Python file or package. This entry point can load repositories from other files or packages.
Must run in an environment where the
dagster-cloud 0.13.2 or later Python packages are installed.
If using Hybrid Deployment:
And you're using an Amazon Elastic Container Service (ECS), Kubernetes, or Docker agent, your code must be packaged into a Docker image and pushed to a registry your agent can access. Dagster Cloud doesn't need access to your image - your agent only needs to be able to pull it.
Additionally, the Dockerfile for your image doesn't need to specify an entry point or command. These will be supplied by the agent when it runs your code using your supplied image.
And you're using a local agent, your code must be in a Python environment that can be accessed on the same machine as your agent.
Additionally, note that:
Editor, Admin, or Organization Admin permissions are required to manage code locations in Dagster Cloud.
Sign in to your Dagster Cloud account.
Click + Add code location. This will open a YAML editor with a schema describing the acceptable fields:
In the editor, define the code location's configuration:
|code_source||Set this key to either |
|executable_path||Optional. Define a specific Python executable if your code should run in a certain Python environment. If left undefined, the code will run using the default dagster command-line entry-point.|
|image||Required if not using a local agent. Specifies a Docker image for use with containerized agents.|
|working_directory||Specifies the directory to use to resolve relative Python imports while loading your code.|
|attribute||Specifies only a specific Dagster repository should be loaded.|
|container_context||Optional. For agent versions 0.14.9 and later. Customizes the code location for a specific execution environment. Refer to the Agent documentation for info on available configuration options for each agent type, including declaring environment variables and secrets.|
For example, the following config specifies that a code location should include a secret named
my_secret and run in a k8s namespace (
my_namespace) whenever the Kubernetes agent creates a pod for the location:
location_name: cloud-examples image: dagster/dagster-cloud-examples:latest code_source: package_name: dagster_cloud_examples container_context: k8s: namespace: my_namespace env_secrets: - my_secret
When finished, click Add code location.
The agent will attempt to load your code and send its metadata to Dagster Cloud. Note: This may take some time.
Once your code has loaded, the location will show a green Loaded status and jobs will appear in Dagster Cloud. If the agent is unable to load your code, the location will show an error with more information.
To modify a code location, click the dropdown menu to the right of the location. In the menu, click Modify:
After a code location is updated, the agent will perform a rolling update of your code and jobs will update in Dagster Cloud. Note: Updating code won't interrupt any currently launched runs.
To reload your code and upload job metadata to Dagster Cloud without modifying the code location, click the Redeploy button:
For example, if the agent was unable to pull your image due to a permissions issue that's since been addressed, clicking Redeploy will tell the agent to try again.
To delete a code location, click the dropdown menu to the right of the location. In the menu, click Remove:
When prompted, confirm the deletion.
You can also use the
dagster-cloud workspace CLI commands to:
These commands perform the same underlying operations as editing your code locations in the Deployment tab in Dagster Cloud. Refer to the dagster-cloud CLI guide for more info and installation instructions.
You can add or update a code location with the
add-location command. For example, to add our public example image, you can run:
# Set up YAML file for example location cat > example_location.yaml <<EOL location_name: cloud-examples image: dagster/dagster-cloud-examples:latest code_source: package_name: dagster_cloud_examples EOL dagster-cloud workspace add-location --from example_location.yaml
Instead of creating a YAML file, you may also pass these values inline as command line options:
dagster-cloud workspace add-location test_location \ --image dagster/dagster-cloud-examples:latest \ --package-name dagster_cloud_examples
You may also selectively overwrite parts of your YAML input using inline options:
dagster-cloud workspace add-location \ --from example_location.yaml \ --image dagster/dagster-cloud-examples:1d9c5d
The arguments to the
update-location commands are similar to the keys in the YAML config editor in the Deployment tab. To see all available options, run:
dagster-cloud workspace add-location --help
Note: If updating an existing code location, the full set of information about the location must be specified even if only one piece of configuration is modified.
Delete an existing code location from Dagster Cloud using the
dagster-cloud workspace delete-location <LOCATION_NAME>
You can also keep the YAML configuration for your entire workspace in a file and use the
dagster-cloud sync command to reconcile the workspace config in Dagster Cloud with that local file.
For example, if you have the following
locations: - location_name: machine-learning image: myregistry/dagster-machine-learning:mytag code_source: package_name: dagster_cloud_machine_learning executable_path: /my/folder/python_executable attribute: my_repo - location_name: data-eng image: myregistry/dagster-data-eng:myothertag code_source: python_file: repo.py working_directory: /my/folder/working_dir/
Reconcile the above file with Dagster Cloud's remote workspace by running:
dagster-cloud workspace sync -w dagster_cloud.yaml