Code locations#

Before the introduction of the DefinitionsAPI, definitions were grouped into repositories, and there could be many repostories in a particular code location. Refer to the Repositories documentation for info on this previous API and mental model.

Code locations

A code location is a collection of Dagster definitions loadable and accessible by Dagster's tools, such as the CLI, Dagit, and Dagster Cloud. A code location comprises:

  • A reference to a Python module that has an instance of Definitions in a top-level variable
  • A Python environment that can successfully load that module

Definitions within a code location have a common namespace and must have unique names. This allows them to be grouped and organized by code location in tools.

A single deployment can have one or multiple code locations.

Code locations are loaded in a different process and communicate with Dagster system processes over an RPC mechanism. This architecture provides several advantages:

  • When there is an update to user code, Dagit can pick up the change without a restart.
  • You can use multiple code locations to organize jobs, but still work on all of your code locations using a single instance of Dagit.
  • The Dagit process can run in a separate Python environment from user code so job dependencies don't need to be installed into the Dagit environment.
  • Each code location can be sourced from a separate Python environment, so teams can manage their dependencies (or even their Python versions) separately.

Relevant APIs#

NameDescription
DefinitionsThe object that contains all the definitions defined within a code location. Definitions include assets, jobs, resources, schedules, and sensors.

Defining code locations#

To define a code location, create a top-level variable that contains a Definitions object in a Python module. For example:

# my_file.py

defs = Definitions(
    assets=[dbt_customers_asset, dbt_orders_asset],
    schedules=[bi_weekly_schedule],
    sensors=[new_data_sensor],
    resources=dbt_resource
)

Definitions can be included in a Python file like my_file.py or a Python module. If using the latter, the Definitions object should be defined in the module's top-level __init__.py file.


Deploying and loading code locations#

Local development#

Dagster can load a file directly as a code location. In the following example, we used the -f argument to supply the name of the file to dagster dev:

dagster dev -f my_file.py

This command loads the definitions in my_file.py as a code location in the current Python environment.

You can also include multiple files at a time, which Dagster will load as individual code locations:

dagster dev -f my_file.py -f my_second_file.py

Cloud deployment#

The dagster_cloud.yaml file is used to create and deploy code locations for Cloud deployments. Each code location entry in this file has a code_source property, which is used to specify how a code location is sourced. Code locations can be sourced from a Python file or module:

To load a code location from a Python file, use the python_file property in your dagster_cloud.yaml:

# dagster_cloud.yaml

locations:
  - location_name: my-code-location
    code_source:
      python_file: my_file.py

Open source deployment#

The workspace.yaml file is used to load code locations for open source (OSS) deployments. This file specifies how to load a collection of code locations and is typically used in advanced use cases. Refer to the Open source deployment guides for more info.


Definitions versus repositories#

If you used @repository in previous Dagster versions, you might be interested in how Definitions and repositories differ. Check out the following table for a high-level comparison:

Definitions (Recommended)Repositories
Minimum Dagster version1.1.70.6
Description
  • Created by using the Definitions object assigned to a top-level variable
  • One Definitions object allowed per code location
ArgumentsEnforced typing and namingNo enforced typing and naming
Resources
  • resources argument can accept definitions and raw objects
  • Top-level resources are automatically bound to all assets
Resources are manually bound to assets (with_resources)
Multiple Python environmentsSupported for OSS deployments (viaworkspace.yaml)Supported

Troubleshooting#

ErrorDescription and resolution
Cannot have more than one Definitions object defined at module scopeDagster found multiple Definitions objects in a single Python module. Only one Definitions object may be in a single code location.