Before the introduction of the
DefinitionsAPI, definitions were grouped into repositories, and there could be many repostories in a particular code location. Refer to the Repositories documentation for info on this previous API and mental model.
A code location is a collection of Dagster definitions loadable and accessible by Dagster's tools, such as the CLI, Dagit, and Dagster Cloud. A code location comprises:
Definitionsin a top-level variable
Definitions within a code location have a common namespace and must have unique names. This allows them to be grouped and organized by code location in tools.
A single deployment can have one or multiple code locations.
Code locations are loaded in a different process and communicate with Dagster system processes over an RPC mechanism. This architecture provides several advantages:
|The object that contains all the definitions defined within a code location. Definitions include assets, jobs, resources, schedules, and sensors.|
To define a code location, create a top-level variable that contains a
Definitions object in a Python module. For example:
# my_file.py defs = Definitions( assets=[dbt_customers_asset, dbt_orders_asset], schedules=[bi_weekly_schedule], sensors=[new_data_sensor], resources=dbt_resource )
Definitions can be included in a Python file like
my_file.py or a Python module. If using the latter, the
Definitions object should be defined in the module's top-level
Refer to the Running Dagster locally guide for more info about local development, including how to configure your local instance.
Dagster can load a file directly as a code location. In the following example, we used the
-f argument to supply the name of the file:
dagster dev -f my_file.py
This command loads the definitions in
my_file.py as a code location in the current Python environment.
You can also include multiple files at a time, where each file will be loaded as a code location:
dagster dev -f my_file.py -f my_second_file.py
dagster_cloud.yaml file is used to create and deploy code locations for Cloud deployments. Each code location entry in this file has a
code_source property, which is used to specify how a code location is sourced. Code locations can be sourced from a Python file or module:
To load a code location from a Python file, use the
python_file property in your
# dagster_cloud.yaml locations: - location_name: my-code-location code_source: python_file: my_file.py
workspace.yaml file is used to load code locations for open source (OSS) deployments. This file specifies how to load a collection of code locations and is typically used in advanced use cases. Refer to the Open source deployment guides for more info.
If you used
@repository in previous Dagster versions, you might be interested in how
Definitions and repositories differ. Check out the following table for a high-level comparison:
|Minimum Dagster version||1.1.7||0.6|
|Arguments||Enforced typing and naming||No enforced typing and naming|
|Resources||Resources are manually bound to assets (|
|Multiple Python environments||Supported for OSS deployments (via||Supported|
|Error||Description and resolution|
|Cannot have more than one Definitions object defined at module scope||Dagster found multiple |