Making an existing code location components-compatible
This guide is only relevant if you are starting from an existing Dagster code location. This setup is unnecessary if you used dg code-location generate to create your code location.
Let's begin with a Dagster code location that has some assets, but no components:
tree
.
├── README.md
├── my_existing_project
│ ├── __init__.py
│ ├── assets.py
│ └── definitions.py
├── my_existing_project_tests
│ ├── __init__.py
│ └── test_assets.py
└── pyproject.toml
3 directories, 7 files
Install dependencies
Install the dg command line tool
We'll need to install the dg command line tool, which is used to scaffold components. We recommend installing dg globally using the uv package manager; it can also be installed using pip.
uv tool install dagster-dg
Install dagster-components
Next, we'll need to install the dagster-components package.
Though this is optional, we generally recommend using a separate virtual environment for each code location, which can be accomplished using uv:
- Using uv virtual environment
- Using pip
uv venv
Then, we can use uv sync to install the dependencies from our pyproject.toml, and then install the dagster-components package:
uv sync && uv add dagster-components
pip install dagster-components
Update project structure
Update pyproject.toml
Add a tool.dg section to your pyproject.toml file. This will tell the dg command line tool that this code location is a valid Dagster code location.
...
[tool.dg]
is_code_location = true
[tool.dagster]
module_name = "my_existing_project.definitions"
code_location_name = "my_existing_project"
...
Create a components directory
Next, you'll want to create a directory to contain any new components you add to your code location. By convention, this directory is named components, and exists at the top level of your code location's Python module.
mkdir my_existing_project/components
Modify top-level definitions
dagster-components provides a utility to create a Definitions object from your components directory. Because you're working with an existing code location, you'll want to combine your existing definitions with the ones from your components directory.
To do so, you'll need to modify your definitions.py file, or whichever file contains your top-level Definitions object.
You can manually construct a set of definitions for your components using build_component_defs, then merge them with your existing definitions using Definitions.merge. You point build_components_defs at the directory you just created that contains components.
- Before
- After
import dagster as dg
from my_existing_project import assets
all_assets = dg.load_assets_from_modules([assets])
defs = dg.Definitions(
assets=all_assets,
)
from pathlib import Path
import dagster_components as dg_components
from my_existing_project import assets
import dagster as dg
all_assets = dg.load_assets_from_modules([assets])
defs = dg.Definitions.merge(
dg.Definitions(assets=all_assets),
dg_components.build_component_defs(Path(__file__).parent / "components"),
)
Now, your code location is ready to use components! dg can be used to scaffold new components directly into the existing code location.