Making a project dg-compatible
This guide is only relevant if you are starting from an existing Dagster project. This setup is unnecessary if you used dg scaffold project
to create your project.
Let's begin with a Dagster project that has some assets:
tree
.
├── README.md
├── my_existing_project
│ ├── __init__.py
│ ├── assets.py
│ └── definitions.py
└── pyproject.toml
2 directories, 5 files
Install dependencies
Install the dg
command line tool
We'll need to install the dg
command line tool, which is used to scaffold components. We recommend installing dg
globally using the uv
package manager; it can also be installed using pip
.
uv tool install dagster-dg
Install dagster-components
Next, we'll need to install the dagster-components
package.
Though this is optional, we generally recommend using a separate virtual environment for each project, which can be accomplished using uv
:
- Using uv virtual environment
- Using pip
uv venv
Then, we can use uv sync
to install the dependencies from our pyproject.toml
, and then install the dagster-components
package:
uv sync && uv add dagster-components
pip install dagster-components
Update project structure
Update pyproject.toml
Add a tool.dg
section to your pyproject.toml
file. This will tell the dg
command line tool that this project is a valid Dagster project.
...
[tool.dg]
directory_type = "project"
[tool.dg.project]
root_module = "my_existing_project"
defs_module = "my_existing_project.defs"
[tool.dagster]
module_name = "my_existing_project.definitions"
code_location_name = "my_existing_project"
...
Create a defs
directory
Next, you'll want to create a directory to contain any new definitions you add to your project. By convention, this directory is named defs
, and exists at the top level of your project's Python module.
mkdir my_existing_project/defs
Modify top-level definitions
dagster-components
provides a utility to create a Definitions
object from your defs directory. Because you're working with an existing project, you'll want to combine your existing definitions with the ones from your defs directory.
To do so, you'll need to modify your definitions.py
file, or whichever file contains your top-level Definitions
object.
You can manually construct a set of definitions using load_defs
, then merge them with your existing definitions using Definitions.merge
. You point load_defs
at the module you just created.
- Before
- After
# isort: skip_file
from my_existing_project import assets
import dagster as dg
all_assets = dg.load_assets_from_modules([assets])
defs = dg.Definitions(
assets=all_assets,
)
import dagster_components as dg_components
import my_existing_project.defs
from my_existing_project import assets
import dagster as dg
all_assets = dg.load_assets_from_modules([assets])
defs = dg.Definitions.merge(
dg.Definitions(assets=all_assets),
dg_components.load_defs(my_existing_project.defs),
)
Now, your project is fully compatible with dg
! dg
can be used to scaffold new definitions directly into the existing project.