Using components in an existing project

info

This feature is still in development and might change in patch releases. It’s not production-ready, and the documentation may also evolve. Stay tuned for updates.

note

This guide is only relevant if you are starting from an existing Dagster project. This setup is unnecessary if you used dg scaffold project to create your project.

Let's begin with a Dagster project that has some assets, but no components:

tree

.
├── README.md
├── my_existing_project
│   ├── __init__.py
│   ├── assets.py
│   └── definitions.py
└── pyproject.toml

2 directories, 5 files

Install dependencies

Install the `dg` command line tool

We'll need to install the dg command line tool, which is used to scaffold components. We recommend installing dg globally using the uv package manager; it can also be installed using pip.

uv tool install dagster-dg

Install `dagster-components`

Next, we'll need to install the dagster-components package.

Though this is optional, we generally recommend using a separate virtual environment for each project, which can be accomplished using uv:

Using uv virtual environment
Using pip

uv venv

Then, we can use uv sync to install the dependencies from our pyproject.toml, and then install the dagster-components package:

uv sync && uv add dagster-components

pip install dagster-components

Update project structure

Update `pyproject.toml`

Add a tool.dg section to your pyproject.toml file. This will tell the dg command line tool that this project is a valid Dagster project.

pyproject.toml
...
[tool.dg]
is_project = true

[tool.dagster]
module_name = "my_existing_project.definitions"
code_location_name = "my_existing_project"
...

Create a `components` directory

Next, you'll want to create a directory to contain any new components you add to your project. By convention, this directory is named components, and exists at the top level of your project's Python module.

mkdir my_existing_project/components

Modify top-level definitions

dagster-components provides a utility to create a Definitions object from your components directory. Because you're working with an existing project, you'll want to combine your existing definitions with the ones from your components directory.

To do so, you'll need to modify your definitions.py file, or whichever file contains your top-level Definitions object.

You can manually construct a set of definitions for your components using build_component_defs, then merge them with your existing definitions using Definitions.merge. You point build_components_defs at the directory you just created that contains components.

Before
After

import dagster as dg
from my_existing_project import assets

all_assets = dg.load_assets_from_modules([assets])

defs = dg.Definitions(
    assets=all_assets,
)

from pathlib import Path

import dagster_components as dg_components
from my_existing_project import assets

import dagster as dg

all_assets = dg.load_assets_from_modules([assets])

defs = dg.Definitions.merge(
    dg.Definitions(assets=all_assets),
    dg_components.build_component_defs(Path(__file__).parent / "components"),
)

Now, your project is ready to use components! dg can be used to scaffold new components directly into the existing project.

Install dependencies​

Install the dg command line tool​

Install dagster-components​

Update project structure​

Update pyproject.toml​

Create a components directory​

Modify top-level definitions​

Next steps​