Converting an existing project

info

Dagster Components is currently in Release Candidate status. APIs are stable, with broader integration coverage and full feature parity in active development and coming soon. Check it out and let us know what you think in the #dg-components channel in the Dagster Community Slack!

Suppose we have an existing Dagster project. Our project defines a Python package with a single Dagster asset. The asset is exposed in a top-level Definitions object in my_existing_project/definitions.py. We'll consider both a case where we have been using uv with pyproject.toml and pip with setup.py.

uv

tree

.
├── my_existing_project
│   ├── __init__.py
│   ├── assets.py
│   ├── definitions.py
│   └── py.typed
├── pyproject.toml
└── uv.lock

2 directories, 6 files

pip

tree

.
├── my_existing_project
│   ├── __init__.py
│   ├── assets.py
│   ├── definitions.py
│   └── py.typed
└── setup.py

2 directories, 5 files

Before proceeding, we'll make sure we have an activated and up-to-date virtual environment in the project root. Having the virtual environment located in the project root is recommended (particularly when using uv) but not required.

uv

If you don't have a virtual environment yet, run:

uv sync

Then activate it:

source .venv/bin/activate

pip

If you don't have a virtual environment yet, run:

python -m venv .venv

Now activate it:

source .venv/bin/activate

And install the project package as an editable install:

pip install --editable .

Install dependencies

Install the dg command line tool into your project virtual environment.

uv

uv add dagster-dg-cli

pip

pip install dagster-dg-cli

Update project structure

Add dg configuration

The dg command recognizes Dagster projects through the presence of TOML configuration. This may be either a pyproject.toml file with a tool.dg section or a dg.toml file. Let's add this configuration:

uv

Since our project already has a pyproject.toml file, we can just add the requisite tool.dg section to the file:

pyproject.toml

...
[tool.dg]
directory_type = "project"

[tool.dg.project]
root_module = "my_existing_project"
code_location_target_module = "my_existing_project.definitions"

pip

Since our sample project has a setup.py and no pyproject.toml, we'll create a dg.toml file:

dg.toml

directory_type = "project"

[project]
root_module = "my_existing_project"
code_location_target_module = "my_existing_project.definitions"

There are three settings:

• directory_type = "project": This is how dg identifies your package as a Dagster project. This is required.
• project.root_module = "my_existing_project": This points to the root module of your project. This is also required.
• project.code_location_target_module = "my_existing_project.definitions": This tells dg where to find the top-level Definitions object in your project. This actually defaults to [root_module].definitions, so it is not strictly necessary for us to set it here, but we are including this setting in order to be explicit--existing projects might have the top-level Definitions object defined in a different module, in which case this setting is required.

Now that these settings are in place, you can interact with your project using dg. If we run dg list defs we can see the sole existing asset in our project:

dg list defs

┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                         ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│         │ ┃ Key      ┃ Group   ┃ Deps ┃ Kinds ┃ Description ┃ │
│         │ ┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│         │ │ my_asset │ default │      │       │             │ │
│         │ └──────────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴─────────────────────────────────────────────────────┘

Add a dagster_dg_cli.registry_modules entry point

We're not quite done adding configuration. dg uses the Python entry point API to expose custom component types and other scaffoldable objects from user projects. Our entry point declaration will specify a submodule as the location where our project exposes registry modules. By convention, this submodule is named <root_module>.lib. In our case, it will be my_existing_project.lib. Let's create this submodule now:

mkdir my_existing_project/components && touch my_existing_project/components/__init__.py

We'll need to add a dagster_dg_cli.registry_modules entry point to our project and then reinstall the project package into our virtual environment. The reinstallation step is crucial. Python entry points are registered at package installation time, so if you simply add a new entry point to an existing editable-installed package, it won't be picked up.

Entry points can be declared in either pyproject.toml or setup.py:

uv

Since our package metadata is in pyproject.toml, we'll add the entry point declaration there:

pyproject.toml

...
[project.entry-points]
"dagster_dg_cli.registry_modules" = { my_existing_project = "my_existing_project.components"}
...

Then we'll reinstall the package. Note that uv sync will not reinstall our package, so we'll use uv pip install instead:

uv pip install --editable .

pip

Our package metadata is in setup.py. While it is possible to add entry point declarations to setup.py directly, we want to be able to read the entry point declaration from dg, and there is no reliable way to read setup.py (since it is arbitrary Python code). So we'll instead add the entry point to a new setup.cfg, which can be used alongside setup.py. Create setup.cfg with the following contents (if your package has existing entry points declared in setup.py, you'll want to move their definitions to setup.cfg as well):

setup.cfg

[options.entry_points]
dagster_dg_cli.registry_modules =
    my_existing_project = my_existing_project.components

Then we'll reinstall the package:

pip install --editable .

If we've done everything correctly, we should now be able to run dg list registry-modules and see the module my_existing_project.components, which we have registered as an entry point, listed in the output.

dg list registry-modules

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Module                         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ dagster                        │
│ my_existing_project.components │
└────────────────────────────────┘

We can now scaffold a new component in our project and it will be available to dg commands. First create the component:

dg scaffold component Foo

Scaffolded Dagster component at /.../my-existing-project/my_existing_project/components/foo.py.

Then run dg list components to confirm that the new component is available:

dg list components

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Key                                ┃ Summary                                                                         ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ dagster.DefinitionsComponent       │ An arbitrary set of Dagster definitions.                                        │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.DefsFolderComponent        │ A component that represents a directory containing multiple Dagster definition  │
│                                    │ modules.                                                                        │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.FunctionComponent          │ Represents a Python function, alongside the set of assets or asset checks that  │
│                                    │ it is responsible for executing.                                                │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.PythonScriptComponent      │ Represents a Python script, alongside the set of assets and asset checks that   │
│                                    │ it is responsible for executing.                                                │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.TemplatedSqlComponent      │ A component which executes templated SQL from a string or file.                 │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ dagster.UvRunComponent             │ Represents a Python script, alongside the set of assets or asset checks that it │
│                                    │ is responsible for executing.                                                   │
├────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ my_existing_project.components.Foo │ COMPONENT SUMMARY HERE.                                                         │
└────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────┘

You should see the my_project.lib.MyComponentType listed in the output.

Create a defs directory

Part of the dg experience is autoloading definitions. This means automatically picking up any definitions that exist in a particular module. We are going to create a new submodule named my_existing_project.defs (defs is the conventional name of the module for where definitions live in dg) from which we will autoload definitions.

mkdir my_existing_project/defs

Modify top-level definitions

Autoloading is provided by a function that returns a Definitions object. Because we already have some other definitions in our project, we'll combine those with the autoloaded ones from my_existing_project.defs.

To do so, you'll need to modify your definitions.py file, or whichever file contains your top-level Definitions object.

You'll autoload definitions using load_defs, then merge them with your existing definitions using Definitions.merge. You pass load_defs the defs module you just created:

Before

import dagster as dg
from my_existing_project.assets import my_asset

defs = dg.Definitions(
    assets=[my_asset],
)

After

from pathlib import Path

from my_existing_project.assets import my_asset

import dagster as dg

defs = dg.Definitions.merge(
    dg.Definitions(assets=[my_asset]),
    dg.load_from_defs_folder(project_root=Path(__file__).parent.parent),
)

Now let's add an asset to the new defs module. Create my_existing_project/defs/autoloaded_asset.py with the following contents:

import dagster as dg


@dg.asset
def autoloaded_asset(): ...

Finally, let's confirm the new asset is being autoloaded. Run dg list defs again and you should see both the new autoloaded_asset and old my_asset:

dg list defs

┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions                                                 ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets  │ ┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│         │ ┃ Key              ┃ Group   ┃ Deps ┃ Kinds ┃ Description ┃ │
│         │ ┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│         │ │ autoloaded_asset │ default │      │       │             │ │
│         │ ├──────────────────┼─────────┼──────┼───────┼─────────────┤ │
│         │ │ my_asset         │ default │      │       │             │ │
│         │ └──────────────────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴─────────────────────────────────────────────────────────────┘

Now your project is fully compatible with dg!

Next steps

• Restructure existing definitions
• Add a new definition to your project