Converting an existing project
In this guide, we walk through converting an existing Dagster project that defines a Python package with a single Dagster asset, which is exposed in a top-level Definitions
object in my_existing_project/definitions.py
.
- uv
- pip
If you used uv
to set up your Dagster project, you will have a pyproject.toml
project configuration file:
tree
.
├── my_existing_project
│ ├── __init__.py
│ ├── assets.py
│ ├── definitions.py
│ └── py.typed
├── pyproject.toml
└── uv.lock
2 directories, 6 files
If you used pip
to set up your Dagster project, you will have a setup.py
project configuration file:
tree
.
├── my_existing_project
│ ├── __init__.py
│ ├── assets.py
│ ├── definitions.py
│ └── py.typed
└── setup.py
2 directories, 5 files
Step 1: Create and activate a virtual environment
First, you will need to make sure you have an activated and up-to-date virtual environment. We recommend creating the virtual environment in the project root (particularly when using uv
), although it is not required.
- uv
- pip
If you don't have a virtual environment yet, run:
uv sync
Then activate it:
source .venv/bin/activate
If you don't have a virtual environment yet, run:
python -m venv .venv
Next, activate it:
source .venv/bin/activate
Finally, install the project package as an editable install:
pip install --editable .
Step 2: Install dependencies
Next, you will need to install the dg
command line tool into your project virtual environment.
- uv
- pip
uv add dagster-dg-cli
pip install dagster-dg-cli
Step 3: Update project structure
Step 3.1: Add dg
configuration
The dg
command recognizes Dagster projects through the presence of TOML
configuration. This may be either a pyproject.toml
file with tool.dg
and tool.dg.project
sections, or a dg.toml
file.
- uv
- pip
Since the example project already has a pyproject.toml
file, you just need to add tool.dg
and tool.dg.project
sections:
...
[tool.dg]
directory_type = "project"
[tool.dg.project]
root_module = "my_existing_project"
code_location_target_module = "my_existing_project.definitions"
Since the example project has a setup.py
file and no pyproject.toml
file, you will need to create a dg.toml
file:
directory_type = "project"
[project]
root_module = "my_existing_project"
code_location_target_module = "my_existing_project.definitions"
Configuration reference
directory_type = "project"
: This is howdg
identifies your package as a Dagster project. This setting is required.root_module = "my_existing_project"
: This points to the root module of your project. This setting is required.code_location_target_module = "my_existing_project.definitions"
: This tellsdg
where to find the top-levelDefinitions
object in your project. Since this setting defaults toNAME_OF_ROOT_MODULE.definitions
, it is not strictly necessary to set it for this example. However, if your project has the top-levelDefinitions
object defined in a different module, this setting is required.
Step 3.2: Check configuration
Now that the project configuration file has been updated, you can interact with your project using dg
. To check the configuration and see the one asset in this project, run dg list defs
:
dg list defs
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets │ ┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Group ┃ Deps ┃ Kinds ┃ Description ┃ │
│ │ ┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│ │ │ my_asset │ default │ │ │ │ │
│ │ └──────────┴─────────┴──────┴───────┴─────────────┘ │
└─────────┴─────────────────────────────────────────────────────┘
Step 3.3: Add an entry point to the project for custom component types
dg
uses the Python entry point API
to expose custom component types and other scaffoldable objects from user projects. In the project configuration file, the entry point specifies the location of a Python submodule where the project exposes registry modules.
By convention, this submodule is named <root_module>.lib
, but in the case of a Dagster project, it is <root_module>.components
.
Step 3.3.1: Create submodule directory
First, create the submodule directory and __init__.py
file:
mkdir my_existing_project/components && touch my_existing_project/components/__init__.py
Step 3.3.2: Add entry point to project configuration file
Next, add a dagster_dg_cli.registry_modules
entry point to your project configuration file, then
reinstall the project package into your virtual environment.
- uv
- pip
Since the package metadata is in pyproject.toml
, you will need to add the entry
point declaration there:
...
[project.entry-points]
"dagster_dg_cli.registry_modules" = { my_existing_project = "my_existing_project.components"}
...
With pip
, the package metadata is in setup.py
. While it is possible to add
entry point declarations to setup.py
directly, you will want to be able to
read the entry point declaration from dg
, and there is no reliable
way to read setup.py
(since it is arbitrary Python code). Instead you will need to add the entry point to a new setup.cfg
file, which can be used alongside setup.py
.
Create setup.cfg
with the following contents:
[options.entry_points]
dagster_dg_cli.registry_modules =
my_existing_project = my_existing_project.components
If your package has existing entry points declared in setup.py
, you will need to move their definitions to setup.cfg
as well.
Step 3.3.3: Reinstall the project package
- uv
- pip
Reinstall the package. Note that uv sync
will not
reinstall your package, so you will need to use uv pip install
instead:
uv pip install --editable .
Reinstall the package:
pip install --editable .
The reinstallation step is crucial. Python entry points are registered at package installation time, so if you simply add a new entry point to an existing editable-installed package, it won't be picked up.
Step 3.4: Check configuration
If you've done everything correctly, you should now be able to run dg list registry-modules
and see the module my_existing_project.components
, which you have registered as an entry point, listed in the output:
dg list registry-modules
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Module ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩