Skip to main content

Migrating existing definitions to components

note

This guide covers migrating existing Python definitions to components. This guide presupposes a components-enabled project. See the getting started guide or Making an existing code location components-compatible guide for more information.

When adding components to an existing Dagster code location, it is often useful to restructure your definitions into component folders, making it easier to eventually migrate them entirely to using components.

Example project​

Let's walk through an example of how to migrate existing definitions to components, with a project that has the following structure:

tree
.
├── README.md
├── my_existing_project
│   ├── __init__.py
│   ├── analytics
│   │   ├── __init__.py
│   │   ├── assets.py
│   │   └── jobs.py
│   ├── components
│   ├── definitions.py
│   └── elt
│       ├── __init__.py
│       ├── assets.py
│       └── jobs.py
└── pyproject.toml

5 directories, 10 files

At the top level, we load definitions from various modules:

my_existing_project/definitions.py
from pathlib import Path

import dagster_components as dg_components

import dagster as dg
from my_existing_project.analytics import assets as analytics_assets
from my_existing_project.analytics.jobs import (
    regenerate_analytics_hourly_schedule,
    regenerate_analytics_job,
)
from my_existing_project.elt import assets as elt_assets
from my_existing_project.elt.jobs import sync_tables_daily_schedule, sync_tables_job

defs = dg.Definitions.merge(
    dg.Definitions(
        assets=dg.load_assets_from_modules([elt_assets, analytics_assets]),
        jobs=[sync_tables_job, regenerate_analytics_job],
        schedules=[sync_tables_daily_schedule, regenerate_analytics_hourly_schedule],
    ),
    dg_components.build_component_defs(Path(__file__).parent / "components"),
)

Each of these modules contains a variety of Dagster definitions, including assets, jobs, and schedules.

Let's migrate the elt module to a component.

Create a Definitions component​

We'll start by creating a Definitions component for the elt module:

dg component scaffold 'definitions@dagster_components' elt-definitions
Using /.../my-existing-project/.venv/bin/dagster-components
Creating a Dagster component instance folder at /.../my-existing-project/my_existing_project/components/elt-definitions.
Using /.../my-existing-project/.venv/bin/dagster-components

This creates a new folder in my_existing_project/components/elt-definitions, with a component.yaml file. This component requires a definitions_path parameter, which points to a file that contains a Definitions object.

Let's begin by moving the elt module's contents to the new component folder:

mv my_existing_project/elt/* my_existing_project/components/elt-definitions

Next, let's create a new definitions.py file in the component folder, which will collect all of the elt module's definitions into a single Definitions object:

my_existing_project/components/elt-definitions/definitions.py
import dagster as dg

from . import assets
from .jobs import sync_tables_daily_schedule, sync_tables_job

defs = dg.Definitions(
    assets=dg.load_assets_from_modules([assets]),
    jobs=[sync_tables_job],
    schedules=[sync_tables_daily_schedule],
)

Finally, we can update the component.yaml file to point to the new definitions.py file:

my_existing_project/components/elt-definitions/component.yaml
type: definitions@dagster_components

params:
  definitions_path: definitions.py

Now that our component is defined, we can update the root definitions.py file to no longer explicitly load the elt module's Definitions:

my_existing_project/definitions.py
from pathlib import Path

import dagster_components as dg_components
from my_existing_project.analytics import assets as analytics_assets
from my_existing_project.analytics.jobs import (
    regenerate_analytics_hourly_schedule,
    regenerate_analytics_job,
)

import dagster as dg

defs = dg.Definitions.merge(
    dg.Definitions(
        assets=dg.load_assets_from_modules([analytics_assets]),
        jobs=[regenerate_analytics_job],
        schedules=[regenerate_analytics_hourly_schedule],
    ),
    dg_components.build_component_defs(Path(__file__).parent / "components"),
)

Now, our project structure looks like this:

tree
.
├── README.md
├── my_existing_project
│   ├── __init__.py
│   ├── analytics
│   │   ├── __init__.py
│   │   ├── assets.py
│   │   └── jobs.py
│   ├── components
│   │   └── elt-definitions
│   │       ├── __init__.py
│   │       ├── assets.py
│   │       ├── component.yaml
│   │       ├── definitions.py
│   │       └── jobs.py
│   └── definitions.py
├── pyproject.toml
└── uv.lock

5 directories, 13 files

We can repeat the same process for our other modules.

Fully migrated project​

Once each of our definitions modules are migrated to components, our project is left with a standardized structure.

tree
.
├── README.md
├── my_existing_project
│   ├── __init__.py
│   ├── components
│   │   ├── analytics-definitions
│   │   │   ├── __init__.py
│   │   │   ├── assets.py
│   │   │   ├── component.yaml
│   │   │   ├── definitions.py
│   │   │   └── jobs.py
│   │   └── elt-definitions
│   │       ├── __init__.py
│   │       ├── assets.py
│   │       ├── component.yaml
│   │       ├── definitions.py
│   │       └── jobs.py
│   └── definitions.py
├── pyproject.toml
└── uv.lock

5 directories, 15 files

Our project root now only constructs Definitions from components:

my_existing_project/definitions.py
from pathlib import Path

import dagster_components as dg_components

defs = dg_components.build_component_defs(Path(__file__).parent / "components")

Next steps​