Ask AI

Quickstart: Run your dbt project with Dagster#

Note: This guide uses the DbtProject class, which is an experimental feature. Visit the dagster-dbt library API reference for more info.

Setup your environment#

Note: We strongly recommend installing Dagster inside a Python virtualenv. Visit the Dagster installation docs for more info.

Install dbt, Dagster, and the Dagster webserver. Run the following to install everything using pip:

pip install dagster-dbt dagster-webserver

The dagster-dbt library installs both dbt-core and dagster as dependencies. Refer to the dbt and Dagster installation docs for more info.


Load your dbt project in Dagster#

Running your dbt project with Dagster can be easily done after creating a single file. For this example, let's consider a basic use case - say you want to represent your dbt models as Dagster assets and run them daily at midnight.

With your text editor of choice, create a Python file in the same directory as your dbt project directory and add the following code. Note that since this file contains all Dagster definitions required for your code location, it is recommended to name it definitions.py.

The following code assumes that your Python file and dbt project directory are adjacent in the same directory. If that's not the case, make sure to update the RELATIVE_PATH_TO_MY_DBT_PROJECT constant so that it points to your dbt project.

from pathlib import Path

from dagster import AssetExecutionContext, Definitions
from dagster_dbt import (
    DbtCliResource,
    DbtProject,
    build_schedule_from_dbt_selection,
    dbt_assets,
)

RELATIVE_PATH_TO_MY_DBT_PROJECT = "./my_dbt_project"

my_project = DbtProject(
    project_dir=Path(__file__)
    .joinpath("..", RELATIVE_PATH_TO_MY_DBT_PROJECT)
    .resolve(),
)


@dbt_assets(manifest=my_project.manifest_path)
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build"], context=context).stream()


my_schedule = build_schedule_from_dbt_selection(
    [my_dbt_assets],
    job_name="materialize_dbt_models",
    cron_schedule="0 0 * * *",
    dbt_select="fqn:*",
)

defs = Definitions(
    assets=[my_dbt_assets],
    schedules=[my_schedule],
    resources={
        "dbt": DbtCliResource(project_dir=my_project),
    },
)

Run your dbt project in Dagster's UI#

Now that your code is ready, you can run Dagster's UI to take a look at your dbt project.

  1. Locate the Dagster file containing your definitions. If you followed our example to create a single Dagster file in the previous section, your file should be named definitions.py.

  2. To start Dagster's UI, run the following:

    dagster dev -f definitions.py
    

    Which will result in output similar to:

    Serving dagster-webserver on http://127.0.0.1:3000 in process 70635
    

In your browser, navigate to http://127.0.0.1:3000. The page will display the asset graph for the job created by the schedule definition:

Asset graph for your job in Dagster's UI, containing dbt models loaded as Dagster assets

In Dagster, running a dbt model corresponds to materializing an asset. Because a schedule definition has been added to your code location, your assets will be materialized at the next cron tick. The assets can also be materialized manually by click the Materialize all button near the top right corner of the page. This will launch a run to materialize the assets.


What's next?#

Congratulations on successfully running your dbt project in Dagster!

In this example, we created a single file to handle a simple use case and run your dbt project. To learn more about our dbt integrations and how to handle more complex use cases, consider the following options:

  • To find out more about integrating dbt with Dagster, begin the official dbt scaffold tutorial, like the dbt & Dagster project.
  • For an end-to-end example, from the project creation to the deployment to Dagster+, checkout out the Dagster & dbt course in Dagster University.