Ask AI

Using dbt with Dagster, part four: Add a downstream asset#

This is part four of the Using dbt with Dagster assets tutorial.

By this point, you've set up a dbt project, loaded dbt models into Dagster as assets, and defined assets upstream of your dbt models.

In this step, you'll:

Step 1: Install the plotly library#

pip install plotly

Step 2: Define the order_count_chart asset#

You've added upstream assets to your data pipeline, but nothing downstream - until now. In this step, you'll define a Dagster asset called order_count_chart that uses the data in the customers dbt model to computes a plotly chart of the number of orders per customer.

Like the raw_customers asset that we added in the previous section, we'll put this asset in our definitions.py file, inside the jaffle_dagster directory.

To add the order_count_chart asset:

  1. Replace the imports section with the following:

    import os

import duckdb
import pandas as pd
import plotly.express as px
from dagster import MetadataValue, AssetExecutionContext, asset
from dagster_dbt import DbtCliResource, dbt_assets, get_asset_key_for_model

from .project import dbt_project

    This adds an import for plotly, as well as get_asset_key_for_model and MetadataValue, which we'll use in our asset.

  2. After your definition of jaffle_shop_dbt_assets, add the definition for the order_count_chart asset:

    @asset(
    compute_kind="python",
    deps=[get_asset_key_for_model([jaffle_shop_dbt_assets], "customers")],
)
def order_count_chart(context: AssetExecutionContext):
    # read the contents of the customers table into a Pandas DataFrame
    connection = duckdb.connect(os.fspath(duckdb_database_path))
    customers = connection.sql("select * from customers").df()

    # create a plot of number of orders by customer and write it out to an HTML file
    fig = px.histogram(customers, x="number_of_orders")
    fig.update_layout(bargap=0.2)
    save_chart_path = duckdb_database_path.parent.joinpath("order_count_chart.html")
    fig.write_html(save_chart_path, auto_open=True)

    # tell Dagster about the location of the HTML file,
    # so it's easy to access from the Dagster UI
    context.add_output_metadata(
        {"plot_url": MetadataValue.url("file://" + os.fspath(save_chart_path))}
    )

    This asset definition looks similar the asset we defined in the previous section. In this case, instead of fetching data from an external source and writing it to DuckDB, it reads data from DuckDB, and then uses it to make a plot.

    The line deps=get_asset_key_for_model([jaffle_shop_dbt_assets], "customers") tells Dagster that this asset is downstream of the customers dbt model. This dependency will be displayed as such in Dagster's UI. If you launch a run to materialize both of them, Dagster won't run order_count_chart until customers completes.

  3. Add the order_count_chart to the Definitions:

    from dagster import Definitions
from dagster_dbt import DbtCliResource

from .assets import jaffle_shop_dbt_assets, order_count_chart, raw_customers
from .project import dbt_project
from .schedules import schedules

defs = Definitions(
    assets=[raw_customers, jaffle_shop_dbt_assets, order_count_chart],
    schedules=schedules,
    resources={
        "dbt": DbtCliResource(project_dir=dbt_project),
    },
)

Step 3: Materialize the order_count_chart asset#

If the Dagster UI is still running from the previous section, click the "Reload Definitions" button in the upper right corner. If you shut it down, then you can launch it again with the same command from the previous section:

dagster dev

The UI will look like this:

Asset group with dbt models and Python asset

A new asset named order_count_chart is at the bottom, downstream of the customers asset. Click on order_count_chart and click Materialize selected.

That's it! When the run successfully completes, the following chart will automatically open in your browser:

plotly chart asset displayed in Chrome

What's next?#

That's the end of this tutorial - congratulations! By now, you should have a working dbt and Dagster integration and a handful of materialized Dagster assets.

What's next? From here, you can: