Integrating asset metadata into Dagster+ Insights#

Any numeric asset materialization metadata attached to your assets can be viewed and aggregated in the Dagster Insights UI. This is a good way to track usage, cost, data size, or other characteristics of your data assets over time, and across different parts of your platform.

Prerequisites#

To complete the steps in this guide, you'll need a Dagster+ account on the Pro plan.

Step 1: Emit numeric values alongside your asset materializations#

To populate Insights with materialization metadata, you must first emit numeric metadata to be aggregated. In the below example, we attached a num_rows metadata value to the output of the table1 asset:

from dagster import Output, asset


@asset
def table1() -> Output[None]:
    ...  # write out some data to table1
    return Output(None, metadata={"num_rows": 25})

Insights lets us track this value over time and aggregate it between any other assets which might provide a num_rows value.

Refer to the Asset metadata documentation for more information about asset metadata.

Step 2: Enable viewing numeric metadata in Dagster+ Insights#

Once one or more assets are emitting numeric metadata values, you'll be able to enable viewing them in the Insights UI. Note: It may take up to 24 hours for new metadata types to be reflected in the UI.

To modify the list of materialization metadata values shown in Insights, click Edit in the sidebar next to the User provided metrics header:

In the dialog that appears, check or uncheck metrics to use them in Insights. Selected metrics will be immediately available to view in Insights in the sidebar and on individual asset pages. You can also change a metric's icon, display name, and description by clicking the pencil icon next to the metric.

Related#