Table metadata#

Table metadata provides additional context about a tabular asset, such as its schema, row count, and more. This metadata can be used to improve collaboration, debugging, and data quality in your data platform.

Dagster supports attaching different types of table metadata to assets, including:

• Column schema, which describes the structure of the table, including column names and types
• Row count, which describes the number of rows in a materialized table
• Column-level lineage, which describes how a column is created and used by other assets

Attaching column schema#

For assets defined in Dagster#

Column schema metadata can be attached to Dagster assets either as definition metadata or materialization metadata, which will then be visible in the Dagster UI. For example:

If the schema of your asset is pre-defined, you can attach it as definition metadata. If the schema is only known when an asset is materialized, you can attach it as metadata to the materialization.

To attach schema metadata to an asset, you will need to:

1. Construct a TableSchema object with TableColumn entries describing each column in the table
2. Attach the TableSchema object to the asset as part of the metadata parameter under the dagster/column_schema key. This can be attached to your asset definition, or to the MaterializeResult object returned by the asset function.

Below are two examples of how to attach column schema metadata to an asset, one as definition metadata and one as materialization metadata:

from dagster import AssetKey, MaterializeResult, TableColumn, TableSchema, asset


# Definition metadata
# Here, we know the schema of the asset, so we can attach it to the asset decorator
@asset(
    deps=[AssetKey("source_bar"), AssetKey("source_baz")],
    metadata={
        "dagster/column_schema": TableSchema(
            columns=[
                TableColumn(
                    "name",
                    "string",
                    description="The name of the person",
                ),
                TableColumn(
                    "age",
                    "int",
                    description="The age of the person",
                ),
            ]
        )
    },
)
def my_asset(): ...


# Materialization metadata
# Here, the schema isn't known until runtime
@asset(deps=[AssetKey("source_bar"), AssetKey("source_baz")])
def my_other_asset():
    column_names = ...
    column_types = ...

    columns = [
        TableColumn(name, column_type)
        for name, column_type in zip(column_names, column_types)
    ]

    yield MaterializeResult(
        metadata={"dagster/column_schema": TableSchema(columns=columns)}
    )

The schema for my_asset will be visible in the Dagster UI.

For assets loaded from integrations#

Dagster's dbt integration enables automatically attaching column schema metadata to assets loaded from dbt models. Refer to the dbt documentation for more information.

Attaching row count#

Row count metadata can be attached to Dagster assets as materialization metadata to provide additional context about the number of rows in a materialized table. This will be highlighted in the Dagster UI. For example:

In addition to showing the latest row count, Dagster will let you track changes in the row count over time, and you can use this information to monitor data quality.

To attach row count metadata to an asset, you will need to attach a numerical value to the dagster/row_count key in the metadata parameter of the MaterializeResult object returned by the asset function. For example:

import pandas as pd

from dagster import AssetKey, MaterializeResult, asset


@asset(deps=[AssetKey("source_bar"), AssetKey("source_baz")])
def my_asset():
    my_df: pd.DataFrame = ...

    yield MaterializeResult(metadata={"dagster/row_count": 374})

Attaching column-level lineage#

Column lineage enables data and analytics engineers alike to understand how a column is created and used in your data platform. Refer to the Column-level lineage documentation for more information.

APIs in this guide#

Related#