Expectation Results

In Dagster, solids can emit structured events to represent the results of a data quality test. The data quality event class is the ExpectationResult. To generate an expectation result, we can yield an ExpectationResult event in our solid.

This would involve changing the following solid:

expectation.py
@solid
def my_simple_solid(context, df):
    do_some_transform(df)
    return df

into something like this:

materialization.py
@solid
def my_expectation_solid(context, df):
    do_some_transform(df)
    yield ExpectationResult(success=len(df) > 0, description="ensure dataframe has rows")
    yield Output(df)

Attaching Metadata to the Expectation ResultΒΆ

Much like materialization events, there are a variety of types of metadata that can be associated with an expectation result event, all through the EventMetadataEntry class. Each expectation event optionally takes a list of metadata entries that are then displayed in the event log.

materialization.py
@solid
def my_metadata_expectation_solid(context, df):
    do_some_transform(df)
    yield ExpectationResult(
        success=len(df) > 0,
        description="ensure dataframe has rows",
        metadata_entries=[
            EventMetadataEntry.text("Text-based metadata for this event", label="text_metadata"),
            EventMetadataEntry.url("http://mycoolsite.com/url_for_my_data", label="dashboard_url"),
            EventMetadataEntry.float(1.0 * len(df), "row count"),
            EventMetadataEntry.float(calculate_bytes(df), "size (bytes)"),
        ],
    )
    yield Output(df)

Check our API docs for EventMetadataEntry for more details on they types of event metadata available.