Dagster Cloud Insights (Experimental)#
This feature is considered experimental.
Using Dagster Cloud Insights, you can gain visibility into historical usage and cost metrics such as Dagster Cloud run duration, credit usage, and failures. You can also integrate other external metrics into the Insights UI and export metrics from Dagster Cloud.
About Insights#
Launched in October 2023, Insights is currently an experimental feature. This means we’re still collecting feedback on how Insights works and could make changes based on that feedback.
Note: These changes may include breaking changes to some aspects of the feature, though we will attempt to avoid it.
Getting access#
Currently, Insights can only be enabled by the Dagster team. To request access, you can:
- Reach out to us directly in your company's Slack channel
- Request access in the Dagster Slack #dagster-insights channel
Note: While visualizing and exploring Dagster-generated metrics within the UI will be available to all cloud users, integrating external metrics into Insights is an Enterprise-only feature.
Future plans#
As part of the experimental release, out-of-the-box utilities for capturing external metadata are limited to Snowflake. Refer to the Integrating external metrics section for more info.
Additional data sources and capabilities for capturing arbitrary metadata, as well as native tooling for enabling these capabilities on pipelines using ops and jobs, will be released in the coming weeks.
Accessing Insights in the Dagster UI#
To access this feature, click Insights in the top navigation bar in the UI:
The left navigation panel on this page contains a list of available metrics. For each metric, the daily, weekly, or monthly aggregated values are displayed in the graph.
These metrics are updated on a daily basis.
Available metrics#
| Metric | Description |
|---|---|
| Dagster credits | The Dagster credit cost associated with computing this object. Dagster credits are charged for every step that is run, and for every asset that is materialized. For more information, refer to the pricing FAQ. |
| Compute duration | The time spent computing steps. For jobs that run steps in parallel, the compute duration may be longer than the wall clock time it takes for the run to complete. |
| Materializations | The number of asset materializations associated with computing this object. |
| Observations | The number of asset observations associated with computing this object. |
| Step failures | The number of times steps failed when computing this object. Note: Steps that retry and succeed are not included in this metric. |
| Step retries | The number of times steps were retried when computing this object. |
| Asset check warnings | The number of asset checks that produced warnings. |
| Asset check errors | The number of asset checks that produced errors. |
| Retry compute | The time spent computing steps, including time spent retrying failed steps. For jobs that run steps in parallel, the compute duration may be longer than the wall clock time it takes for the run to complete. |
Uses#
Integrating external metrics#
External metrics, such as Snowflake credits, can be integrated into Dagster Insights. Insights currently supports integrating Snowflake metrics for:
- Usage generated by queries made to Snowflake resources. Refer to the Integrating direct Snowflake usage with Dagster Cloud Insights guide for more info.
- Snowflake usage generated by dbt. Refer to the dbt + Snowflake + Dagster Cloud Insights guide for more info.
Exporting metrics#
Metrics in Dagster Cloud Insights can be exported using a GraphQL API endpoint. Refer to the Exporting Insights metrics from Dagster Cloud guide for details.