Added experimental functionality which hides user code errors from the Dagster UI. You may enable this functionality by setting the DAGSTER_REDACT_USER_CODE_ERRORS environment variable to 1.
[dagster-dbt] @dbt_assets now accepts a required_resource_keys argument.
Fixed a bug where a run that targets no steps is launched by an asset backfill when code updates are pushed after backfill launch time.
Previously a graphQL error would be thrown on the asset backfill page if certain unpartitioned assets were changed to a partitioned assets. This has been fixed.
[ui] Show run log timestamps in the user’s preferred hour cycle (12/24h) format.
[ui] The “Export to SVG” option now works as expected in the improved asset graph UI.
[ui] On the asset graph, hovering over a collapsed group or the title bar of an expanded group highlights all edges in/out of the group.
Fixed occasional CI/CD errors when building documentation on a feature branch
Asset lineage graph UI revamp, to make it easier to visualize and navigate large graphs
Lineage now flows left-to-right instead of top-to-bottom.
You can expand and collapse asset groups in the graph.
A new left-hand sidebar provides a list of assets, organized by asset group and code location.
You can right-click on assets or groups to filter or materialize them.
You can filter by compute kind.
Dark mode for the Dagster UI – By default, Dagster will match your system’s light or dark theme but you can adjust this in the user settings in the top right of the UI.
Report asset materializations from the UI – I.e. you record an asset materialization event without executing the code to materialize the asset. This is useful in cases where you overwrote data outside of Dagster, and you want Dagster to know about it and represent it in the UI. It’s also useful when you have a preexisting partitioned asset and start managing it with Dagster: you want Dagster to show the historical partitions as materialized instead of missing.
MaterializeResult, AssetSpec, and AssetDep now marked stable – These APIs, introduced in Dagster 1.5, were previously marked experimental. They offer a more straightforward way of defining assets when you don’t want to use I/O managers.
Backfill previews – When launching a backfill that covers assets with different partitions, can you now click “Preview” to see that partitions for each asset that will be covered by the backfill.
Viewing logs for a sensor or schedule tick is no longer considered experimental – previously, accessing this functionality required turning on a feature flag in user settings.
Runs triggered by a sensor or schedule link to the tick that triggered them.
Default op/asset concurrency limits are now configurable at the deployment level, using the concurrency > default_op_concurrency_limit configuration in your dagster.yaml (OSS) or Deployment Settings page (Dagster Cloud). In OSS, this feature first requires a storage migration (e.g. dagster instance migrate).
Zero-value op/asset concurrency limits are now supported. In OSS, this feature first requires a storage migration (e.g. dagster instance migrate).
When a Nothing-typed output is returned from an asset or op, the handle_output function of the I/O manager will no longer be called. Users of most Dagster-maintained I/O managers will see no behavioral changes, but users of the In-Memory I/O manager, or custom I/O managers that store Nothing-typed outputs should reference the migration guide for more information.
[ui] The updated asset graph is no longer behind an experimental flag. The new version features a searchable left sidebar, a horizontal DAG layout, context menus and collapsible groups!
Previously, if a code location was re-deployed with modified assets during an iteration of the asset daemon, empty auto-materialize runs could be produced. This has been fixed.
The CLI command dagster asset materialize will now return a non-zero exit code upon failure.
[ui] The Dagster UI now shows resource descriptions as markdown instead of plain text.
[ui] Viewing stdout/stderr logs for steps emitting hundreds of thousands of messages is much more performant and does not render the Run page unusable.
[ui] Fixed an issue where sensors with intervals that were less than 30 seconds were shown with an interval of “~30s” in the UI. The correct interval is now shown.
[dagster-graphql] Fixed an issue where the GraphQL Python client raised an unclear error if the request failed due to a permissions error.
A slight change has been made to run status sensors cursor values for Dagster instance using the default SQLite storage implementation. If you are using the default SQLite storage and you are upgrading directly from a version of dagster<1.5.1, you may see the first tick of your run status sensor skip runs that completed but were not yet registered by the sensor during your upgrade. This should not be common, but to avoid any chance of that, you may consider an interim upgrade to dagster>=1.5.1,<1.6.0 first.
Added the ability to annotate code locations with custom agent queues, allowing you to route requests for code locations in a single deployment to different agents. For example, you can route requests for one code location to an agent running in an on-premise data center but requests for all other code locations to another agent running in the cloud. For more information, see the docs.
Viewing logs for a sensor or schedule tick is now a generally available feature.
The feature flag to view sensor or schedule tick logs has been removed, as the feature is now enabled by default.
Logs can now be viewed even when the sensor or schedule tick fails.
The logs are now viewable in the sensor or schedule tick modal.
graph_multi_assets can now accept inputs as kwargs.
[ui] The tick timeline for schedules and sensors now defaults to showing all ticks, instead of excluding skipped ticks. The previous behavior can be enabled by unchecking the “Skipped” checkbox below the timeline view.
[ui] The updated asset graph is no longer behind an experimental flag. The new version features a searchable left sidebar, a horizontal DAG layout, context menus and collapsible groups!
[ui] Fix layout and scrolling issues that arise when a global banner alert is displayed in the app.
[ui] Use a larger version of the run config dialog in the Runs list in order to maximize the amount of visible config yaml.
[ui] When a software-defined asset is removed from a code location, it will now also be removed from global search.
[ui] When selecting assets in the catalog, you can now opt to materialize only “changed and missing” items in your selection.
[ui] The “Open in Launchpad” option on asset run pages has been updated to link to the graph of assets or asset job instead of an unusable launchpad page.
[ui] Partition status dots of multi-dimensional assets no longer wrap on the Asset > Partitions page.
[asset checks] Fixed a bug that caused the resource_defs parameter of @asset_check to not be respected
[ui] Fixed an issue where schedules or sensors with the same name in two different code locations sometimes showed each others runs in the list of runs for that schedule or sensor.
[pipes] Fixed an issue with the PipesFileMessageReader that could cause a crash on Windows.
Previously, calling context.log in different threads within a single op could result in some of those log messages being dropped. This has been fixed (thanks @quantum-byte!)
[dagster-dbt] On Dagster run termination, the dbt subprocess now exits gracefully to terminate any inflight queries that are materializing models.
The file_manager property on OpExecutionContext and AssetExecutionContext has been removed. This is an ancient property that was deprecated prior to Dagster 1.0, and since then had been raising a NotImplementedError whenever invoked.
[dagster-deltalake] Fixed an error when passing non-string valued options and extended the supported data types by the arrow type handler to support pyarrow datasets which allows for lazily loading delta tables. Thanks @roeap!
[dagster-pipes] The subprocess and databricks clients now forward termination to the external process if the orchestration process is terminated. A forward_termination argument is available for opting out.
The UI now correctly displays failed partitions after a single-run backfill occurs. Previously, if a single-run backfill failed, the corresponding partitions would not display as failed.
Several performance improvements when submitting Snowflake metrics to Dagster Cloud Insights.
Fixed an error which would occur when submitting Snowflake metrics for a removed or renamed asset to Dagster Cloud Insights.
The SensorEvaluationContext object has two new properties: last_sensor_start_time and is_first_tick_since_sensor_start. This enables sensor evaluation functions to vary behavior on the first tick vs subsequent ticks after the sensor has started.
The asset_selection argument to @sensor and SensorDefinition now accepts sequence of AssetsDefinitions, a sequences of strings, or a sequence of AssetKeys, in addition to AssetSelections.
[dagster-dbt] Support for dbt-core==1.3.* has been removed.
[ui] In code locations view, link to git repo when it’s a valid URL.
[ui] To improve consistency and legibility, when displaying elapsed time, most places in the app will now no longer show milliseconds.
[ui] Runs that were launched by schedules or sensors now show information about the relevant schedule or sensor in the header, with a link to view other runs associated with the same tick.
[dagster-gcp] Added a show_url_only parameter to GCSComputeLogManager that allows you to configure the compute log manager so that it displays a link to the GCS console rather than loading the logs from GCS, which can be useful if giving Dagster access to GCS credentials is undesirable.
Fixed a bug where dagster would error on FIPS-enabled systems by explicitly marking callsites of hashlib.md5 as not used for security purposes (Thanks @jlloyd-widen!)
[dagster-k8s] Changed execute_k8s_job to be aware of run-termination and op failure by deleting the executing k8s job (Thanks @Taadas!).
[dagstermill] Fixed dagstermill integration with the Dagster web UI to allow locally-scoped static resources (required to show certain frontend-components like plotly graphs) when viewing dagstermill notebooks (Thanks @aebrahim!).
[dagster-dbt] Fixed type annotation typo in the DbtCliResource API docs (Thanks @akan72!)
[pipes] Methods have been added to facilitate passing non-Dagster data back from the external process (report_custom_message ) to the orchestration process (get_custom_messages).
[ui] Added a “System settings” option for UI theming, which will use your OS preference to set light or dark mode.