The AssetSpec constructor now raises an error if an invalid group name is provided, instead of an error being raised when constructing the Definitions object.
dagster/relation_identifier metadata is now automatically attached to assets which are stored using a DbIOManager.
[ui] Streamlined the code location list view.
[ui] The “group by” selection on the Timeline Overview page is now part of the query parameters, meaning it will be retained when linked to directly or when navigating between pages.
[dagster-dbt] When instantiating DbtCliResource, the project_dir argument will now override the DBT_PROJECT_DIR environment variable if it exists in the local environment (thanks, @marijncv!).
[dagster-embedded-elt] dlt assets now generate rows_loaded metadata (thanks, @kristianandre!).
Fixed a bug where setting asset_selection=[] on RunRequest objects yielded from sensors using asset_selection would select all assets instead of none.
Fixed bug where the tick status filter for batch-fetched graphql sensors was not being respected.
[examples] Fixed missing assets in assets_dbt_python example.
[dagster-airbyte] Updated the op names generated for Airbyte assets to include the full connection ID, avoiding name collisions.
[dagster-dbt] Fixed issue causing dagster-dbt to be unable to load dbt projects where the adapter did not have a database field set (thanks, @dargmuesli!)
[dagster-dbt] Removed a warning about not being able to load the dbt.adapters.duckdb module when loading dbt assets without that package installed.
You may now wipe specific asset partitions directly from the execution context in user code by calling DagsterInstance.wipe_asset_partitions.
Dagster+ users with a "Viewer" role can now create private catalog views.
Fixed an issue where the default IOManager used by Dagster+ Serverless did not respect setting allow_missing_partitions as metadata on a downstream asset.
Fixed an issue where runs in Dagster+ Serverless that materialized partitioned assets would sometimes fail with an object has no attribute '_base_path' error.
[dagster-graphql] Fixed an issue where the statuses filter argument to the sensorsOrError GraphQL field was sometimes ignored when querying GraphQL for multiple sensors at the same time.
Updated multi-asset sensor definition to be less likely to timeout queries against the asset history storage.
Consolidated the CapturedLogManager and ComputeLogManager APIs into a single base class.
[ui] Added an option under user settings to clear client side indexeddb caches as an escape hatch for caching related bugs.
[dagster-aws, dagster-pipes] Added a new PipesECSClient to allow Dagster to interface with ECS tasks.
[dagster-dbt] Increased the default timeout when terminating a run that is running a dbt subprocess to wait 25 seconds for the subprocess to cleanly terminate. Previously, it would only wait 2 seconds.
[dagster-sdf] Increased the default timeout when terminating a run that is running an sdf subprocess to wait 25 seconds for the subprocess to cleanly terminate. Previously, it would only wait 2 seconds.
[dagster-sdf] Added support for caching and asset selection (Thanks, akbog!)
[dagster-dlt] Added support for AutomationCondition using DagsterDltTranslator.get_automation_condition() (Thanks, aksestok!)
[ui] Fixed a bug where in-progress runs from a backfill could not be terminated from the backfill UI.
[ui] Fixed a bug that caused an "Asset must be part of at least one job" error when clicking on an external asset in the asset graph UI
Fixed an issue where viewing run logs with the latest 5.0 release of the watchdog package raised an exception.
[ui] Fixed issue causing the “filter to group” action in the lineage graph to have no effect.
[ui] Fixed case sensitivity when searching for partitions in the launchpad.
[ui] Fixed a bug which would redirect to the events tab for an asset if you loaded the partitions tab directly.
[ui] Fixed issue causing runs to get skipped when paging through the runs list (Thanks, @HynekBlaha!)
[ui] Fixed a bug where the asset catalog list view for a particular group would show all assets.
[dagster-dbt] fix bug where empty newlines in raw dbt logs were not being handled correctly.
[dagster-k8s, dagster-celery-k8s] Correctly set dagster/image label when image is provided from user_defined_k8s_config. (Thanks, @HynekBlaha!)
[dagster-duckdb] Fixed an issue for DuckDB versions older than 1.0.0 where an unsupported configuration option, custom_user_agent, was provided by default
[dagster-k8s] Fixed an issue where Kubernetes Pipes failed to create a pod if the op name contained capital or non-alphanumeric containers.
[dagster-embedded-elt] Fixed an issue where dbt assets downstream of Sling were skipped
[dagser-aws]: Direct AWS API arguments in PipesGlueClient.run have been deprecated and will be removed in 1.9.0. The new params argument should be used instead.
The default io_manager on Serverless now supports the allow_missing_partitions configuration option.
Fixed a bug that caused an error when loading the launchpad for a partition, when using in Dagster+ with an agent with version below 1.8.2
1.8.3 (core) / 0.24.3 (libraries) (YANKED - This version of Dagster resulted in errors when trying to launch runs that target individual asset partitions)#
When different assets within a code location have different PartitionsDefinitions, there will no longer be an implicit asset job __ASSET_JOB_... for each PartitionsDefinition; there will just be one with all the assets. This reduces the time it takes to load code locations with assets with many different PartitionsDefinitions.
[ui] Fixed a collection of broken links pointing to renamed Declarative Automation pages.
[dagster-dbt] Fixed issue preventing usage of MultiPartitionMapping with @dbt_assets (Thanks, @arookieds!)
[dagster-azure] Fixed issue that would cause an error when configuring an AzureBlobComputeLogManager without a secret_key (Thanks, @ion-elgreco and @HynekBlaha!)
[ui] For users whose light/dark mode theme setting is set to match their system setting, the theme will update automatically when the system changes modes (e.g. based on time of day), with no page reload required.
[ui] We have introduced the typefaces Geist and Geist Mono as our new default fonts throughout the Dagster app, with the goal of improving legibility, consistency, and maintainability.
[ui][experimental] We have begun experimenting with a new navigation structure for the Dagster UI. The change can be enabled via User Settings.
[ui][experimental] Made performance improvements to the Concurrency settings page.
Fixed an issue where pressing the “Reset sensor status” button in the UI would also reset the sensor’s cursor.
Fixed a bug that caused input loading time not to be included in the reported step duration.
Pydantic warnings are no longer raised when importing Dagster with Pydantic 2.0+.
Fixed an issue which would cause incorrect behavior when auto-materializing partitioned assets based on updates to a parent asset in a different code location.
Fixed an issue which would cause every tick of the auto-materialize sensor to produce an evaluation for each asset, even if nothing had changed from the previous tick.
[dagster-dbt] Fixed a bug that could raise Duplicate check specs errors with singular tests ingested as asset checks.
[embedded-elt] resolved an issue where subset of resources were not recognized when using source.with_resources(...)
[ui] Fixed an issue where a sensor that targeted an invalid set of asset keys could cause the asset catalog to fail to load.
[ui] Fixed an issue in which runs in the Timeline that should have been considered overlapping were not correctly grouped together, leading to visual bugs.
[ui] On the asset overview page, job tags no longer render poorly when an asset appears in several jobs.
[ui] On the asset overview page, hovering over the timestamp tags in the metadata table explains where each entry originated.
[ui] Right clicking the background of the asset graph now consistently shows a context menu, and the lineage view supports vertical as well as horizontal layout.
[ui][experimental] A new Overview > Asset Health page provides visibility into failed and missing materializations, check warnings and check errors.
[ui] You can now share feedback with the Dagster team directly from the app. Open the Help menu in the top nav, then “Share feedback”. Bugs and feature requests are submitted directly to the Dagster team.
[ui] When editing a team, the list of team members is now virtualized, allowing for the UI to scale better for very large team sizes.
Performance improvements when loading large asset graphs in the Dagster UI.
@asset_check functions can now be invoked directly for unit testing.
dagster-embedded-elt dlt resource DagsterDltResource can now be used from @op definitions in addition to assets.
UPathIOManager.load_partitions has been added to assist with helping UpathIOManager subclasses deal with serialization formats which support partitioning. Thanks @danielgafni!
[dagster-polars] now supports other data types rather than only string for the partitioning columns. Also PolarsDeltaIOManager now supports MultiPartitionsDefinition with DeltaLake native partitioning. Metadata value "partition_by": {"dim_1": "col_1", "dim_2": "col_2"} should be specified to enable this feature. Thanks @danielgafni!
[dagster-airbyte] Auto materialization policies passed to load_assets_from_airbyte_instance and load_assets_from_airbyte_project will now be properly propagated to the created assets.
Fixed an issue where deleting a run that was intended to materialize a partitioned asset would sometimes leave the status of that asset as “Materializing” in the Dagster UI.
Fixed an issue with build_time_partition_freshness_checks where it would incorrectly intuit that an asset was not fresh in certain cases.
[dagster-k8s] Fix an error on transient ‘none’ responses for pod waiting reasons. Thanks @piotrmarczydlo!
[dagster-dbt] Failing to build column schema metadata will now result in a warning rather than an error.
Fixed an issue where incorrect asset keys would cause a backfill to fail loudly.
Fixed an issue where syncing unmaterialized assets could include source assets.
[dagster-polars] PolarsDeltaIOManager no longer supports loading natively partitioned DeltaLake tables as dictionaries. They should be loaded as a single pl.DataFrame/pl.LazyFrame instead.
Branch deployments now have Change Tracking. Assets in each branch deployment will be compared to the main deployment. New assets and changes to code version, dependencies, partitions definitions, tags, and metadata will be marked in the UI of the branch deployment.
Pagerduty alerting is now supported with Pro plans. See the documentation for more info.
Asset metadata is now included in the insights metrics for jobs materializing those assets.
Per-run Insights are now available on individual assets.
Previously, the before_storage_id / after_storage_id values in the AssetRecordsFilter class were ignored. This has been fixed.
Updated the output of dagster-cloud deployment alert-policies list to match the format of sync.
Fixed an issue where Dagster Cloud agents with many code locations would sometimes leave code servers running after the agent shut down.
[dagster-dbt][experimental] A new cli command dagster-dbt project prepare-for-deployment has been added in conjunction with DbtProject for managing the behavior of rebuilding the manifest during development and preparing a pre-built one for production.
Fixed an issue with duplicate asset check keys when loading checks from a package.
A bug with the new build_last_update_freshness_checks and build_time_partition_freshness_checks has been fixed where multi_asset checks passed in would not be executable.
[dagster-dbt] Fixed some issues with building column lineage for incremental models, models with implicit column aliases, and models with columns that have multiple dependencies on the same upstream column.
Major Changes since 1.6.0 (core) / 0.22.0 (libraries)#
Asset definitions can now have tags, via the tags argument on @asset, AssetSpec, and AssetOut. Tags are meant to be used for organizing, filtering, and searching for assets.
The Asset Details page has been revamped to include an “Overview” tab that centralizes the most important information about the asset – such as current status, description, and columns – in a single place.
Assets can now be assigned owners.
Asset checks are now considered generally available and will no longer raise experimental warnings when used.
Asset checks can now be marked blocking, which causes downstream assets in the same run to be skipped if the check fails with ERROR-level severity.
The new @multi_asset_check decorator enables defining a single op that executes multiple asset checks.
The new build_last_updated_freshness_checks and build_time_partition_freshness_checks APIs allow defining asset checks that error or warn when an asset is overdue for an update. Refer to the Freshness checks guide for more info.
The new build_column_schema_change_checks API allows defining asset checks that warn when an asset’s columns have changed since its latest materialization.
In the asset graph UI, the “Upstream data”, “Code version changed”, and “Upstream code version” statuses have been collapsed into a single “Unsynced” status. Clicking on “Unsynced” displays more detailed information.
I/O managers are now optional. This enhances flexibility for scenarios where they are not necessary. For guidance, see When to use I/O managers.
Assets with None or MaterializeResult return type annotations won't use I/O managers; dependencies for these assets can be set using the deps parameter in the @asset decorator.
[dagster-dbt] dbt tests are now pulled in as Dagster asset checks by default.
[dagster-dbt] dbt resource tags are now automatically pulled in as Dagster asset tags.
[dagster-snowflake][dagster-gcp] The dagster-snowflake and dagster-gcp packages now both expose a fetch_last_updated_timestamps API, which makes it straightforward to collect data freshness information in source asset observation functions.
Changes since 1.6.14 (core) / 0.22.14 (libraries)#
Metadata attached during asset or op execution can now be accessed in the I/O manager using OutputContext.output_metadata.
[experimental] Single-run backfills now support batched inserts of asset materialization events. This is a major performance improvement for large single-run backfills that have database writes as a bottleneck. The feature is off by default and can be enabled by setting the DAGSTER_EVENT_BATCH_SIZE environment variable in a code server to an integer (25 recommended, 50 max). It is only currently supported in Dagster Cloud and OSS deployments with a postgres backend.
[ui] The new Asset Details page is now enabled for new users by default. To turn this feature off, you can toggle the feature in the User Settings.
[ui] Queued runs now display a link to view all the potential reasons why a run might remain queued.
[ui] Starting a run status sensor with a stale cursor will now warn you in the UI that it will resume from the point that it was paused.
[asset-checks] Asset checks now support asset names that include ., which can occur when checks are ingested from dbt tests.
[dagster-dbt] The env var DBT_INDIRECT_SELECTION will no longer be set to empty when executing dbt tests as asset checks, unless specific asset checks are excluded. dagster-dbt will no longer explicitly select all dbt tests with the dbt cli, which had caused argument length issues.
[dagster-dbt] Singular tests with a single dependency are now ingested as asset checks.
[dagster-dbt] Singular tests with multiple dependencies must have the primary dependency must be specified using dbt meta.
{{
config(
meta={
'dagster': {
'ref': {
'name': <ref_name>,'package': ...# Optional, if included in the ref.'version': ...# Optional, if included in the ref.
},
}
}
)
}}
...
[dagster-dbt] Column lineage metadata can now be emitted when invoking dbt. See the documentation for details.
[experimental][dagster-embedded-elt] Add the data load tool (dlt) integration for easily building and integration dlt ingestion pipelines with Dagster.
[dagster-dbt][community-contribution] You can now specify a custom schedule name for schedules created with build_schedule_from_dbt_selection. Thanks @dragos-pop!
[helm][community-contribution] You can now specify a custom job namespace for your user code deployments. Thanks @tmatthews0020!
[dagster-polars][community-contribution] Column schema metadata is now integrated using the dagster-specific metadata key in dagster_polars. Thanks @danielgafni!
[dagster-datadog][community-contribution] Added datadog.api module to the DatadogClient resource, enabling direct access to API methods. Thanks @shivgupta!
Fixed a bug where run status sensors configured to monitor a specific job would trigger for jobs with the same name in other code locations.
Fixed a bug where multi-line asset check result descriptions were collapsed into a single line.
Fixed a bug that caused a value to show up under “Target materialization” in the asset check UI even when an asset had had observations but never been materialized.
Changed typehint of metadata argument on multi_asset and AssetSpec to Mapping[str, Any].
[dagster-snowflake-pandas] Fixed a bug introduced in 0.22.4 where column names were not using quote identifiers correctly. Column names will now be quoted.
[dagster-aws] Fixed an issue where a race condition where simultaneously materializing the same asset more than once would sometimes raise an Exception when using the s3_io_manager.
[ui] Fixed a bug where resizable panels could inadvertently be hidden and never recovered, for instance the right panel on the global asset graph.
[ui] Fixed a bug where opening a run with an op selection in the Launchpad could lose the op selection setting for the subsequently launched run. The op selection is now correctly preserved.
[community-contribution] Fixed dagster-polars tests by excluding Decimal types. Thanks @ion-elgreco!
[community-contribution] Fixed a bug where auto-materialize rule evaluation would error on FIPS-compliant machines. Thanks @jlloyd-widen!
[community-contribution] Fixed an issue where an excessive DeprecationWarning was being issued for a ScheduleDefinition passed into the Definitions object. Thanks @2Ryan09!
Creating a run with a custom non-UUID run_id was previously private and only used for testing. It will now raise an exception.
[community-contribution] Previously, calling get_partition_keys_in_range on a MultiPartitionsDefinition would erroneously return partition keys that were within the one-dimensional range of alphabetically-sorted partition keys for the definition. Now, this method returns the cartesian product of partition keys within each dimension’s range. Thanks, @mst!
Added AssetCheckExecutionContext to replace AssetExecutionContext as the type of the context param passed in to @asset_check functions. @asset_check was an experimental decorator.
[experimental] @classmethod decorators have been removed from dagster-embedded-slt.slingDagsterSlingTranslator
[dagster-dbt] @classmethod decorators have been removed from DagsterDbtTranslator.
[dagster-k8s] The default merge behavior when raw kubernetes config is supplied at multiple scopes (for example, at the instance level and for a particluar job) has been changed to be more consistent. Previously, configuration was merged shallowly by default, with fields replacing other fields instead of appending or merging. Now, it is merged deeply by default, with lists appended to each other and dictionaries merged, in order to be more consistent with how kubernetes configuration is combined in all other places. See the docs for more information, including how to restore the previous default merge behavior.
AssetSelection.keys() has been deprecated. Instead, you can now supply asset key arguments to AssetSelection.assets() .
Run tag keys with long lengths and certain characters are now deprecated. For consistency with asset tags, run tags keys are expected to only contain alpha-numeric characters, dashes, underscores, and periods. Run tag keys can also contain a prefix section, separated with a slash. The main section and prefix section of a run tag are limited to 63 characters.
AssetExecutionContext has been simplified. Op-related methods and methods with existing access paths have been marked deprecated. For a full list of deprecated methods see this GitHub Discussion.
The metadata property on InputContext and OutputContext has been deprecated and renamed to definition_metadata .
FreshnessPolicy is now deprecated. For monitoring freshness, use freshness checks instead. If you are using AutoMaterializePolicy.lazy(), FreshnessPolicy is still recommended, and will continue to be supported until an alternative is provided.
The Dagster Cloud agent will now monitor the code servers that it spins to detect whether they have stopped serving requests, and will automatically redeploy the code server if it has stopped responding for an extended period of time.
New additions and bugfixes in Insights:
Added per-metric cost estimation. Estimates can be added via the “Insights settings” button, and will appear in the table and chart for that metric.
Branch deployments are now included in the deployment filter control.
In the Deployments view, fixed deployment links in the data table.