- Added a new
MetadataValue.job metadata type, which can be used to link to a Dagster job from other objects in the UI. - [asset backfills] Previously, when partitions definitions were changed after backfill launch, the asset backfill page would be blank. Now, when partitions definitions are changed, the backfill page will display statuses by asset.
- [dagster-bigquery, dagster-duckdb, dagster-snowflake]. The BigQuery, DuckDB, and Snowflake I/O Managers will now determine the schema (dataset for BigQuery) in the following order of precedence:
schema metadata set on the asset or op, I/O manager schema/ dataset configuration, key_prefix set on the asset. Previously, all methods for setting the schema/dataset were mutually exclusive, and setting more than one would raise an exception. - [dagster-shell] Added option to exclude the shell command from logs.
- [dagster-dbt] When running
DAGSTER_DBT_PARSE_PROJECT_ON_LOAD=1 dagster dev in a new scaffolded project from dagster-dbt project scaffold, dbt artifacts for loading the project are now created in a static target/ directory.
- Problematic inheritance that was causing pydantic warnings to be emitted has been corrected.
- It's now possible to use the logger of
ScheduleEvaluationContext when testing via build_schedule_context. - The
metadata from a Failure exception is now hoisted up to the failure that culminates when retry limits are exceeded. - Fixed bug in which the second instance of an hour partition at a DST boundary would never be shown as “materialized” in certain UI views.
- Fixed an issue where backfilling an hourly partition that occurred during a fall Daylight Savings Time transition sometimes raised an error.
- [auto-materialize] Fix issue where assets which were skipped because required parent partitions did not exist would not be materialized once those partitions came into existence.
- [dagster ecs] The exit code of failed containers is now included in the failure message.
- [dagster pipes] The
PipesK8sClient now correctly raises on failed containers. - [dagster pipes] Using pipes within ops instead of assets no longer enforces problematic constraints.
- [helm] Added
maxCatchupRuns and maxTickRetries configuration options for the scheduler in the Helm chart. - [embedded-elt] Fixed crashes for non-unicode logs.
- [UI] Fixed an issue where the test sensor dialog for a sensor that targeted multiple jobs would claim that all of the runs were targeting the same job.
- [UI] Asset keys, job names, and other strings in Dagster UI no longer truncate unnecessarily in Firefox in some scenarios
- [UI] A larger “View prior events” button on the Asset > Partitions page makes it easier to see the historical materializations of a specific partition of an asset.
- [asset-checks, dbt] Fixed a bug that that caused asset checks to not execute when a run was not a subset. As part of the fix, the default dbt selection selection string will not be used for dbt runs, even when not in a subset. Instead we pass the explicit set of models and tests to execute, with
DBT_INDIRECT_SELECTION=empty. - [asset-checks] Fixed a bug that caused asset checks defined with
@asset(check_specs=... to not cooperate with the key_prefix argument of the load_assets_from_modules method and it’s compatriots. - [asset-checks] Fixed a bug that caused errors when launching a job from the UI that excluded asset checks.
- [asset-checks] Fixed a bug that caused UI errors when a check run was deleted.
- Marked the experimental Airbyte ingestion-as-code feature as deprecated, to be removed in a future release. We suggest users interested in managing their Airbyte connections in code use the Airbyte terraform provider.
define_asset_job now accepts an op_retry_policy argument, which specifies a default retry policies for all of the ops in the job. (thanks Eugenio Contreras!)- Fix IOManager not being able to load assets with MultiPartitionsDefinition - thanks @cyberosa!
- [dagster-essentials] Three typo fixes in Lesson 8 - thanks Colton @cmpadden!
- The
observable_source_asset decorator now accepts a key argument. - [dagster pipes] an
implicit_materializations argument has been added to get_results and get_materialize_result to control whether an implicit materialization event is created or not. - [embedded-elt] Added a new builder and
SlingConnectionResource to allow reusing sources and targets interoperably. - [UI] Updated the experimental concurrency limits configuration page to show per-op runtime info and control.
- [UI] The Auto-materialize history tab for each asset now only includes rows for evaluations where the result of evaluating the policy has changed. Previously, it would also show a row in the table representing periods of time where nothing changed.
- [asset-checks, dbt]
build_dbt_asset_selection now also selects asset checks based on their underlying dbt tests. E.g. build_dbt_asset_selection([my_dbt_assets], dbt_select="tag:data_quality") will select the assets and checks for any models and tests tagged with ‘data_quality’.
- Branch deployments now use the same timeouts for starting and canceling runs that are set for their parent full deployment, instead of a fixed value of 10 minutes.
- [k8s agent] Setting labels on a code location will now apply those labels to the kubernetes deployment and service for that code location, rather than just applying them to the pod for that code location.
- [ui] Enabled collapsing asset groups in the global asset view when the new experimental asset graph is turned on in User Settings
- [ui] The experimental asset graph sidebar now supports keyboard navigation via arrow keys
- [ui] You can now right click nodes in the asset graph to materialize them or filter the graph
- [ui] Jobs can now be searched by run ID
- [ui] You can now launch runs from the job actions menu
- [auto-materialize] A new
AutoMaterializeRule.materialize_on_cron() rule makes it possible to create policies which materialize assets on a regular cadence. - [auto-materialize] If a partition-mapping-related error occurs within the Asset Daemon, a more informative error message will be provided.
- [dagster-databricks] Extended the set of available config options to the Databricks step launcher - thanks @zyd14!
- Fixed an issue where some schedules incorrectly skipped ticks during Daylight Savings Times transitions.
- Returning a
SensorResult from a sensor no longer overwrites a cursor if it was set via the context. - Fixed issue which could cause incorrect execution order when executing multi-assets with
can_subset=True alongside assets which were upstream of some assets in the multi-asset, and downstream of others. - Previously, when creating an
HourlyPartitionsDefinition with a non-UTC timezone and the default format string (or any format string not including a UTC-offset), there was no way to disambiguate between the first and second instance of the repeated hour during a daylight saving time transition. Now, for the one hour per year in which this ambiguity exists, the partition key of the second instance of the hour will have the UTC offset automatically appended to it. - [asset checks] Fixed a bug that caused an error when passing
check_specs to AssetsDefinition.from_graph - [dagster-dbt] Fixed a bug in
dagster-dbt that caused some dbt tests to not be selected as asset checks. - [dagster-dbt] Fixed an issue where multiple copies of the dbt manifest were held in memory when loading a dbt project as software-defined assets.
- The
email_on_failure sensor called deprecated methods on the context. This has been fixed
- [dagster-deltalake] Added Delta Lake support along with support for pandas and polars. Thanks Robert Pack @roeap!
- [dagster-graphql] Fixed the asset cursor format to use a normalized format - thanks @sisidra!
- [dagster-databricks] Extended the set of available config options to the Databricks step launcher - thanks @zyd14!
DagsterInstance.report_runless_asset_event is now public.AutoMaterializeRule.materialize_on_parent_updated now accepts an updated_parents_filter of type AutoMaterializeAssetPartitionsFilter, which allows only materializing based on updates from runs with a required set of tags.
- Reporting runless events and manually marking an asset as successfully materialized are no possible with “Launcher” level permissions
- [ui] Improved search and render performance of Users page, especially for large lists of users.
- [billing] Fixed issues with correctly displaying your tax ID
- Fixed an error when trying to directly invoke a run status sensor when passing resources.
- [dagster-airbyte][dagster-fivetran] Fixed an issue where
EnvVars used in Airbyte or Fivetran resources would show up as their processed values in the launchpad when loading assets from a live Fivetran or Airbyte instance.
- Substantially improved performance of the Dagster insights DBT/Snowflake usage job.
- The
OpExecutionContext and AssetExecutionContext now have a partition_keys property - [dagster-ui] The asset graph layout algorithm has been changed to a much faster one called “tight-tree”
- [dagster-ui] The Runs table filters has a top level filter for partitions
- [dagster-dbt]
dbt-core==1.7.* is now supported.
- Fixed an issue where some schedules skipped a tick on the day after a fall Daylight Savings Time transition.
- Fixed a bug that caused backfill policies that execute multiple partitions in a single run not to work with dynamic partitions.
- Fixed a bug that caused an error when
build_schedule_from_partitioned_job was used with a job with multi-partitioned assets and the partitions_def argument wasn’t provided to define_asset_job. - We now raise an error early if the empty string is provided as an asset’s group name (Thanks Sierrra!)
- Fixed an issue where custom setup and teardown methods were not properly called on nested Pythonic resources.
- Added a warning message when op or asset config is passed as an argument not named
config. - [dagster-cloud] Fixed an issue where overriding the default I/O manager could break the Snowflake-dbt insights job.
- [auto-materialize] Fixed an issue where materializing an unpartitioned parent of a dynamic-partitioned asset would only result in the latest dynamic partition of that asset being requested. Now, all partitions will be requested.
- [dagster-embedded-elt] Fixed an issue in
dagster-embedded-elt where sling’s updated_at parameter was set to the incorrect type - [dagster-ui] Fixed an issue in the launchpad where selecting a partition wouldn’t correctly overwrite fields using the partition’s specific configuration
- A docs fix to the testing concepts page, thanks @NicolaiLolansen!
- The schema can now be overridden per asset in DB IO managers, thanks @jrstats!
- Improved failure recovery and retry behavior when the daemon that launches auto-materialization runs fails or crashes in the middle of a tick.
- [asset-checks] UI performance for displaying check results is improved
- [asset-checks] Removed noisy experimental warning about
AssetCheckKey - [op-concurrency] Fixed a bug where concurrency slots were not getting assigned if a run that was assigned a slot was deleted before the slot had actually been claimed during execution.
- [dagster-pipes] The
PipesSubprocessClient now inherits the environment variables of the parent process in the launched subprocess. - [dagster-pipes] Exceptions are now reported in the event log for framework components and from the external process.
- [dagster-k8s] The
PipesK8sClient will now attempt to load the appropriate kubernetes config, and exposes arguments for controlling this process. - [ui] The launch asset backfill modal now offers a preview dialog showing the targeted asset partitions and their backfill policies when partition mapping or varying backfill policies are present.
- [asset-checks] New
load_asset_checks_from_modules functions for loading asset checks in tandem with load_assets_from_modules. - Previously, the daemon process would terminate with an error if it believed that a thread might be hanging, which sometimes caused undesirable terminations when doing large backfills or auto-materializing many assets. Now, the daemon process will emit a warning instead of terminate.
- [dagster-dbt]
dagster-dbt project scaffold now uses ~/.dbt/profiles.yml if a profiles.yml is not present in the dbt project directory. - [dagster-dbt]
@dbt_assets now support PartitionMapping using DagsterDbtTranslator.get_partition_mapping. - [dagster-dbt] Self dependencies can now be enabled for dbt models that are represented by
@dbt_assets. To enable this, add the following metadata to your dbt model’s metadata in your dbt project:
meta:
dagster:
has_self_dependency: True
- Fixed an issue where Dagster imports would throw errors when using
pydantic<2.0.0 but having pydantic-core installed. - Previously, asset backfills that targeted partitioned assets with a
LastPartitionMapping dependency would raise an error. This has been fixed. - Fixed a multipartitions partition mapping bug where a
instance is not available to load partitions error was raised. - [asset-checks] Fixed an issue with conflicting op names when using
build_asset_with_blocking_check - [ui] Viewing run logs containing very large messages no longer causes the UI to crash in Safari on macOS
- [ui] Drilling into the graph of a graph-backed asset with 4+ inputs or outputs no longer causes the asset graph to crash with a rendering error.
- [ui] On the backfill details page, clicking to a specific asset correctly shows the partitions that were materialized for that asset when partition mapping is in use.
- [ui] The Asset > Partition page now loads partition details more quickly in cases where calculating the staleness of the partition took a significant amount of time.
- Fixed a bug introduced in
1.5.0 where instances that haven’t been migrated to the latest schema hit an error upon run deletion. - [auto-materialize] Previously, if an asset partition was skipped on a tick for one reason, and then processed and skipped on a later tick for an additional reason, only the most recent skip reason would be tracked. Now, all skip reasons are tracked.
- [dagster-dbt] Fixed an issue where if an
exclude that didn’t match any dbt nodes was used in @dbt_assets, an error would be raised. The error is now properly handled. - [dagster-dbt] When invoking
DbtCliResource.cli(...) in an op, AssetMaterialization's instead of Output are now emitted.
- Global op concurrency slots are now released in between retries of op execution failures.
- Updated the tutorial to reflect best practices and APIs as of Dagster 1.5
- The
report_asset_observation REST endpoint for reporting runless events is now available.