[ui] Asset graph can now be filtered by compute kind.
[ui] When backfilling failed and missing partitions of assets, a “Preview” button allows you to see which ranges will be materialized.
[dagster-dbt] When running DAGSTER_DBT_PARSE_PROJECT_ON_LOAD=1 dagster dev in a new scaffolded project from dagster-dbt project scaffold, dbt logs from creating dbt artifacts to loading the project are now silenced.
[dagster-airbyte] Added a new connection_meta_to_group_fn argument which allows configuring loaded asset groups based on the connection’s metadata dict.
[dagster-k8s] Debug information about failed run workers in errors surfaced by run monitoring now includes logs from sidecar containers, not just the main dagster container.
The QueuedRunCoordinatorDaemon has been refactored to paginate over runs when applying priority sort and tag concurrency limits. Previously, it loaded all runs into memory causing large memory spikes when many runs were enqueued.
Callable objects can once again be used to back sensor definitions.
UPathIOManager has been updated to use the correct path delimiter when interacting with cloud storages from a Windows process.
In the default multiprocess executor, the STEP_WORKER_STARTED event now fires before importing code in line with the other executors.
During execution, skipping a step now takes precedence over “abandoning” it due to upstream failure. This is expected to substantially improve the “retry from failure” workflow when conditional branching is in use.
Fixed an issue where default config values set to EnvVar did not work properly.
Fixed an issue where resources which implemented IAttachDifferentObjectToOpContext would pass the incorrect object to schedules and sensors.
Fixed a bug that caused auto-materialize failures when using the materialize_on_cron rule with dynamically partitioned assets.
Fixed an issue where sensor ticks would sporadically fail with a StopIteration exception.
[ui] For a job launchpad with a large number of tabs, the “Remove all” option was pushed offscreen. This has been fixed.
[ui] The asset backfill page now correctly shows backfills that target only unpartitioned assets.
[ui] Launching an asset job that was defined without_checks no longer fails by attempting to include the checks.
[dagster-databricks] fix bug that caused crash when polling a submitted job that is still in the Databricks queue (due to concurrency limit).
Patched issue where the local compute log path exposed file content outside of the compute log base directory - thanks r1b!
[dagster-databricks] Added ability to authenticate using an Azure service principal and fix minor bugs involving authenticating with a service principal while DATABRICKS_HOST is set. Thanks @zyd14!
[ui] Dark mode is now available via the User Settings dialog, currently in an experimental state. By default, the app will use a “legacy” theme, closely matching our current colors. A new light mode theme is also available.
[ui] Asset graph group nodes can be collapsed/expanded by right clicking on the collapsed group node or the header of the expanded group node.
[ui] Asset graph group nodes can be all collapsed or all expanded by right clicking anywhere on the graph and selecting the appropriate action.
[ui] The tree view was removed from the asset graph.
[pipes] PipesLambdaClient, an AWS Lambda pipes client has been added to dagster_aws.
Fixed a performance regression introduced in the 1.5.10 release where auto-materializing multi-assets became slower.
When a Dagster Cloud agent starts up, it will now wait to display as Running on the Agents tab in the Dagster Cloud UI until it has launched all the code servers that it needs in order to serve requests.
Added a new MetadataValue.job metadata type, which can be used to link to a Dagster job from other objects in the UI.
[asset backfills] Previously, when partitions definitions were changed after backfill launch, the asset backfill page would be blank. Now, when partitions definitions are changed, the backfill page will display statuses by asset.
[dagster-bigquery, dagster-duckdb, dagster-snowflake]. The BigQuery, DuckDB, and Snowflake I/O Managers will now determine the schema (dataset for BigQuery) in the following order of precedence: schema metadata set on the asset or op, I/O manager schema/ dataset configuration, key_prefix set on the asset. Previously, all methods for setting the schema/dataset were mutually exclusive, and setting more than one would raise an exception.
[dagster-shell] Added option to exclude the shell command from logs.
[dagster-dbt] When running DAGSTER_DBT_PARSE_PROJECT_ON_LOAD=1 dagster dev in a new scaffolded project from dagster-dbt project scaffold, dbt artifacts for loading the project are now created in a static target/ directory.
Problematic inheritance that was causing pydantic warnings to be emitted has been corrected.
It's now possible to use the logger of ScheduleEvaluationContext when testing via build_schedule_context.
The metadata from a Failure exception is now hoisted up to the failure that culminates when retry limits are exceeded.
Fixed bug in which the second instance of an hour partition at a DST boundary would never be shown as “materialized” in certain UI views.
Fixed an issue where backfilling an hourly partition that occurred during a fall Daylight Savings Time transition sometimes raised an error.
[auto-materialize] Fix issue where assets which were skipped because required parent partitions did not exist would not be materialized once those partitions came into existence.
[dagster ecs] The exit code of failed containers is now included in the failure message.
[dagster pipes] The PipesK8sClient now correctly raises on failed containers.
[dagster pipes] Using pipes within ops instead of assets no longer enforces problematic constraints.
[helm] Added maxCatchupRuns and maxTickRetries configuration options for the scheduler in the Helm chart.
[embedded-elt] Fixed crashes for non-unicode logs.
[UI] Fixed an issue where the test sensor dialog for a sensor that targeted multiple jobs would claim that all of the runs were targeting the same job.
[UI] Asset keys, job names, and other strings in Dagster UI no longer truncate unnecessarily in Firefox in some scenarios
[UI] A larger “View prior events” button on the Asset > Partitions page makes it easier to see the historical materializations of a specific partition of an asset.
[asset-checks, dbt] Fixed a bug that that caused asset checks to not execute when a run was not a subset. As part of the fix, the default dbt selection selection string will not be used for dbt runs, even when not in a subset. Instead we pass the explicit set of models and tests to execute, with DBT_INDIRECT_SELECTION=empty.
[asset-checks] Fixed a bug that caused asset checks defined with @asset(check_specs=... to not cooperate with the key_prefix argument of the load_assets_from_modules method and it’s compatriots.
[asset-checks] Fixed a bug that caused errors when launching a job from the UI that excluded asset checks.
[asset-checks] Fixed a bug that caused UI errors when a check run was deleted.
Marked the experimental Airbyte ingestion-as-code feature as deprecated, to be removed in a future release. We suggest users interested in managing their Airbyte connections in code use the Airbyte terraform provider.
define_asset_job now accepts an op_retry_policy argument, which specifies a default retry policies for all of the ops in the job. (thanks Eugenio Contreras!)
Fix IOManager not being able to load assets with MultiPartitionsDefinition - thanks @cyberosa!
[dagster-essentials] Three typo fixes in Lesson 8 - thanks Colton @cmpadden!
The observable_source_asset decorator now accepts a key argument.
[dagster pipes] an implicit_materializations argument has been added to get_results and get_materialize_result to control whether an implicit materialization event is created or not.
[embedded-elt] Added a new builder and SlingConnectionResource to allow reusing sources and targets interoperably.
[UI] Updated the experimental concurrency limits configuration page to show per-op runtime info and control.
[UI] The Auto-materialize history tab for each asset now only includes rows for evaluations where the result of evaluating the policy has changed. Previously, it would also show a row in the table representing periods of time where nothing changed.
[asset-checks, dbt] build_dbt_asset_selection now also selects asset checks based on their underlying dbt tests. E.g. build_dbt_asset_selection([my_dbt_assets], dbt_select="tag:data_quality") will select the assets and checks for any models and tests tagged with ‘data_quality’.
Branch deployments now use the same timeouts for starting and canceling runs that are set for their parent full deployment, instead of a fixed value of 10 minutes.
[k8s agent] Setting labels on a code location will now apply those labels to the kubernetes deployment and service for that code location, rather than just applying them to the pod for that code location.
[ui] Enabled collapsing asset groups in the global asset view when the new experimental asset graph is turned on in User Settings
[ui] The experimental asset graph sidebar now supports keyboard navigation via arrow keys
[ui] You can now right click nodes in the asset graph to materialize them or filter the graph
[ui] Jobs can now be searched by run ID
[ui] You can now launch runs from the job actions menu
[auto-materialize] A new AutoMaterializeRule.materialize_on_cron() rule makes it possible to create policies which materialize assets on a regular cadence.
[auto-materialize] If a partition-mapping-related error occurs within the Asset Daemon, a more informative error message will be provided.
[dagster-databricks] Extended the set of available config options to the Databricks step launcher - thanks @zyd14!
Fixed an issue where some schedules incorrectly skipped ticks during Daylight Savings Times transitions.
Returning a SensorResult from a sensor no longer overwrites a cursor if it was set via the context.
Fixed issue which could cause incorrect execution order when executing multi-assets with can_subset=True alongside assets which were upstream of some assets in the multi-asset, and downstream of others.
Previously, when creating an HourlyPartitionsDefinition with a non-UTC timezone and the default format string (or any format string not including a UTC-offset), there was no way to disambiguate between the first and second instance of the repeated hour during a daylight saving time transition. Now, for the one hour per year in which this ambiguity exists, the partition key of the second instance of the hour will have the UTC offset automatically appended to it.
[asset checks] Fixed a bug that caused an error when passing check_specs to AssetsDefinition.from_graph
[dagster-dbt] Fixed a bug in dagster-dbt that caused some dbt tests to not be selected as asset checks.
[dagster-dbt] Fixed an issue where multiple copies of the dbt manifest were held in memory when loading a dbt project as software-defined assets.
The email_on_failure sensor called deprecated methods on the context. This has been fixed
DagsterInstance.report_runless_asset_event is now public.
AutoMaterializeRule.materialize_on_parent_updated now accepts an updated_parents_filter of type AutoMaterializeAssetPartitionsFilter, which allows only materializing based on updates from runs with a required set of tags.
Fixed an error when trying to directly invoke a run status sensor when passing resources.
[dagster-airbyte][dagster-fivetran] Fixed an issue where EnvVars used in Airbyte or Fivetran resources would show up as their processed values in the launchpad when loading assets from a live Fivetran or Airbyte instance.