Fixed a bug with load_assets_from_x functions where we began erroring when a spec and AssetsDefinition had the same key in a given module. We now only error in this case if include_specs=True.
[dagster-azure] Fixed a bug in 1.9.6 and 1.9.7 where the default behavior of the compute log manager switched from showing logs in the UI to showing a URL. You can toggle the show_url_only option to True to enable the URL showing behavior.
[dagster-dbt] Fixed an issue where group names set on partitioned dbt assets created using the @dbt_assets decorator would be ignored
Fixed an error when trying to directly invoke a run status sensor when passing resources.
[dagster-airbyte][dagster-fivetran] Fixed an issue where EnvVars used in Airbyte or Fivetran resources would show up as their processed values in the launchpad when loading assets from a live Fivetran or Airbyte instance.
Fixed an issue where some schedules skipped a tick on the day after a fall Daylight Savings Time transition.
Fixed a bug that caused backfill policies that execute multiple partitions in a single run not to work with dynamic partitions.
Fixed a bug that caused an error when build_schedule_from_partitioned_job was used with a job with multi-partitioned assets and the partitions_def argument wasn’t provided to define_asset_job.
We now raise an error early if the empty string is provided as an asset’s group name (Thanks Sierrra!)
Fixed an issue where custom setup and teardown methods were not properly called on nested Pythonic resources.
Added a warning message when op or asset config is passed as an argument not named config.
[dagster-cloud] Fixed an issue where overriding the default I/O manager could break the Snowflake-dbt insights job.
[auto-materialize] Fixed an issue where materializing an unpartitioned parent of a dynamic-partitioned asset would only result in the latest dynamic partition of that asset being requested. Now, all partitions will be requested.
[dagster-embedded-elt] Fixed an issue in dagster-embedded-elt where sling’s updated_at parameter was set to the incorrect type
[dagster-ui] Fixed an issue in the launchpad where selecting a partition wouldn’t correctly overwrite fields using the partition’s specific configuration
Improved failure recovery and retry behavior when the daemon that launches auto-materialization runs fails or crashes in the middle of a tick.
[asset-checks] UI performance for displaying check results is improved
[asset-checks] Removed noisy experimental warning about AssetCheckKey
[op-concurrency] Fixed a bug where concurrency slots were not getting assigned if a run that was assigned a slot was deleted before the slot had actually been claimed during execution.
[dagster-pipes] The PipesSubprocessClient now inherits the environment variables of the parent process in the launched subprocess.
[dagster-pipes] Exceptions are now reported in the event log for framework components and from the external process.
[dagster-k8s] The PipesK8sClient will now attempt to load the appropriate kubernetes config, and exposes arguments for controlling this process.
[ui] The launch asset backfill modal now offers a preview dialog showing the targeted asset partitions and their backfill policies when partition mapping or varying backfill policies are present.
[asset-checks] New load_asset_checks_from_modules functions for loading asset checks in tandem with load_assets_from_modules.
Previously, the daemon process would terminate with an error if it believed that a thread might be hanging, which sometimes caused undesirable terminations when doing large backfills or auto-materializing many assets. Now, the daemon process will emit a warning instead of terminate.
[dagster-dbt] dagster-dbt project scaffold now uses ~/.dbt/profiles.yml if a profiles.yml is not present in the dbt project directory.
[dagster-dbt] @dbt_assets now support PartitionMapping using DagsterDbtTranslator.get_partition_mapping.
[dagster-dbt] Self dependencies can now be enabled for dbt models that are represented by @dbt_assets. To enable this, add the following metadata to your dbt model’s metadata in your dbt project:
Fixed an issue where Dagster imports would throw errors when using pydantic<2.0.0 but having pydantic-core installed.
Previously, asset backfills that targeted partitioned assets with a LastPartitionMapping dependency would raise an error. This has been fixed.
Fixed a multipartitions partition mapping bug where a instance is not available to load partitions error was raised.
[asset-checks] Fixed an issue with conflicting op names when using build_asset_with_blocking_check
[ui] Viewing run logs containing very large messages no longer causes the UI to crash in Safari on macOS
[ui] Drilling into the graph of a graph-backed asset with 4+ inputs or outputs no longer causes the asset graph to crash with a rendering error.
[ui] On the backfill details page, clicking to a specific asset correctly shows the partitions that were materialized for that asset when partition mapping is in use.
[ui] The Asset > Partition page now loads partition details more quickly in cases where calculating the staleness of the partition took a significant amount of time.
Fixed a bug introduced in 1.5.0 where instances that haven’t been migrated to the latest schema hit an error upon run deletion.
[auto-materialize] Previously, if an asset partition was skipped on a tick for one reason, and then processed and skipped on a later tick for an additional reason, only the most recent skip reason would be tracked. Now, all skip reasons are tracked.
[dagster-dbt] Fixed an issue where if an exclude that didn’t match any dbt nodes was used in @dbt_assets, an error would be raised. The error is now properly handled.
[dagster-dbt] When invoking DbtCliResource.cli(...) in an op, AssetMaterialization's instead of Output are now emitted.
Dagster now supports using Pydantic 2 models for Config and Resources. Pydantic 1.10 continues to be supported.
Added a report_asset_observation REST API endpoint for runless external asset observation events
Dramatically improved the performance of partition-mapping, for basic hourly and daily partitions definitions
[ui] When viewing a list of runs, you can quickly add the tag in the “Launched by” column as a filter on the list view. Hover over the tag to see the “Add to filter” button.
[helm] The env vars DAGSTER_K8S_PG_PASSWORD_SECRET and DAGSTER_K8S_INSTANCE_CONFIG_MAP will no longer be set in all pods.
[dagster-pyspark] build_pyspark_zip now takes an exclude parameter that can be used to customize the set of excluded files.
[ui] Links beginning with http://, https:// in unstructured run logs (via context.log) are automatically converted to clickable links
Added a report_asset_check REST API endpoint for runless external asset check evaluation events. This is available in cloud as well.
The config argument is now supported on @graph_multi_asset
[ui] Improved performance for global search UI, especially for deployments with very large numbers of jobs or assets.
[dagster-pipes] Add S3 context injector/reader.
[dagster-dbt] When an exception when running a dbt command, error messages from the underlying dbt invocation are now properly surfaced to the Dagster exception.
[dagster-dbt] The path to the dbt executable is now configurable in DbtCliResource.
Fixed a bug introduced in 1.5.3 that caused errors when launching specific Ops in a Job.
Fixed a bug introduced in 1.5.0 that prevented the AssetExecutionContext type annotation for the context parameter in @asset_check functions.
Fixed an issue where the Dagster scheduler would sometimes fail to retry a tick if there was an error reloading a code location in the middle of the tick.
[dagster-dbt] Fixed an issue where explicitly passing in profiles_dir=None into DbtCliResource would cause incorrect validation.
[dagster-dbt] Fixed an issue where partial parsing was not working when reusing existing target paths in subsequent dbt invocations.
[ui] Fixed an issue where the job partitions UI would show “0 total partitions” if the job consisted of more than 100 assets
[dagster-duckdb] The DuckDBResource and DuckDBIOManager accept a connection_config configuration that will be passed as config to the DuckDB connection. Thanks @xjhc!
Added events in the run log when a step is blocked by a global op concurrency limit.
Added a backoff for steps querying for open concurrency slots.
Auto-materialize logic to skip materializing when (1) a backfill is in progress or (2) parent partitions are required but nonexistent are now refactored to be skip rules.
[ui] Added 2 new asset graph layout algorithms under user settings that are significantly faster for large graphs (1000+ assets).
Running multiple agents is no longer considered experimental.
When the agent spins up a new code server while updating a code location, it will now wait until the new code location uploads any changes to Dagster Cloud before allowing the new server to serve requests.