Changelog#

1.9.8 (core) / 0.25.8 (libraries)#

Fixed a bug with load_assets_from_x functions where we began erroring when a spec and AssetsDefinition had the same key in a given module. We now only error in this case if include_specs=True.
[dagster-azure] Fixed a bug in 1.9.6 and 1.9.7 where the default behavior of the compute log manager switched from showing logs in the UI to showing a URL. You can toggle the show_url_only option to True to enable the URL showing behavior.
[dagster-dbt] Fixed an issue where group names set on partitioned dbt assets created using the @dbt_assets decorator would be ignored

Dagster code servers now wait to shut down until any calls that they are running have finished, preventing them from stopping while in the middle of executing sensor ticks or other long-running operations.
The dagster execute job cli now accepts —-op-selection (thanks @silent-lad!)
[ui] Option (Alt) + R now reloads all code locations (OSS only)

Adds a check to validate partition mappings when directly constructing AssetsDefinition instances.
Assets invoked in composition functions like @graph and @job now work again, fixing a regression introduced in 1.4.5.
Fixed an issue where a race condition with parallel runs materializing the same asset could cause a run to raise a RecursionError during execution.
Fixed an issue where including a resource in both a schedule and a job raised a “Cannot specify resource requirements” exception when the definitions were loaded.
The ins argument to graph_asset is now respected correctly.
Fixed an issue where the daemon process could sometimes stop with a heartbeat failure when the first sensor it ran took a long time to execute.
Fixed an issue where dagster dev failed on startup when the DAGSTER_GRPC_PORT `environment variable was set in the environment.
deps arguments for an asset can now be specified as an iterable instead of a sequence, allowing for sets to be passed.
[dagster-aws] Fixed a bug where the S3PickleIOManager didn’t correctly handle missing partitions when allow_missing_partitions was set. Thanks @o-sirawat!
[dagster-k8s] in the helm chart, the daemon securityContext setting now applies correctly to all init containers (thanks @maowerner!)

[dagster-databricks] Migrated to use new official databricks Python SDK. Thanks @judahrand!

New APIs for defining and executing checks on software-defined assets. These APIs are very early and subject to change. The corresponding UI has limited functionality. Docs
Adds a new auto-materialize skip rule AutoMaterializeRule.skip_on_not_all_parents_updated that enforces that an asset can only be materialized if all parents have been materialized since the asset's last materialization.
Exposed an auto-materialize skip rule – AutoMaterializeRule.skip_on_parent_missing –which is already part of the behavior of the default auto-materialize policy.
Auto-materialize evaluation history will now be stored for 1 month, instead of 1 week.
The auto-materialize asset daemon now includes more logs about what it’s doing for each asset in each tick in the Dagster Daemon process output.

Fixed an issue where the Docker agent would sometimes fail to load code locations with long names with a hostname connection error.

A new @partitioned_config decorator has been added for defined configuration for partitioned jobs. Thanks @danielgafni!
[dagster-aws] The ConfigurablePickledObjectS3IOManager has been renamed S3PickleIOManager for simplicity. The ConfigurablePickledObjecS3IOManager will continue to be available but is considered deprecated in favor of S3PickleIOManager. There is no change in the functionality of the I/O manager.
[dagster-azure] The ConfigurablePickledObjectADLS2IOManager has been renamed ADLS2PickleIOManager for simplicity. The ConfigurablePickledObjectADLS2IOManager will continue to be available but is considered deprecated in favor of ADLS2PickleIOManager. There is no change in the functionality of the I/O manager.
[dagster-dbt] When an exception is raised when invoking a dbt command using DbtCliResource, the exception message now includes a link to the dbt.log produced. This log file can be inspected for debugging.
[dagster-gcp] The ConfigurablePickledObjectGCSIOManager has been renamed GCSPickleIOManager for simplicity. The ConfigurablePickledObjecGCSIOManager will continue to be available but is considered deprecated in favor of GCSPickleIOManager. There is no change in the functionality of the I/O manager.

Fixed a bug that caused a DagsterInvariantViolationError when executing a multi-asset where both assets have self-dependencies on earlier partitions.
Fixed an asset backfill issue where some runs continue to be submitted after a backfill is requested for cancellation.
[dagster-dbt] Fixed an issue where using the --debug flag raised an exception in the Dagster framework.
[ui] “Launched run” and “Launched backfill” toasts in the Dagster UI behave the same way. To open in a new tab, hold the cmd/ctrl key when clicking “View”
[ui] When opening step compute logs, the view defaults to stderr which aligns with Python’s logging defaults.
[ui] When viewing a global asset graph with more than 100 assets, the “choose a subset to display” prompt is correctly aligned to the query input.

[dagster-graphql] The Dagster GraphQL Python client now includes a default timeout of 300 seconds for each query, to ensure that GraphQL requests don’t hang and never return a response. If you are running a query that is expected to take longer than 300 seconds, you can set the timeout argument when constructing a DagsterGraphQLClient.
[ui] We are continuing to improve the new horizontal rendering of the asset graph, which you can enable in Settings. This release increases spacing between nodes and improves the traceability of arrows on the graph.

Several Pythonic resources and I/O managers now have API docs entries.
Updated the tutorial’s example project and content to be more explicit about resources.
[dagster-dbt] Added API docs examples for DbtCliResource and DbtCliResource.cli(...).
Some code samples in API docs for InputContext and OutputContext have been fixed. Thanks @Sergey Mezentsev!

When setting up a new organization by importing a dbt project, using GitLab is now supported.

Added a respect_materialization_data_versions option to auto materialization. It can enabled in dagster.yaml with


auto_materialize:
  respect_materialization_data_versions: True

This flag may be changed or removed in the near future.