Ask AI

Changelog#

1.9.6 (core) / 0.25.6 (libraries)#

New#

  • Updated cronitor pin to allow versions >= 5.0.1 to enable use of DayOfWeek as 7. Cronitor 4.0.0 is still disallowed. (Thanks, @joshuataylor!)
  • Added flag checkDbReadyInitContainer to optionally disable db check initContainer.
  • [ui] Added Google Drive icon for kind tags. (Thanks, @dragos-pop!)
  • [ui] Renamed the run lineage sidebar on the Run details page to Re-executions.
  • [ui] Sensors and schedules that appear in the Runs page are now clickable.
  • [ui] Runs targeting assets now show more of the assets in the Runs page.
  • [dagster-airbyte] The destination type for an Airbyte asset is now added as a kind tag for display in the UI.
  • [dagster-gcp] DataprocResource now receives an optional parameter labels to be attached to Dataproc clusters. (Thanks, @thiagoazcampos!)
  • [dagster-k8s] Added a checkDbReadyInitContainer flag to the Dagster Helm chart to allow disabling the default init container behavior. (Thanks, @easontm!)
  • [dagster-k8s] K8s pod logs are now logged when a pod fails. (Thanks, @apetryla!)
  • [dagster-sigma] Introduced build_materialize_workbook_assets_definition which can be used to build assets that run materialize schedules for a Sigma workbook.
  • [dagster-snowflake] SnowflakeResource and SnowflakeIOManager both accept additional_snowflake_connection_args config. This dictionary of arguments will be passed to the snowflake.connector.connect method. This config will be ignored if you are using the sqlalchemy connector.
  • [helm] Added the ability to set user-deployments labels on k8s deployments as well as pods.

Bugfixes#

  • Assets with self dependencies and BackfillPolicy are now evaluated correctly during backfills. Self dependent assets no longer result in serial partition submissions or disregarded upstream dependencies.
  • Previously, the freshness check sensor would not re-evaluate freshness checks if an in-flight run was planning on evaluating that check. Now, the freshness check sensor will kick off an independent run of the check, even if there's already an in flight run, as long as the freshness check can potentially fail.
  • Previously, if the freshness check was in a failing state, the sensor would wait for a run to update the freshness check before re-evaluating. Now, if there's a materialization later than the last evaluation of the freshness check and no planned evaluation, we will re-evaluate the freshness check automatically.
  • [ui] Fixed run log streaming for runs with a large volume of logs.
  • [ui] Fixed a bug in the Backfill Preview where a loading spinner would spin forever if an asset had no valid partitions targeted by the backfill.
  • [dagster-aws] PipesCloudWatchMessageReader correctly identifies streams which are not ready yet and doesn't fail on ThrottlingException. (Thanks, @jenkoian!)
  • [dagster-fivetran] Column metadata can now be fetched for Fivetran assets using FivetranWorkspace.sync_and_poll(...).fetch_column_metadata().
  • [dagster-k8s] The k8s client now waits for the main container to be ready instead of only waiting for sidecar init containers. (Thanks, @OrenLederman!)

Documentation#

  • Fixed a typo in the dlt_assets API docs. (Thanks, @zilto!)

1.9.5 (core) / 0.25.5 (libraries)#

New#

  • The automatic run retry daemon has been updated so that there is a single source of truth for if a run will be retried and if the retry has been launched. Tags are now added to run at failure time indicating if the run will be retried by the automatic retry system. Once the automatic retry has been launched, the run ID of the retry is added to the original run.
  • When canceling a backfill of a job, the backfill daemon will now cancel all runs launched by that backfill before marking the backfill as canceled.
  • Dagster execution info (tags such as dagster/run-id, dagster/code-location, dagster/user and Dagster Cloud environment variables) typically attached to external resources are now available under DagsterRun.dagster_execution_info.
  • SensorReturnTypesUnion is now exported for typing the output of sensor functions.
  • [dagster-dbt] dbt seeds now get a valid code version (Thanks @marijncv!).
  • Manual and automatic retries of runs launched by backfills that occur while the backfill is still in progress are now incorporated into the backfill's status.
  • Manual retries of runs launched by backfills are no longer considered part of the backfill if the backfill is complete when the retry is launched.
  • [dagster-fivetran] Fivetran assets can now be materialized using the FivetranWorkspace.sync_and_poll(…) method in the definition of a @fivetran_assets decorator.
  • [dagster-fivetran] load_fivetran_asset_specs has been updated to accept an instance of DagsterFivetranTranslator or custom subclass.
  • [dagster-fivetran] The fivetran_assets decorator was added. It can be used with the FivetranWorkspace resource and DagsterFivetranTranslator translator to load Fivetran tables for a given connector as assets in Dagster. The build_fivetran_assets_definitions factory can be used to create assets for all the connectors in your Fivetran workspace.
  • [dagster-aws] ECSPipesClient.run now waits up to 70 days for tasks completion (waiter parameters are configurable) (Thanks @jenkoian!)
  • [dagster-dbt] Update dagster-dbt scaffold template to be compatible with uv (Thanks @wingyplus!).
  • [dagster-airbyte] A load_airbyte_cloud_asset_specs function has been added. It can be used with the AirbyteCloudWorkspace resource and DagsterAirbyteTranslator translator to load your Airbyte Cloud connection streams as external assets in Dagster.
  • [ui] Add an icon for the icechunk kind.
  • [ui] Improved ui for manual sensor/schedule evaluation.

Bugfixes#

  • Fixed database locking bug for the ConsolidatedSqliteEventLogStorage, which is mostly used for tests.
  • [dagster-aws] Fixed a bug in the ECSRunLauncher that prevented it from accepting a user-provided task definition when DAGSTER_CURRENT_IMAGE was not set in the code location.
  • [ui] Fixed an issue that would sometimes cause the asset graph to fail to render on initial load.
  • [ui] Fix global auto-materialize tick timeline when paginating.

1.9.4 (core) / 0.25.4 (libraries)#

New#

  • Global op concurrency is now enabled on the default SQLite storage. Deployments that have not been migrated since 1.6.0 may need to run dagster instance migrate to enable.
  • Introduced map_asset_specs to enable modifying AssetSpecs and AssetsDefinitions in bulk.
  • Introduced AssetSpec.replace_attributes and AssetSpec.merge_attributes to easily alter properties of an asset spec.
  • [ui] Add a "View logs" button to open tick logs in the sensor tick history table.
  • [ui] Add Spanner kind icon.
  • [ui] The asset catalog now supports filtering using the asset selection syntax.
  • [dagster-pipes, dagster-aws] PipesS3MessageReader now has a new parameter include_stdio_in_messages which enables log forwarding to Dagster via Pipes messages.
  • [dagster-pipes] Experimental: A new Dagster Pipes message type log_external_stream has been added. It can be used to forward external logs to Dagster via Pipes messages.
  • [dagster-powerbi] Opts in to using admin scan APIs to pull data from a Power BI instance. This can be disabled by passing load_powerbi_asset_specs(..., use_workspace_scan=False).
  • [dagster-sigma] Introduced an experimental dagster-sigma snapshot command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.

Bugfixes#

  • Fixed a bug that caused DagsterExecutionStepNotFoundError errors when trying to execute an asset check step of a run launched by a backfill.
  • Fixed an issue where invalid cron strings like "0 0 30 2 *" that represented invalid dates in February were still allowed as Dagster cron strings, but then failed during schedule execution. Now, these invalid cronstrings will raise an exception when they are first loaded.
  • Fixed a bug where owners added to AssetOuts when defining a @graph_multi_asset were not added to the underlying AssetsDefinition.
  • Fixed a bug where using the & or | operators on AutomationConditions with labels would cause that label to be erased.
  • [ui] Launching partitioned asset jobs from the launchpad now warns if no partition is selected.
  • [ui] Fixed unnecessary middle truncation occurring in dialogs.
  • [ui] Fixed timestamp labels and "Now" line rendering bugs on the sensor tick timeline.
  • [ui] Opening Dagster's UI with a single job defined takes you to the Overview page rather than the Job page.
  • [ui] Fix stretched tags in backfill table view for non-partitioned assets.
  • [ui] Open automation sensor evaluation details in a dialog instead of navigating away.
  • [ui] Fix scrollbars in dark mode.
  • [dagster-sigma] Workbooks filtered using a SigmaFilter no longer fetch lineage information.
  • [dagster-powerbi] Fixed an issue where reports without an upstream dataset dependency would fail to translate to an asset spec.

Deprecations#

  • [dagster-powerbi] DagsterPowerBITranslator.get_asset_key is deprecated in favor of DagsterPowerBITranslator.get_asset_spec().key
  • [dagster-looker] DagsterLookerApiTranslator.get_asset_key is deprecated in favor of DagsterLookerApiTranslator.get_asset_spec().key
  • [dagster-sigma] DagsterSigmaTranslator.get_asset_key is deprecated in favor of DagsterSigmaTranslator.get_asset_spec().key
  • [dagster-tableau] DagsterTableauTranslator.get_asset_key is deprecated in favor of DagsterTableauTranslator.get_asset_spec().key

1.9.3 (core) / 0.25.3 (libraries)#

New#

  • Added run_id to the run_tags index to improve database performance. Run dagster instance migrate to update the index. (Thanks, @HynekBlaha!)

  • Added icons for kind tags: Cassandra, ClickHouse, CockroachDB, Doris, Druid, Elasticsearch, Flink, Hadoop, Impala, Kafka, MariaDB, MinIO, Pinot, Presto, Pulsar, RabbitMQ, Redis, Redpanda, ScyllaDB, Starrocks, and Superset. (Thanks, @swrookie!)

  • Added a new icon for the Denodo kind tag. (Thanks, @tintamarre!)

  • Errors raised from defining more than one Definitions object at module scope now include the object names so that the source of the error is easier to determine.

  • [ui] Asset metadata entries like dagster/row_count now appear on the events page and are properly hidden on the overview page when they appear in the sidebar.

  • [dagster-aws] PipesGlueClient now attaches AWS Glue metadata to Dagster results produced during Pipes invocation.

  • [dagster-aws] PipesEMRServerlessClient now attaches AWS EMR Serverless metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the job run.

  • [dagster-aws] PipesECSClient now attaches AWS ECS metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the ECS task.

  • [dagster-aws] PipesEMRClient now attaches AWS EMR metadata to Dagster results produced during Pipes invocation.

  • [dagster-databricks] PipesDatabricksClient now attaches Databricks metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the Databricks job.

  • [dagster-fivetran] Added load_fivetran_asset_specs function. It can be used with the FivetranWorkspace resource and DagsterFivetranTranslator translator to load your Fivetran connector tables as external assets in Dagster.

  • [dagster-looker] Errors are now handled more gracefully when parsing derived tables.

  • [dagster-sigma] Sigma assets now contain extra metadata and kind tags.

  • [dagster-sigma] Added support for direct workbook to warehouse table dependencies.

  • [dagster-sigma] Added include_unused_datasets field to SigmaFilter to disable pulling datasets that aren't used by a downstream workbook.

  • [dagster-sigma] Added skip_fetch_column_data option to skip loading Sigma column lineage. This can speed up loading large instances.

  • [dagster-sigma] Introduced an experimental dagster-sigma snapshot command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.

    Introducing: dagster-airlift (experimental)#

    dagster-airlift is coming out of stealth. See the initial Airlift RFC here, and the following documentation to learn more:

    More Airflow-related content is coming soon! We'd love for you to check it out, and post any comments / questions in the #airflow-migration channel in the Dagster slack.

Bugfixes#

  • Fixed a bug in run status sensors where setting incompatible arguments monitor_all_code_locations and monitored_jobs did not raise the expected error. (Thanks, @apetryla!)
  • Fixed an issue that would cause the label for AutomationCondition.any_deps_match() and AutomationCondition.all_deps_match() to render incorrectly when allow_selection or ignore_selection were set.
  • Fixed a bug which could cause code location load errors when using CacheableAssetsDefinitions in code locations that contained AutomationConditions
  • Fixed an issue where the default multiprocess executor kept holding onto subprocesses after their step completed, potentially causing Too many open files errors for jobs with many steps.
  • [ui] Fixed an issue introduced in 1.9.2 where the backfill overview page would sometimes display extra assets that were targeted by the backfill.
  • [ui] Fixed "Open in Launchpad" button when testing a schedule or sensor by ensuring that it opens to the correct deployment.
  • [ui] Fixed an issue where switching a user setting was immediately saved, rather than waiting for the change to be confirmed.
  • [dagster-looker] Unions without unique/distinct criteria are now properly handled.
  • [dagster-powerbi] Fixed an issue where reports without an upstream dataset dependency would fail to translate to an asset spec.
  • [dagster-sigma] Fixed an issue where API fetches did not paginate properly.

Documentation#

Dagster Plus#

  • [ui] Fixed an issue with filtering and catalog search in branch deployments.
  • [ui] Fixed an issue where the asset graph would reload unexpectedly.

1.9.2 (core) / 0.25.2 (libraries)#

New#

  • Introduced a new constructor, AssetOut.from_spec, that will construct an AssetOut from an AssetSpec.
  • [ui] Column tags are now displayed in the Column name section of the asset overview page.
  • [ui] Introduced an icon for the gcs (Google Cloud Storage) kind tag.
  • [ui] Introduced icons for report and semanticmodel kind tags.
  • [ui] The tooltip for a tag containing a cron expression now shows a human-readable, timezone-aware cron string.
  • [ui] Asset check descriptions are now sourced from docstrings and rendered in the UI. (Thanks, @marijncv!)
  • [dagster-aws] Added option to propagate tags to ECS tasks when using the EcsRunLauncher. (Thanks, @zyd14!)
  • [dagster-dbt] You can now implement DagsterDbtTranslator.get_code_version to customize the code version for your dbt assets. (Thanks, @Grzyblon!)
  • [dagster-pipes] Added the ability to pass arbitrary metadata to PipesClientCompletedInvocation. This metadata will be attached to all materializations and asset checks stored during the pipes invocation.
  • [dagster-powerbi] During a full workspace scan, owner and column metadata is now automatically attached to assets.

Bugfixes#

  • Fixed an issue with AutomationCondition.execution_in_progress which would cause it to evaluate to True for unpartitioned assets that were part of a run that was in progress, even if the asset itself had already been materialized.
  • Fixed an issue with AutomationCondition.run_in_progress that would cause it to ignore queued runs.
  • Fixed an issue that would cause a default_automation_condition_sensor to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy auto_materialize: use_sensors configuration setting was set to False.
  • [ui] Fixed an issue when executing asset checks where the wrong job name was used in some situations. The correct job name is now used.
  • [ui] Selecting assets with 100k+ partitions no longer causes the asset graph to temporarily freeze.
  • [ui] Fixed an issue that could cause a GraphQL error on certain pages after removing an asset.
  • [ui] The asset events page no longer truncates event history in cases where both materialization and observation events are present.
  • [ui] The backfill coordinator logs tab no longer sits in a loading state when no logs are available to display.
  • [ui] Fixed issue which would cause the "Partitions evaluated" label on an asset's automation history page to incorrectly display 0 in cases where all partitions were evaluated.
  • [ui] Fix "Open in Playground" link when testing a schedule or sensor by ensuring that it opens to the correct deployment.
  • [ui] Fixed an issue where the asset graph would reload unexpectedly.
  • [dagster-dbt] Fixed an issue where the SQL filepath for a dbt model was incorrectly resolved when the dbt manifest file was built on a Windows machine, but executed on a Unix machine.
  • [dagster-pipes] Asset keys containing embedded / characters now work correctly with Dagster Pipes.

Documentation#

Deprecations#

  • The types-sqlalchemy package is no longer included in the dagster[pyright] extra package.

Dagster Plus#

  • [ui] The Environment Variables table can now be sorted by name and update time.
  • [ui] The code location configuration dialog now contains more metadata about the code location.
  • [ui] Fixed an issue where the incorrect user icons were shown in the Users table when a search filter had been applied.

1.9.1 (core) / 0.25.1 (libraries)#

New#

  • dagster project scaffold now has an option to create dagster projects from templates with excluded files/filepaths.
  • [ui] Filters in the asset catalog now persist when navigating subdirectories.
  • [ui] The Run page now displays the partition(s) a run was for.
  • [ui] Filtering on owners/groups/tags is now case-insensitive.
  • [dagster-tableau] the helper function parse_tableau_external_and_materializable_asset_specs is now available to parse a list of Tableau asset specs into a list of external asset specs and materializable asset specs.
  • [dagster-looker] Looker assets now by default have owner and URL metadata.
  • [dagster-k8s] Added a per_step_k8s_config configuration option to the k8s_job_executor, allowing the k8s configuration of individual steps to be configured at run launch time (thanks @Kuhlwein!)
  • [dagster-fivetran] Introduced DagsterFivetranTranslator to customize assets loaded from Fivetran.
  • [dagster-snowflake] dagster_snowflake.fetch_last_updated_timestamps now supports ignoring tables not found in Snowflake instead of raising an error.

Bugfixes#

  • Fixed issue which would cause a default_automation_condition_sensor to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy auto_materialize: use_sensors configuration setting was set to False.
  • Fixed an issue where running dagster instance migrate on Dagster version 1.9.0 constructed a SQL query that exceeded the maximum allowed depth.
  • Fixed an issue where wiping a dynamically partitioned asset causes an error.
  • [dagster-polars] ImportErrors are no longer raised when bigquery libraries are not installed [#25708]

Documentation#

  • [dagster-dbt] A guide on how to use dbt defer with Dagster branch deployments has been added to the dbt reference.

1.6.14 (core) / 0.22.14 (libraries)#

Bugfixes#

  • [dagster-dbt] Fixed some issues with building column lineage metadata.

1.6.13 (core) / 0.22.13 (libraries)#

Bugfixes#

  • Fixed a bug where an asset with a dependency on a subset of the keys of a parent multi-asset could sometimes crash asset job construction.
  • Fixed a bug where a Definitions object containing assets having integrated asset checks and multiple partitions definitions could not be loaded.

1.6.12 (core) / 0.22.12 (libraries)#

New#

  • AssetCheckResult now has a text description property. Check evaluation descriptions are shown in the Checks tab on the asset details page.
  • Introduced TimestampMetadataValue. Timestamp metadata values are represented internally as seconds since the Unix epoch. They can be constructed using MetadataValue.timestamp. In the UI, they’re rendered in the local timezone, like other timestamps in the UI.
  • AssetSelection.checks can now accept AssetCheckKeys as well as AssetChecksDefinition.
  • [community-contribution] Metadata attached to an output at runtime (via either add_output_metadata or by passing to Output) is now available on HookContext under the op_output_metadata property. Thanks @JYoussouf!
  • [experimental] @asset, AssetSpec, and AssetOut now accept a tags property. Tags are key-value pairs meant to be used for organizing asset definitions. If "__dagster_no_value" is set as the value, only the key will be rendered in the UI. AssetSelection.tag allows selecting assets that have a particular tag.
  • [experimental] Asset tags can be used in asset CLI selections, e.g. dagster asset materialize --select tag:department=marketing
  • [experimental][dagster-dbt] Tags can now be configured on dbt assets, using DagsterDbtTranslator.get_tags. By default, we take the dbt tags configured on your dbt models, seeds, and snapshots.
  • [dagster-gcp] Added get_gcs_keys sensor helper function.

Bugfixes#

  • Fixed a bug that prevented external assets with dependencies from displaying properly in Dagster UI.
  • Fix a performance regression in loading code locations with large multi-assets.
  • [community-contribution][dagster-databricks] Fix a bug with the DatabricksJobRunner that led to an inability to use dagster-databricks with Databricks instance pools. Thanks @smats0n!
  • [community-contribution] Fixed a bug that caused a crash when external assets had hyphens in their AssetKey. Thanks @maxfirman!
  • [community-contribution] Fix a bug with load_assets_from_package_module that would cause a crash when any submodule had the same directory name as a dependency. Thanks @CSRessel!
  • [community-contribution] Fixed a mypy type error, thanks @parthshyara!
  • [community-contribution][dagster-embedded-elt] Fixed an issue where Sling assets would not properly read group and description metadata from replication config, thanks @jvyoralek!
  • [community-contribution] Ensured annotations from the helm chart properly propagate to k8s run pods, thanks @maxfirman!

Dagster Cloud#

  • Fixed an issue in Dagster Cloud Serverless runs where multiple runs simultaneously materializing the same asset would sometimes raise a “Key not found” exception.
  • Fixed an issue when using agent replicas where one replica would sporadically remove a code server created by another replica due to a race condition, leading to a “code server not found” or “Deployment not found” exception.
  • [experimental] The metadata key for specifying column schema that will be rendered prominently on the new Overview tab of the asset details page has been changed from "columns" to "dagster/column_schema". Materializations using the old metadata key will no longer result in the Columns section of the tab being filled out.
  • [ui] Fixed an Insights bug where loading a view filtered to a specific code location would not preserve that filter on pageload.

1.6.11 (core) / 0.22.11 (libraries)#

Bugfixes#

  • Fixed an issue where dagster dev or the Dagster UI would display an error when loading jobs created with op or asset selections.

1.6.10 (core) / 0.22.10 (libraries)#

New#

  • Latency improvements to the scheduler when running many simultaneous schedules.

Bugfixes#

  • The performance of loading the Definitions snapshot from a code server when large @multi_asset s are in use has been drastically improved.
  • The snowflake quickstart example project now renames the “by” column to avoid reserved snowflake names. Thanks @jcampbell!
  • The existing group name (if any) for an asset is now retained if the_asset.with_attributes is called without providing a group name. Previously, the existing group name was erroneously dropped. Thanks @ion-elgreco!
  • [dagster-dbt] Fixed an issue where Dagster events could not be streamed from dbt source freshness.
  • [dagster university] Removed redundant use of MetadataValue in Essentials course. Thanks @stianthaulow!
  • [ui] Increased the max number of plots on the asset plots page to 100.

Breaking Changes#

  • The tag_keys argument on DagsterInstance.get_run_tagsis no longer optional. This has been done to remove an easy way of accidentally executing an extremely expensive database operation.

Dagster Cloud#

  • The maximum number of concurrent runs across all branch deployments is now configurable. This setting can now be set via GraphQL or the CLI.
  • [ui] In Insights, fixed display of table rows with zero change in value from the previous time period.
  • [ui] Added deployment-level Insights.
  • [ui] Fixed an issue causing void invoices to show up as “overdue” on the billing page.
  • [experimental] Branch deployments can now indicate the new and modified assets in the branch deployment as compared to the main deployment. To enable this feature, turn on the “Enable experimental branch deployment asset graph diffing” user setting.

1.6.9 (core) / 0.22.9 (libraries)#

New#

  • [ui] When viewing logs for a run, the date for a single log row is now shown in the tooltip on the timestamp. This helps when viewing a run that takes place over more than one date.
  • Added suggestions to the error message when selecting asset keys that do not exist as an upstream asset or in an AssetSelection.
  • Improved error messages when trying to materialize a subset of a multi-asset which cannot be subset.
  • [dagster-snowflake] dagster-snowflake now requires snowflake-connector-python>=3.4.0
  • [embedded-elt] @sling_assets accepts an optional name parameter for the underlying op
  • [dagster-openai] dagster-openai library is now available.
  • [dagster-dbt] Added a new setting on DagsterDbtTranslatorSettings called enable_duplicate_source_asset_keys that allows users to set duplicate asset keys for their dbt sources. Thanks @hello-world-bfree!
  • Log messages in the Dagster daemon for unloadable sensors and schedules have been removed.
  • [ui] Search now uses a cache that persists across pageloads which should greatly improve search performance for very large orgs.
  • [ui] groups/code locations in the asset graph’s sidebar are now sorted alphabetically.

Bugfixes#

  • Fixed issue where the input/output schemas of configurable IOManagers could be ignored when providing explicit input / output run config.
  • Fixed an issue where enum values could not properly have a default value set in a ConfigurableResource.
  • Fixed an issue where graph-backed assets would sometimes lose user-provided descriptions due to a bug in internal copying.
  • [auto-materialize] Fixed an issue introduced in 1.6.7 where updates to ExternalAssets would be ignored when using AutoMaterializePolicies which depended on parent updates.
  • [asset checks] Fixed a bug with asset checks in step launchers.
  • [embedded-elt] Fix a bug when creating a SlingConnectionResource where a blank keyword argument would be emitted as an environment variable
  • [dagster-dbt] Fixed a bug where emitting events from dbt source freshness would cause an error.
  • [ui] Fixed a bug where using the “Terminate all runs” button with filters selected would not apply the filters to the action.
  • [ui] Fixed an issue where typing a search query into the search box before the search data was fetched would yield “No results” even after the data was fetched.

Community Contributions#

  • [docs] fixed typo in embedded-elt.mdx (thanks @cameronmartin)!
  • [dagster-databricks] log the url for the run of a databricks job (thanks @smats0n)!
  • Fix missing partition property (thanks christeefy)!
  • Add op_tags to @observable_source_asset decorator (thanks @maxfirman)!
  • [docs] typo in MultiPartitionMapping docs (thanks @dschafer)
  • Allow github actions to checkout branch from forked repo for docs changes (ci fix) (thanks hainenber)!

Experimental#

  • [asset checks] UI performance of asset checks related pages has been improved.
  • [dagster-dbt] The class DbtArtifacts has been added for managing the behavior of rebuilding the manifest during development but expecting a pre-built one in production.

Documentation#

  • Added example of writing compute logs to AWS S3 when customizing agent configuration.
  • "Hello, Dagster" is now "Dagster Quickstart" with the option to use a Github Codespace to explore Dagster.
  • Improved guides and reference to better running multiple isolated agents with separate queues on ECS.

Dagster Cloud#

  • Microsoft Teams is now supported for alerts. Documentation
  • A send sample alert button now exists on both the alert policies page and in the alert policies editor to make it easier to debug and configure alerts without having to wait for an event to kick them off.

1.6.8 (core) / 0.22.8 (libraries)#

Bugfixes#

  • [dagster-embedded-elt] Fixed a bug in the SlingConnectionResource that raised an error when connecting to a database.

Experimental#

  • [asset checks] graph_multi_assets with check_specs now support subsetting.

1.6.7 (core) / 0.22.7 (libraries)#

New#

  • Added a new run_retries.retry_on_op_or_asset_failures setting that can be set to false to make run retries only occur when there is an unexpected failure that crashes the run, allowing run-level retries to co-exist more naturally with op or asset retries. See the docs for more information.
  • dagster dev now sets the environment variable DAGSTER_IS_DEV_CLI allowing subprocesses to know that they were launched in a development context.
  • [ui] The Asset Checks page has been updated to show more information on the page itself rather than in a dialog.

Bugfixes#

  • [ui] Fixed an issue where the UI disallowed creating a dynamic partition if its name contained the “|” pipe character.
  • AssetSpec previously dropped the metadata and code_version fields, resulting in them not being attached to the corresponding asset. This has been fixed.

Experimental#

  • The new @multi_observable_source_asset decorator enables defining a set of assets that can be observed together with the same function.
  • [dagster-embedded-elt] New Asset Decorator @sling_assets and Resource SlingConnectionResource have been added for the [dagster-embedded-elt.sling](http://dagster-embedded-elt.sling) package. Deprecated build_sling_asset, SlingSourceConnection and SlingTargetConnection.
  • Added support for op-concurrency aware run dequeuing for the QueuedRunCoordinator.

Documentation#

  • Fixed reference documentation for isolated agents in ECS.
  • Corrected an example in the Airbyte Cloud documentation.
  • Added API links to OSS Helm deployment guide.
  • Fixed in-line pragmas showing up in the documentation.

Dagster Cloud#

  • Alerts now support Microsoft Teams.
  • [ECS] Fixed an issue where code locations could be left undeleted.
  • [ECS] ECS agents now support setting multiple replicas per code server.
  • [Insights] You can now toggle the visibility of a row in the chart by clicking on the dot for the row in the table.
  • [Users] Added a new column “Licensed role” that shows the user's most permissive role.

1.6.6 (core) / 0.22.6 (libraries)#

New#

  • Dagster officially supports Python 3.12.
  • dagster-polars has been added as an integration. Thanks @danielgafni!
  • [dagster-dbt] @dbt_assets now supports loading projects with semantic models.
  • [dagster-dbt] @dbt_assets now supports loading projects with model versions.
  • [dagster-dbt] get_asset_key_for_model now supports retrieving asset keys for seeds and snapshots. Thanks @aksestok!
  • [dagster-duckdb] The Dagster DuckDB integration supports DuckDB version 0.10.0.
  • [UPath I/O manager] If a non-partitioned asset is updated to have partitions, the file containing the non-partitioned asset data will be deleted when the partitioned asset is materialized, rather than raising an error.

Bugfixes#

  • Fixed an issue where creating a backfill of assets with dynamic partitions and a backfill policy would sometimes fail with an exception.
  • Fixed an issue with the type annotations on the @asset decorator causing a false positive in Pyright strict mode. Thanks @tylershunt!
  • [ui] On the asset graph, nodes are slightly wider allowing more text to be displayed, and group names are no longer truncated.
  • [ui] Fixed an issue where the groups in the asset graph would not update after an asset was switched between groups.
  • [dagster-k8s] Fixed an issue where setting the security_context field on the k8s_job_executor didn't correctly set the security context on the launched step pods. Thanks @krgn!

Experimental#

  • Observable source assets can now yield ObserveResults with no data_version.
  • You can now include FreshnessPolicys on observable source assets. These assets will be considered “Overdue” when the latest value for the “dagster/data_time” metadata value is older than what’s allowed by the freshness policy.
  • [ui] In Dagster Cloud, a new feature flag allows you to enable an overhauled asset overview page with a high-level stakeholder view of the asset’s health, properties, and column schema.

Documentation#

  • Updated docs to reflect newly-added support for Python 3.12.

Dagster Cloud#

  • [kubernetes] Fixed an issue where the Kubernetes agent would sometimes leave dangling kubernetes services if the agent was interrupted during the middle of being terminated.

1.6.5 (core) / 0.22.5 (libraries)#

New#

  • Within a backfill or within auto-materialize, when submitting runs for partitions of the same assets, runs are now submitted in lexicographical order of partition key, instead of in an unpredictable order.
  • [dagster-k8s] Include k8s pod debug info in run worker failure messages.
  • [dagster-dbt] Events emitted by DbtCliResource now include metadata from the dbt adapter response. This includes fields like rows_affected, query_id from the Snowflake adapter, or bytes_processed from the BigQuery adapter.

Bugfixes#

  • A previous change prevented asset backfills from grouping multiple assets into the same run when using BackfillPolicies under certain conditions. While the backfills would still execute in the proper order, this could lead to more individual runs than necessary. This has been fixed.
  • [dagster-k8s] Fixed an issue introduced in the 1.6.4 release where upgrading the Helm chart without upgrading the Dagster version used by user code caused failures in jobs using the k8s_job_executor.
  • [instigator-tick-logs] Fixed an issue where invoking context.log.exception in a sensor or schedule did not properly capture exception information.
  • [asset-checks] Fixed an issue where additional dependencies for dbt tests modeled as Dagster asset checks were not properly being deduplicated.
  • [dagster-dbt] Fixed an issue where dbt model, seed, or snapshot names with periods were not supported.

Experimental#

  • @observable_source_asset-decorated functions can now return an ObserveResult. This allows including metadata on the observation, in addition to a data version. This is currently only supported for non-partitioned assets.
  • [auto-materialize] A new AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron class allows you to construct AutoMaterializePolicys which wait for all parents to be updated after the latest tick of a given cron schedule.
  • [Global op/asset concurrency] Ops and assets now take run priority into account when claiming global op/asset concurrency slots.

Documentation#

  • Fixed an error in our asset checks docs. Thanks @vaharoni!
  • Fixed an error in our Dagster Pipes Kubernetes docs. Thanks @cameronmartin!
  • Fixed an issue on the Hello Dagster! guide that prevented it from loading.
  • Add specific capabilities of the Airflow integration to the Airflow integration page.
  • Re-arranged sections in the I/O manager concept page to make info about using I/O versus resources more prominent.

1.6.4 (core) / 0.22.4 (libraries)#

New#

  • build_schedule_from_partitioned_job now supports creating a schedule from a static-partitioned job (Thanks @craustin!)
  • [dagster-pipes] PipesK8sClient will now autodetect the namespace when using in-cluster config. (Thanks @aignas!)
  • [dagster-pipes] PipesK8sClient can now inject the context in to multiple containers. (Thanks @aignas!)
  • [dagster-snowflake] The Snowflake Pandas I/O manager now uses the write_pandas method to load Pandas DataFrames in Snowflake. To support this change, the database connector was switched from SqlDbConnection to SnowflakeConnection .
  • [ui] On the overview sensors page you can now filter sensors by type.
  • [dagster-deltalake-polars] Added LazyFrame support (Thanks @ion-elgreco!)
  • [dagster-dbt] When using @dbt_assets and multiple dbt resources produce the same AssetKey, we now display an exception message that highlights the file paths of the misconfigured dbt resources in your dbt project.
  • [dagster-k8s] The debug info reported upon failure has been improved to include additional information from the Job. (Thanks @jblawatt!)
  • [dagster-k8s] Changed the Dagster Helm chart to apply automountServiceAccountToken: false to the default service account used by the Helm chart, in order to better comply with security policies. (Thanks @MattyKuzyk!)

Bugfixes#

  • A unnecessary thread lock has been removed from the sensor daemon. This should improve sensor throughput for users with many sensors who have enabled threading.
  • Retry from failure behavior has been improved for cases where dynamic steps were interrupted.
  • Previously, when backfilling a set of assets which shared a BackfillPolicy and PartitionsDefinition, but had a non-default partition mapping between them, a run for the downstream asset could be launched at the same time as a separate run for the upstream asset, resulting in inconsistent partition ordering. Now, the downstream asset will only execute after the parents complete. (Thanks @ruizh22!)
  • Previously, asset backfills would raise an exception if the code server became unreachable mid-iteration. Now, the backfill will pause until the next evaluation.
  • Fixed a bug that was causing ranged backfills over dynamically partitioned assets to fail.
  • [dagster-pipes] PipesK8sClient has improved handling for init containers and additional containers. (Thanks @aignas!)
  • Fixed the last_sensor_start_time property of the SensorEvaluationContext, which would get cleared on ticks after the first tick after the sensor starts.
  • [dagster-mysql] Fixed the optional dagster instance migrate --bigint-migration, which caused some operational errors on mysql storages.
  • [dagster-dbt] Fixed a bug introduced in 1.6.3 that caused errors when ingesting asset checks with multiple dependencies.

Deprecations#

  • The following methods on AssetExecutionContext have been marked deprecated, with their suggested replacements in parenthesis:
    • context.op_config (context.op_execution_context.op_config)
    • context.node_handle (context.op_execution_context.node_handle)
    • context.op_handle (context.op_execution_context.op_handle)
    • context.op (context.op_execution_context.op)
    • context.get_mapping_key (context.op_execution_context.get_mapping_key)
    • context.selected_output_names (context.op_execution_context.selected_output_names)
    • context.dagster_run (context.run)
    • context.run_id (context.run.run_id)
    • context.run_config (context.run.run_config)
    • context.run_tags (context.run.tags)
    • context.has_tag (key in context.run.tags)
    • context.get_tag (context.run.tags.get(key))
    • context.get_op_execution_context (context.op_execution_context)
    • context.asset_partition_key_for_output (context.partition_key)
    • context.asset_partition_keys_for_output (context.partition_keys)
    • context.asset_partitions_time_window_for_output (context.partition_time_window)
    • context.asset_partition_key_range_for_output (context.partition_key_range)

Experimental#

  • [asset checks] @asset_check now has a blocking parameter. When this is enabled, if the check fails with severity ERROR then any downstream assets in the same run won’t execute.

Documentation#

  • The Branch Deployment docs have been updated to reflect support for backfills
  • Added Dagster’s maximum supported Python version (3.11) to Dagster University and relevant docs
  • Added documentation for recommended partition limits (a maximum of 25K per asset).
  • References to the Enterprise plan have been renamed to Pro, to reflect recent plan name changes
  • Added syntax example for setting environment variables in PowerShell to our dbt with Dagster tutorial
  • [Dagster University] Dagster Essentials to Dagster v1.6, and introduced the usage of MaterializeResult
  • [Dagster University] Fixed a typo in the Dagster University section on adding partitions to an asset (Thanks Brandon Peebles!)
  • [Dagster University] Corrected lesson where sensors are covered (Thanks onefloid!)

Dagster Cloud#

  • Agent tokens can now be locked down to particular deployments. Agents will not be able to run any jobs scheduled for deployments that they are not permitted to access. By default, agent tokens have access to all deployments in an organization. Use the Edit button next to an agent token on the Tokens tab in Org Settings to configure permissions for a particular token. You must be an Organization Admin to edit agent token permissions.

1.6.3 (core) / 0.22.3 (libraries)#

New#

  • Added support for the 3.0 release of the pendulum library, for Python versions 3.9 and higher.
  • Performance improvements when starting run worker processes or step worker processes for runs in code locations with a large number of jobs.
  • AllPartitionMapping now supports mapping to downstream partitions, enabling asset backfills with these dependencies. Thanks @craustin!
  • [asset checks][experimental] @asset_check has new fields additional_deps and additional_ins to allow dependencies on assets other than the asset being checked.
  • [ui] Asset graph group nodes now show status counts.
    • [dagster-snowflake] The Snowflake I/O Manager now has more specific error handling when a table doesn’t exist.
  • [ui][experimental] A new experimental UI for the auto-materialize history of a specific asset has been added. This view can be enabled under your user settings by setting “Use new asset auto-materialize history page”.
  • [ui] Command clicking on an asset group will now select or deselect all assets in that group.
  • [dagster-k8s] Added the ability to customize resource limits for initContainers used by Dagster system components in the Dagster Helm chart. Thanks @MattyKuzyk!
  • [dagster-k8s] Added the ability to specify additional containers and initContainers in code locations in the Helm chart. Thanks @craustin!
  • [dagster-k8s] Explicitly listed the set of RBAC permissions used by the agent Helm chart role instead of using a wildcard. Thanks @easontm!
  • [dagster-dbt] Support for dbt-core==1.4.* is now removed because the version has reached end-of-life.

Bugfixes#

  • Previously, calling get_partition_keys_not_in_subset on a BaseTimeWindowPartitionsSubset that targeted a partitions definition with no partitions (e.g. a future start date) would raise an error. Now, it returns an empty list.
  • Fixed issue which could cause invalid runs to be launched if a code location was updated during the course of an AMP evaluation.
  • Previously, some asset backfills raised an error when targeting multi-assets with internal asset dependencies. This has been fixed.
  • Previously, using the LocalComputeLogManager on Windows could result in errors relating to invalid paths. This has been resolved. Thanks @hainenber!
  • An outdated path in the contribution guide has been updated. Thanks @hainenber!
  • [ui] Previously an error was sometimes raised when attempting to create a dynamic partition within a multi-partitioned asset via the UI. This has been fixed.
  • [ui] The “Upstream materializations are missing” warning when launching a run has been expanded to warn about failed upstream materializations as well.
  • [ui] The community welcome modal now renders properly in dark mode and some elements of Asset and Op graphs have higher contrast in both themes.
  • [ui] Fixed dark mode colors for datepicker, error message, and op definition elements.
  • [ui] Pressing the arrow keys to navigate op/asset graphs while the layout is loading no longer causes errors.
  • [ui] Exporting asset and op graphs to SVG no longer fails when chrome extensions inject additional stylesheets into Dagster’s UI.
  • [ui] Dagster now defaults to UTC when the user’s default timezone cannot be identified, rather than crashing with a date formatting error.
  • [ui] Fixed an issue in the asset graph sidebar that caused groups to only list their first asset.
  • [ui] Fixed an issue where sensors runs would undercount the number of dynamic partition requests added or deleted if there were multiple requests for additions/deletions.
  • [docs] Fixed a typo in the “Using Dagster with Delta Lake” guide. Thanks @avriiil!
  • [asset checks] Fixed an issue which could cause errors when using asset checks with step launchers.
  • [dagster-webserver] A bug preventing WebSocket connections from establishing on python 3.11+ has been fixed.
  • [dagster-databricks] DatabricksJobRunner now ensures the correctdatabricks-sdk is installed. Thanks @zyd14!
  • [dagster-dbt] On run termination, an interrupt signal is now correctly forwarded to any in-progress dbt subprocesses.
  • [dagster-dbt] Descriptions for dbt tests ingested as asset checks can now be populated using the config.meta.description. Thanks @CapitanHeMo!
  • [dagster-dbt] Previously, the error message displayed when no dbt profiles information was found would display an incorrect path. This has been fixed. Thanks @zoltanctoth!
  • [dagster-k8s] PipesK8sClient can now correctly handle load_incluster_config . Thanks @aignas!

Documentation#

  • Added a new category to Concepts: Automation. This page provides a high-level overview of the various ways Dagster allows you run data pipelines without manual intervention.
  • Moved several concept pages under Concepts > Automation: Schedules, Sensors, Asset Sensors, and Auto-materialize Policies.

Dagster Cloud#

  • Fixed an issue where configuring the agent_queue key in a dagster_cloud.yaml file incorrectly failed to validate when using the dagster-cloud ci init or dagster-cloud ci check commands during CI/CD.