Ask AI

Changelog#

1.9.4 (core) / 0.25.4 (libraries)#

New#

  • Global op concurrency is now enabled on the default SQLite storage. Deployments that have not been migrated since 1.6.0 may need to run dagster instance migrate to enable.
  • Introduced map_asset_specs to enable modifying AssetSpecs and AssetsDefinitions in bulk.
  • Introduced AssetSpec.replace_attributes and AssetSpec.merge_attributes to easily alter properties of an asset spec.
  • [ui] Add a "View logs" button to open tick logs in the sensor tick history table.
  • [ui] Add Spanner kind icon.
  • [ui] The asset catalog now supports filtering using the asset selection syntax.
  • [dagster-pipes, dagster-aws] PipesS3MessageReader now has a new parameter include_stdio_in_messages which enables log forwarding to Dagster via Pipes messages.
  • [dagster-pipes] Experimental: A new Dagster Pipes message type log_external_stream has been added. It can be used to forward external logs to Dagster via Pipes messages.
  • [dagster-powerbi] Opts in to using admin scan APIs to pull data from a Power BI instance. This can be disabled by passing load_powerbi_asset_specs(..., use_workspace_scan=False).
  • [dagster-sigma] Introduced an experimental dagster-sigma snapshot command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.

Bugfixes#

  • Fixed a bug that caused DagsterExecutionStepNotFoundError errors when trying to execute an asset check step of a run launched by a backfill.
  • Fixed an issue where invalid cron strings like "0 0 30 2 *" that represented invalid dates in February were still allowed as Dagster cron strings, but then failed during schedule execution. Now, these invalid cronstrings will raise an exception when they are first loaded.
  • Fixed a bug where owners added to AssetOuts when defining a @graph_multi_asset were not added to the underlying AssetsDefinition.
  • Fixed a bug where using the & or | operators on AutomationConditions with labels would cause that label to be erased.
  • [ui] Launching partitioned asset jobs from the launchpad now warns if no partition is selected.
  • [ui] Fixed unnecessary middle truncation occurring in dialogs.
  • [ui] Fixed timestamp labels and "Now" line rendering bugs on the sensor tick timeline.
  • [ui] Opening Dagster's UI with a single job defined takes you to the Overview page rather than the Job page.
  • [ui] Fix stretched tags in backfill table view for non-partitioned assets.
  • [ui] Open automation sensor evaluation details in a dialog instead of navigating away.
  • [ui] Fix scrollbars in dark mode.
  • [dagster-sigma] Workbooks filtered using a SigmaFilter no longer fetch lineage information.
  • [dagster-powerbi] Fixed an issue where reports without an upstream dataset dependency would fail to translate to an asset spec.

Deprecations#

  • [dagster-powerbi] DagsterPowerBITranslator.get_asset_key is deprecated in favor of DagsterPowerBITranslator.get_asset_spec().key
  • [dagster-looker] DagsterLookerApiTranslator.get_asset_key is deprecated in favor of DagsterLookerApiTranslator.get_asset_spec().key
  • [dagster-sigma] DagsterSigmaTranslator.get_asset_key is deprecated in favor of DagsterSigmaTranslator.get_asset_spec().key
  • [dagster-tableau] DagsterTableauTranslator.get_asset_key is deprecated in favor of DagsterTableauTranslator.get_asset_spec().key

1.9.3 (core) / 0.25.3 (libraries)#

New#

  • Added run_id to the run_tags index to improve database performance. Run dagster instance migrate to update the index. (Thanks, @HynekBlaha!)

  • Added icons for kind tags: Cassandra, ClickHouse, CockroachDB, Doris, Druid, Elasticsearch, Flink, Hadoop, Impala, Kafka, MariaDB, MinIO, Pinot, Presto, Pulsar, RabbitMQ, Redis, Redpanda, ScyllaDB, Starrocks, and Superset. (Thanks, @swrookie!)

  • Added a new icon for the Denodo kind tag. (Thanks, @tintamarre!)

  • Errors raised from defining more than one Definitions object at module scope now include the object names so that the source of the error is easier to determine.

  • [ui] Asset metadata entries like dagster/row_count now appear on the events page and are properly hidden on the overview page when they appear in the sidebar.

  • [dagster-aws] PipesGlueClient now attaches AWS Glue metadata to Dagster results produced during Pipes invocation.

  • [dagster-aws] PipesEMRServerlessClient now attaches AWS EMR Serverless metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the job run.

  • [dagster-aws] PipesECSClient now attaches AWS ECS metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the ECS task.

  • [dagster-aws] PipesEMRClient now attaches AWS EMR metadata to Dagster results produced during Pipes invocation.

  • [dagster-databricks] PipesDatabricksClient now attaches Databricks metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the Databricks job.

  • [dagster-fivetran] Added load_fivetran_asset_specs function. It can be used with the FivetranWorkspace resource and DagsterFivetranTranslator translator to load your Fivetran connector tables as external assets in Dagster.

  • [dagster-looker] Errors are now handled more gracefully when parsing derived tables.

  • [dagster-sigma] Sigma assets now contain extra metadata and kind tags.

  • [dagster-sigma] Added support for direct workbook to warehouse table dependencies.

  • [dagster-sigma] Added include_unused_datasets field to SigmaFilter to disable pulling datasets that aren't used by a downstream workbook.

  • [dagster-sigma] Added skip_fetch_column_data option to skip loading Sigma column lineage. This can speed up loading large instances.

  • [dagster-sigma] Introduced an experimental dagster-sigma snapshot command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.

    Introducing: dagster-airlift (experimental)#

    dagster-airlift is coming out of stealth. See the initial Airlift RFC here, and the following documentation to learn more:

    More Airflow-related content is coming soon! We'd love for you to check it out, and post any comments / questions in the #airflow-migration channel in the Dagster slack.

Bugfixes#

  • Fixed a bug in run status sensors where setting incompatible arguments monitor_all_code_locations and monitored_jobs did not raise the expected error. (Thanks, @apetryla!)
  • Fixed an issue that would cause the label for AutomationCondition.any_deps_match() and AutomationCondition.all_deps_match() to render incorrectly when allow_selection or ignore_selection were set.
  • Fixed a bug which could cause code location load errors when using CacheableAssetsDefinitions in code locations that contained AutomationConditions
  • Fixed an issue where the default multiprocess executor kept holding onto subprocesses after their step completed, potentially causing Too many open files errors for jobs with many steps.
  • [ui] Fixed an issue introduced in 1.9.2 where the backfill overview page would sometimes display extra assets that were targeted by the backfill.
  • [ui] Fixed "Open in Launchpad" button when testing a schedule or sensor by ensuring that it opens to the correct deployment.
  • [ui] Fixed an issue where switching a user setting was immediately saved, rather than waiting for the change to be confirmed.
  • [dagster-looker] Unions without unique/distinct criteria are now properly handled.
  • [dagster-powerbi] Fixed an issue where reports without an upstream dataset dependency would fail to translate to an asset spec.
  • [dagster-sigma] Fixed an issue where API fetches did not paginate properly.

Documentation#

Dagster Plus#

  • [ui] Fixed an issue with filtering and catalog search in branch deployments.
  • [ui] Fixed an issue where the asset graph would reload unexpectedly.

1.9.2 (core) / 0.25.2 (libraries)#

New#

  • Introduced a new constructor, AssetOut.from_spec, that will construct an AssetOut from an AssetSpec.
  • [ui] Column tags are now displayed in the Column name section of the asset overview page.
  • [ui] Introduced an icon for the gcs (Google Cloud Storage) kind tag.
  • [ui] Introduced icons for report and semanticmodel kind tags.
  • [ui] The tooltip for a tag containing a cron expression now shows a human-readable, timezone-aware cron string.
  • [ui] Asset check descriptions are now sourced from docstrings and rendered in the UI. (Thanks, @marijncv!)
  • [dagster-aws] Added option to propagate tags to ECS tasks when using the EcsRunLauncher. (Thanks, @zyd14!)
  • [dagster-dbt] You can now implement DagsterDbtTranslator.get_code_version to customize the code version for your dbt assets. (Thanks, @Grzyblon!)
  • [dagster-pipes] Added the ability to pass arbitrary metadata to PipesClientCompletedInvocation. This metadata will be attached to all materializations and asset checks stored during the pipes invocation.
  • [dagster-powerbi] During a full workspace scan, owner and column metadata is now automatically attached to assets.

Bugfixes#

  • Fixed an issue with AutomationCondition.execution_in_progress which would cause it to evaluate to True for unpartitioned assets that were part of a run that was in progress, even if the asset itself had already been materialized.
  • Fixed an issue with AutomationCondition.run_in_progress that would cause it to ignore queued runs.
  • Fixed an issue that would cause a default_automation_condition_sensor to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy auto_materialize: use_sensors configuration setting was set to False.
  • [ui] Fixed an issue when executing asset checks where the wrong job name was used in some situations. The correct job name is now used.
  • [ui] Selecting assets with 100k+ partitions no longer causes the asset graph to temporarily freeze.
  • [ui] Fixed an issue that could cause a GraphQL error on certain pages after removing an asset.
  • [ui] The asset events page no longer truncates event history in cases where both materialization and observation events are present.
  • [ui] The backfill coordinator logs tab no longer sits in a loading state when no logs are available to display.
  • [ui] Fixed issue which would cause the "Partitions evaluated" label on an asset's automation history page to incorrectly display 0 in cases where all partitions were evaluated.
  • [ui] Fix "Open in Playground" link when testing a schedule or sensor by ensuring that it opens to the correct deployment.
  • [ui] Fixed an issue where the asset graph would reload unexpectedly.
  • [dagster-dbt] Fixed an issue where the SQL filepath for a dbt model was incorrectly resolved when the dbt manifest file was built on a Windows machine, but executed on a Unix machine.
  • [dagster-pipes] Asset keys containing embedded / characters now work correctly with Dagster Pipes.

Documentation#

Deprecations#

  • The types-sqlalchemy package is no longer included in the dagster[pyright] extra package.

Dagster Plus#

  • [ui] The Environment Variables table can now be sorted by name and update time.
  • [ui] The code location configuration dialog now contains more metadata about the code location.
  • [ui] Fixed an issue where the incorrect user icons were shown in the Users table when a search filter had been applied.

1.9.1 (core) / 0.25.1 (libraries)#

New#

  • dagster project scaffold now has an option to create dagster projects from templates with excluded files/filepaths.
  • [ui] Filters in the asset catalog now persist when navigating subdirectories.
  • [ui] The Run page now displays the partition(s) a run was for.
  • [ui] Filtering on owners/groups/tags is now case-insensitive.
  • [dagster-tableau] the helper function parse_tableau_external_and_materializable_asset_specs is now available to parse a list of Tableau asset specs into a list of external asset specs and materializable asset specs.
  • [dagster-looker] Looker assets now by default have owner and URL metadata.
  • [dagster-k8s] Added a per_step_k8s_config configuration option to the k8s_job_executor, allowing the k8s configuration of individual steps to be configured at run launch time (thanks @Kuhlwein!)
  • [dagster-fivetran] Introduced DagsterFivetranTranslator to customize assets loaded from Fivetran.
  • [dagster-snowflake] dagster_snowflake.fetch_last_updated_timestamps now supports ignoring tables not found in Snowflake instead of raising an error.

Bugfixes#

  • Fixed issue which would cause a default_automation_condition_sensor to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy auto_materialize: use_sensors configuration setting was set to False.
  • Fixed an issue where running dagster instance migrate on Dagster version 1.9.0 constructed a SQL query that exceeded the maximum allowed depth.
  • Fixed an issue where wiping a dynamically partitioned asset causes an error.
  • [dagster-polars] ImportErrors are no longer raised when bigquery libraries are not installed [#25708]

Documentation#

  • [dagster-dbt] A guide on how to use dbt defer with Dagster branch deployments has been added to the dbt reference.

1.7.9 (core) / 0.23.9 (libraries)#

New#

  • Dagster will now display a “storage kind” tag on assets in the UI, similar to the existing compute kind. To set storage kind for an asset, set its dagster/storage_kind tag.
  • You can now set retry policy on dbt assets, to enable coarse-grained retries with delay and jitter. For fine-grained partial retries, we still recommend invoking dbt retry within a try/except block to avoid unnecessary, duplicate work.
  • AssetExecutionContext now exposes a has_partition_key_range property.
  • The owners, metadata, tags, and deps properties on AssetSpec are no longer Optional. The AssetSpec constructor still accepts None values, which are coerced to empty collections of the relevant type.
  • The docker_executor and k8s_job_executor now consider at most 1000 events at a time when loading events from the current run to determine which steps should be launched. This value can be tuned by setting the DAGSTER_EXECUTOR_POP_EVENTS_LIMIT environment variable in the run process.
  • Added a dagster/retry_on_asset_or_op_failure tag that can be added to jobs to override run retry behavior for runs of specific jobs. See the docs for more information.
  • Improved the sensor produced by build_sensor_for_freshness_checks to describe when/why it skips evaluating freshness checks.
  • A new “Runs” tab on the backfill details page allows you to see list and timeline views of the runs launched by the backfill.
  • [dagster-dbt] dbt will now attach relation identifier metadata to asset materializations to indicate where the built model is materialized to.
  • [dagster-graphql] The GraphQL Python client will now include the HTTP error code in the exception when a query fails. Thanks @yuvalgimmunai!

Bugfixes#

  • Fixed sensor logging behavior with the @multi_asset_sensor.
  • ScheduleDefinition now properly supports being passed a RunConfig object.
  • When an asset function returns a MaterializeResult, but the function has no type annotation, previously, the IO manager would still be invoked with a None value. Now, the IO manager is not invoked.
  • The AssetSpec constructor now raises an error if an invalid owner string is passed to it.
  • When using the graph_multi_asset decorator, the code_version property on AssetOuts passed in used to be ignored. Now, they no longer are.
  • [dagster-deltalake] Fixed GcsConfig import error and type error for partitioned assets (Thanks @thmswt)
  • The asset graph and asset catalog now show the materialization status of External assets (when manually reported) rather than showing “Never observed”

Documentation#

  • The External Assets REST APIs now have their own reference page
  • Added details, updated copy, and improved formatting to External Assets REST APIs

Dagster Plus#

  • The ability to set a custom base deployment when creating a branch deployment has been enabled for all organizations.
  • When a code location fails to deploy, the Kubernetes agent now includes additional any warning messages from the underlying replicaset in the failure message to aid with troubleshooting.
  • Serverless deployments now support using a requirements.txt with hashes.
  • Fixed an issue where the dagster-cloud job launch command did not support specifying asset keys with prefixes in the --asset-key argument.
  • [catalog UI] Catalog search now allows filtering by type, i.e. group:, code location:, tag:, owner:.
  • New dagster+ accounts will now start with two default alert policies; one to alert if the default free credit budget for your plan is exceeded, and one to alert if a single run goes over 24 hours. These alerts will be sent as emails to the email with which the account was initially created.

1.7.8 (core) / 0.23.8 (libraries)#

New#

  • Backfills created via GQL can have a custom title and description.
  • Definitions now has a get_all_asset_specs method, which allows iterating over properties of the defined assets
  • [ui] In filter dropdowns, it’s now possible to submit before all the suggestions have been loaded (thanks @bmalehorn!)
  • [ui] Performance improvements when loading the Dagster UI for asset graphs with thousands of partition keys.
  • [dagster-dbt] Dbt asset checks now emit execution duration and the number of failing rows as metadata
  • [dagster-embedded-elt] Added support for partitioning in dlt assets (thanks @edsoncezar16!)
  • [dagster-embedded-elt] Added ability to set custom metadata on dlt assets (thanks @edsoncezar16!)
  • [dagster-graphql] Added a terminate_runs method to the Python GraphQL Client. (thanks @baumann-t!)
  • [dagster-polars] dagster-polars IO managers now emit dagster/row_count metadata (thanks @danielgafni!)
  • [dagster-dbt] DbtCliInvocation now has a .get_error() method that can be useful when using dbt.cli(..., raise_on_error=False).

Bugfixes#

  • Fix a bug with legacy DynamicPartitionsDefinition (using partitions_fn) that caused a crash during job backfills.
  • [ui] On the asset graph, filtering to one or more code locations via the Filter dropdown now works as expected.
  • [ui] On the asset overview page, viewing an asset with no definition in a loaded code location no longer renders a clipped empty state.

Experimental#

  • The new build_metadata_bounds_checks API creates asset checks which verify that numeric metadata values on asset materializations fall within min or max values. See the documentation for more information.

Documentation#

Dagster Plus#

  • Fixed an incompatibility between build_sensor_for_freshness_checks and Dagster Plus. This API should now work when used with Dagster Plus.
  • [ui] Billing / usage charts no longer appear black-on-black in Dagster’s dark mode.
  • [ui] The asset catalog is now available for teams plans.
  • [ui] Fixed a bug where the alert policy editor would misinterpret the threshold on a long-running job alert.
  • [kubernetes] Added a dagsterCloudAgent.additionalPodSpecConfig to the Kubernetes agent Helm chart allowing arbitrary pod configuration to be applied to the agent pod.
  • [ECS] Fixed an issue where the ECS agent would sometimes raise a “Too many concurrent attempts to create a new revision of the specified family” exception when using agent replicas.

1.7.7 (core) / 0.23.7 (libraries)#

New#

  • [ui] Command clicking on nodes in the asset lineage tab will now open them in a separate tab. Same with external asset links in the asset graph.
  • Added support for setting a custom job namespace in user code deployments. (thanks @tmatthews0020!)
  • Removed warnings due to use of datetime.utcfromtimestamp (thanks @dbrtly!)
  • Custom smtp user can now be used for e-mail alerts (thanks @edsoncezar16!)
  • [dagster-dbt] Added support for dbt-core==1.8.*.
  • [dagster-embedded-elt] Failed dlt pipelines are now accurately reflected on the asset materialization (thanks @edsoncezar16!)

Bugfixes#

  • Fixed spurious errors in logs due to module shadowing.
  • Fixed an issue in the Backfill Daemon where if the assets to be materialized had different BackfillPolicys, each asset would get materialized in its own run, rather than grouping assets together into single run.
  • Fixed an issue that could cause the Asset Daemon to lose information in its cursor about an asset if that asset’s code location was temporarily unavailable.
  • [dagster-dbt] Mitigated issues with cli length limits by only listing specific dbt tests as needed when the tests aren’t included via indirect selection, rather than listing all tests.

Documentation#

  • Markdoc tags can now be used in place of MDX components (thanks @nikomancy)

1.7.6 (core) / 0.23.6 (libraries)#

New#

  • The backfill daemon now has additional logging to document the progression through each tick and why assets are and are not materialized during each evaluation of a backfill.
  • Made performance improvements in both calculating and storing data version for assets, especially for assets with a large fan-in.
  • Standardized table row count metadata output by various integrations to dagster/row_count .
  • [dagster-aws][community-contribution] Additional parameters can now be passed to the following resources: CloudwatchLogsHandler, ECRPublicClient, SecretsManagerResource, SSMResource thanks @jacob-white-simplisafe !
  • Added additional frontend telemetry. See https://docs.dagster.io/about/telemetry for more information.

Bugfixes#

  • Fixed issue that could cause runs to fail if they targeted any assets which had a metadata value of type TableMetadataValue, TableSchemaMetadataValue, or TableColumnLineageMetadataValue defined.
  • Fixed an issue which could cause evaluations produced via the Auto-materialize system to not render the “skip”-type rules.
  • Backfills of asset jobs now correctly use the BackfillPolicy of the underlying assets in the job.
  • [dagster-databricks][community-contribution] databricks-sdk version bumped to 0.17.0, thanks @lamalex !
  • [helm][community-contribution] resolved incorrect comments about dagster code-server start , thanks @SanjaySiddharth !

Documentation#

  • Added section headings to Pipes API references, along with explanatory copy and links to relevant pages
  • Added a guide for subletting asset checks
  • Add more detailed steps to transition from serverless to hybrid
  • [community-contribution] asset selection syntax corrected, thanks @JonathanLai2004!

Dagster Plus#

  • Fixed an issue where Dagster Cloud agents would wait longer than necessary when multiple code locations were timing out during a deployment.

1.7.5 (core) / 0.23.5 (libraries)#

New#

  • The Asset > Checks tab now allows you to view plots of numeric metadata emitted by your checks.
  • The Asset > Events tab now supports infinite-scrolling, making it possible to view all historical materialization and observation events.
  • When constructing a MaterializeResult, ObserveResult, or Output, you can now include tags that will be attached to the corresponding AssetMaterialization or AssetObservation event. These tags will be rendered on these events in the UI.

Bugfixes#

  • Fixed an issue where backfills would sometimes fail if a partition definition was changed in the middle of the backfill.
  • Fixed an issue where if the code server became unavailable during the first tick of a backfill, the backfill would stall and be unable to submit runs once the code server became available.
  • Fixed an issue where the status of an external asset would not get updated correctly.
  • Fixed an issue where run status sensors would sometimes fall behind in deployments with large numbers of runs.
  • The descriptions and metadata on the experimental build_last_update_freshness_checks and build_time_partition_freshness_checks APIs have been updated to be clearer.
  • The headers of tables no longer become misaligned when a scrollbar is present in some scenarios.
  • The sensor type, instigation type, and backfill status filters on their respective pages are now saved to the URL, so sharing the view or reloading the page preserve your filters.
  • Typing a % into the asset graph’s query selector no longer crashes the UI.
  • “Materializing” states on the asset graph animate properly in both light and dark themes.
  • Thanks to @lautaro79 for fixing a helm chart issue.

Breaking Changes#

  • Subclasses of MetadataValue have been changed from NamedTuples to Pydantic models. NamedTuple functionality on these classes was not part of Dagster’s stable public API, but usages relying on their tuple-ness may break. For example: calling json.dumps on collections that include them.

Deprecations#

Dagster Plus#

  • Fixed an issue in the dagster-cloud CLI where the --deployment argument was ignored when the DAGSTER_CLOUD_URL environment variable was set.
  • Fixed an issue where dagster-cloud-cli package wouldn’t work unless the dagster-cloud package was installed as well.
  • A new “budget alerts” feature has launched for users on self-serve plans. This feature will alert you when you hit your credit limit.
  • The experimental asset health overview now allows you to group assets by compute kind, tag, and tag value.
  • The concurrency and locations pages in settings correctly show Dagster Plus-specific options when experimental navigation is enabled.