Ask AI

Changelog#

0.9.13#

Bugfixes

  • Fixes an issue using build_reconstructable_pipeline.
  • Improved loading times for the asset catalog in Dagit.

Documentations

  • Improved error messages when invoking dagit from the CLI with bad arguments.

0.9.12#

Breaking Changes

  • Dagster now warns when a solid, pipeline, or other definition is created with an invalid name (for example, a Python keyword). This warning will become an error in the 0.9.13 release.

Community Contributions

  • Added an int type to EventMetadataEntry (Thanks @ChocoletMousse!)
  • Added a build_composite_solid_definition method to Lakehouse (Thanks @sd2k!)
  • Improved broken link detection in Dagster docs (Thanks @keyz!)

New

  • Improvements to log filtering on Run view in Dagit
  • Improvements to instance level scheduler page
  • Log engine events when pipeline termination is initiated

Bugfixes

  • Syntax errors in user code now display the file and line number with the error in Dagit
  • Dask executor no longer fails when using intermediate_storage
  • In the Celery K8s executor, we now mark the step as failed when the step job fails
  • Changed the DagsterInvalidAssetKey error so that it no longer fails upon being thrown

Documentation

  • Added API docs for dagster-dbt experimental library
  • Fixed some cosmetic issues with docs.dagster.io
  • Added code snippets from Solids examples to test path, and fixed some inconsistencies regarding parameter ordering
  • Changed to using markers instead of exact line numbers to mark out code snippets

0.9.10#

Breaking Changes

  • [dagster-dask] Removed the compute option from Dask DataFrame materialization configs for all output types. Setting this option to False (default True) would result in a future that is never computed, leading to missing materializations

Community Contributions

New

  • Console log messages are now streamlined to live on a single line per message
  • Added better messaging around $DAGSTER_HOME if it is not set or improperly setup when starting up a Dagster instance
  • Tools for exporting a file for debugging a run have been added:
    • dagster debug export - a new CLI entry added for exporting a run by id to a file
    • dagit-debug - a new CLI added for loading dagit with a run to debug
    • dagit now has a button to download the debug file for a run via the action menu on the runs page
  • The dagster api grpc command now defaults to the current working directory if none is specified
  • Added retries to dagster-postgres connections
  • Fixed faulty warning message when invoking the same solid multiple times in the same context
  • Added ability to specify custom liveness probe for celery workers in kubernetes deployment

Bugfixes

  • Fixed a bug where Dagster types like List/Set/Tuple/Dict/Optional were not displaying properly on dagit logs
  • Fixed endless spinners on dagit --empty-workspace
  • Fixed incorrect snapshot banner on pipeline view
  • Fixed visual overlapping of overflowing dagit logs
  • Fixed a bug where hanging runs when executing against a gRPC server could cause the Runs page to be unable to load
  • Fixed a bug in celery integration where celery tasks could return None when an iterable is expected, causing errors in the celery execution loop.

Experimental

  • [lakehouse] Each time a Lakehouse solid updates an asset, it automatically generates an AssetMaterialization event
  • [lakehouse] Lakehouse computed_assets now accept a version argument that describes the version of the computation
  • Setting the “dagster/is_memoized_run” tag to true will cause the run to skip any steps whose versions match the versions of outputs produced in prior runs.
  • [dagster-dbt] Solids for running dbt CLI commands
  • Added extensive documentation to illuminate how versions are computed
  • Added versions for step inputs from config, default values, and from other step outputs

0.9.9#

New

  • [Databricks] solids created with create_databricks_job_solid now log a URL for accessing the job in the Databricks UI.
  • The pipeline execute command now defaults to using your current directory if you don’t specify a working directory.

Bugfixes

  • [Celery-K8s] Surface errors to Dagit that previously were not caught in the Celery workers.
  • Fix issues with calling add_run_tags on tags that already exist.
  • Add “Unknown” step state in Dagit’s pipeline run logs view for when pipeline has completed but step has not emitted a completion event

Experimental

  • Version tags for resources and external inputs.

Documentation

  • Fix rendering of example solid config in “Basics of Solids” tutorial.

0.9.8#

New

  • Support for the Dagster step selection DSL: reexecute_pipeline now takes step_selection, which accepts queries like *solid_a.compute++ (i.e., solid_a.compute, all of its ancestors, its immediate descendants, and their immediate descendants). steps_to_execute is deprecated and will be removed in 0.10.0.

Community contributions

  • [dagster-databricks] Improved setup of Databricks environment (Thanks @sd2k!)
  • Enabled frozenlist pickling (Thanks @kinghuang!)

Bugfixes

  • Fixed a bug that pipeline-level hooks were not correctly applied on a pipeline subset.
  • Improved error messages when execute command can't load a code pointer.
  • Fixed a bug that prevented serializing Spark intermediates with configured intermediate storages.

Dagit

  • Enabled subset reexecution via Dagit when part of the pipeline is still running.
  • Made Schedules clickable and link to View All page in the schedule section.
  • Various Dagit UI improvements.

Experimental

  • [lakehouse] Added CLI command for building and executing a pipeline that updates a given set of assets: house update --module package.module —assets my_asset*

Documentation

  • Fixes and improvements.