cronitor pin to allow versions >= 5.0.1 to enable use of DayOfWeek as 7. Cronitor 4.0.0 is still disallowed. (Thanks, @joshuataylor!)checkDbReadyInitContainer to optionally disable db check initContainer.kind tags. (Thanks, @dragos-pop!)Re-executions.kind tag for display in the UI.DataprocResource now receives an optional parameter labels to be attached to Dataproc clusters. (Thanks, @thiagoazcampos!)checkDbReadyInitContainer flag to the Dagster Helm chart to allow disabling the default init container behavior. (Thanks, @easontm!)build_materialize_workbook_assets_definition which can be used to build assets that run materialize schedules for a Sigma workbook.SnowflakeResource and SnowflakeIOManager both accept additional_snowflake_connection_args config. This dictionary of arguments will be passed to the snowflake.connector.connect method. This config will be ignored if you are using the sqlalchemy connector.BackfillPolicy are now evaluated correctly during backfills. Self dependent assets no longer result in serial partition submissions or disregarded upstream dependencies.PipesCloudWatchMessageReader correctly identifies streams which are not ready yet and doesn't fail on ThrottlingException. (Thanks, @jenkoian!)FivetranWorkspace.sync_and_poll(...).fetch_column_metadata().dlt_assets API docs. (Thanks, @zilto!)dagster/run-id, dagster/code-location, dagster/user and Dagster Cloud environment variables) typically attached to external resources are now available under DagsterRun.dagster_execution_info.SensorReturnTypesUnion is now exported for typing the output of sensor functions.@fivetran_assets decorator.load_fivetran_asset_specs has been updated to accept an instance of DagsterFivetranTranslator or custom subclass.fivetran_assets decorator was added. It can be used with the FivetranWorkspace resource and DagsterFivetranTranslator translator to load Fivetran tables for a given connector as assets in Dagster. The build_fivetran_assets_definitions factory can be used to create assets for all the connectors in your Fivetran workspace.ECSPipesClient.run now waits up to 70 days for tasks completion (waiter parameters are configurable) (Thanks @jenkoian!)load_airbyte_cloud_asset_specs function has
been added. It can be used with the AirbyteCloudWorkspace resource and DagsterAirbyteTranslator translator to load your Airbyte Cloud connection streams as external assets in Dagster.icechunk kind.ConsolidatedSqliteEventLogStorage, which is mostly used for tests.1.6.0 may need to run dagster instance migrate to enable.map_asset_specs to enable modifying AssetSpecs and AssetsDefinitions in bulk.AssetSpec.replace_attributes and AssetSpec.merge_attributes to easily alter properties of an asset spec.PipesS3MessageReader now has a new parameter include_stdio_in_messages which enables log forwarding to Dagster via Pipes messages.log_external_stream has been added. It can be used to forward external logs to Dagster via Pipes messages.load_powerbi_asset_specs(..., use_workspace_scan=False).dagster-sigma snapshot command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.DagsterExecutionStepNotFoundError errors when trying to execute an asset check step of a run launched by a backfill.owners added to AssetOuts when defining a @graph_multi_asset were not added to the underlying AssetsDefinition.& or | operators on AutomationConditions with labels would cause that label to be erased.SigmaFilter no longer fetch lineage information.DagsterPowerBITranslator.get_asset_key is deprecated in favor of DagsterPowerBITranslator.get_asset_spec().keyDagsterLookerApiTranslator.get_asset_key is deprecated in favor of DagsterLookerApiTranslator.get_asset_spec().keyDagsterSigmaTranslator.get_asset_key is deprecated in favor of DagsterSigmaTranslator.get_asset_spec().keyDagsterTableauTranslator.get_asset_key is deprecated in favor of DagsterTableauTranslator.get_asset_spec().keyAdded run_id to the run_tags index to improve database performance. Run dagster instance migrate to update the index. (Thanks, @HynekBlaha!)
Added icons for kind tags: Cassandra, ClickHouse, CockroachDB, Doris, Druid, Elasticsearch, Flink, Hadoop, Impala, Kafka, MariaDB, MinIO, Pinot, Presto, Pulsar, RabbitMQ, Redis, Redpanda, ScyllaDB, Starrocks, and Superset. (Thanks, @swrookie!)
Added a new icon for the Denodo kind tag. (Thanks, @tintamarre!)
Errors raised from defining more than one Definitions object at module scope now include the object names so that the source of the error is easier to determine.
[ui] Asset metadata entries like dagster/row_count now appear on the events page and are properly hidden on the overview page when they appear in the sidebar.
[dagster-aws] PipesGlueClient now attaches AWS Glue metadata to Dagster results produced during Pipes invocation.
[dagster-aws] PipesEMRServerlessClient now attaches AWS EMR Serverless metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the job run.
[dagster-aws] PipesECSClient now attaches AWS ECS metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the ECS task.
[dagster-aws] PipesEMRClient now attaches AWS EMR metadata to Dagster results produced during Pipes invocation.
[dagster-databricks] PipesDatabricksClient now attaches Databricks metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the Databricks job.
[dagster-fivetran] Added load_fivetran_asset_specs function. It can be used with the FivetranWorkspace resource and DagsterFivetranTranslator translator to load your Fivetran connector tables as external assets in Dagster.
[dagster-looker] Errors are now handled more gracefully when parsing derived tables.
[dagster-sigma] Sigma assets now contain extra metadata and kind tags.
[dagster-sigma] Added support for direct workbook to warehouse table dependencies.
[dagster-sigma] Added include_unused_datasets field to SigmaFilter to disable pulling datasets that aren't used by a downstream workbook.
[dagster-sigma] Added skip_fetch_column_data option to skip loading Sigma column lineage. This can speed up loading large instances.
[dagster-sigma] Introduced an experimental dagster-sigma snapshot command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.
dagster-airlift (experimental)#dagster-airlift is coming out of stealth. See the initial Airlift RFC here, and the following documentation to learn more:
More Airflow-related content is coming soon! We'd love for you to check it out, and post any comments / questions in the #airflow-migration channel in the Dagster slack.
monitor_all_code_locations and monitored_jobs did not raise the expected error. (Thanks, @apetryla!)AutomationCondition.any_deps_match() and AutomationCondition.all_deps_match() to render incorrectly when allow_selection or ignore_selection were set.CacheableAssetsDefinitions in code locations that contained AutomationConditionsToo many open files errors for jobs with many steps.dagster-dingtalk to the list of community supported libraries.dagster-wandb (Weights and Biases) documentation. (Thanks, @matt-weingarten!)dagster-sigma documentation.AssetOut.from_spec, that will construct an AssetOut from an AssetSpec.Column name section of the asset overview page.gcs (Google Cloud Storage) kind tag.report and semanticmodel kind tags.EcsRunLauncher. (Thanks, @zyd14!)DagsterDbtTranslator.get_code_version to customize the code version for your dbt assets. (Thanks, @Grzyblon!)PipesClientCompletedInvocation. This metadata will be attached to all materializations and asset checks stored during the pipes invocation.AutomationCondition.execution_in_progress which would cause it to evaluate to True for unpartitioned assets that were part of a run that was in progress, even if the asset itself had already been materialized.AutomationCondition.run_in_progress that would cause it to ignore queued runs.default_automation_condition_sensor to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy auto_materialize: use_sensors configuration setting was set to False.0 in cases where all partitions were evaluated./ characters now work correctly with Dagster Pipes.dagster-airlift.types-sqlalchemy package is no longer included in the dagster[pyright] extra package.dagster project scaffold now has an option to create dagster projects from templates with excluded files/filepaths.parse_tableau_external_and_materializable_asset_specs is now available to parse a list of Tableau asset specs into a list of external asset specs and materializable asset specs.DagsterFivetranTranslator to customize assets loaded from Fivetran.dagster_snowflake.fetch_last_updated_timestamps now supports ignoring tables not found in Snowflake instead of raising an error.default_automation_condition_sensor to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy auto_materialize: use_sensors configuration setting was set to False.dagster instance migrate on Dagster version 1.9.0 constructed a SQL query that exceeded the maximum allowed depth.ImportErrors are no longer raised when bigquery libraries are not installed [#25708]Community Contributions
k8s-example by 25% (104 MB) (thanks @alex-treebeard and @mrdavidlaing!)snowflake_resource can now be configured to use the SQLAlchemy connector (thanks @basilvetas!)New
userDeployments.deployments in the Helm chart, replicaCount now defaults to 1 if not specified.Bugfixes
env, envConfigMaps, and envSecrets.Documentation
QueuedRunCoordinator to limit run concurrency.SystemCronScheduler or K8sScheduler to the new scheduler.IOManager abstraction provides a new, streamlined primitive for granular control over where
and how solid outputs are stored and loaded. This is intended to replace the (deprecated)
intermediate/system storage abstractions, See the
IO Manager Overview for more
information.required_resource_keys parameter on @resource.DynamicOutputDefinition API.
Dagster can now map the downstream dependencies over a dynamic output at runtime.Dropping Python 2 support
Removal of deprecated APIs
These APIs were marked for deprecation with warnings in the 0.9.0 release, and have been removed in the 0.10.0 release.
input_hydration_config has been removed. Use the dagster_type_loader decorator
instead.output_materialization_config has been removed. Use dagster_type_materializer
instead.SystemStorageDefinition,
@system_storage, and default_system_storage_defs . Use the new IOManagers API instead. See
the IO Manager Overview for more
information.config_field argument on decorators and definitions classes has been removed and replaced
with config_schema. This is a drop-in rename.step_keys_to_execute to the functions reexecute_pipeline and
reexecute_pipeline_iterator has been removed. Use the step_selection argument to select
subsets for execution instead.repository key in your workspace.yaml;
use load_from instead. See the
Workspaces Overview for
documentation about how to define a workspace.Breaking API Changes
SolidExecutionResult.compute_output_event_dict has been renamed to
SolidExecutionResult.compute_output_events_dict. A solid execution result is returned from
methods such as result_for_solid. Any call sites will need to be updated..compute suffix is no longer applied to step keys. Step keys that were previously named
my_solid.compute will now be named my_solid. If you are using any API method that takes a
step_selection argument, you will need to update the step keys accordingly.pipeline_def property has been removed from the InitResourceContext passed to functions
decorated with @resource.Dagstermill
If you are using define_dagstermill_solid with the output_notebook parameter set to True,
you will now need to provide a file manager resource (subclass of
dagster.core.storage.FileManager) on your pipeline mode under the resource key "file_manager",
e.g.:
from dagster import ModeDefinition, local_file_manager, pipeline from dagstermill import define_dagstermill_solid my_dagstermill_solid = define_dagstermill_solid("my_dagstermill_solid", output_notebook=True, ...) @pipeline(mode_defs=[ModeDefinition(resource_defs={"file_manager": local_file_manager})]) def my_dagstermill_pipeline(): my_dagstermill_solid(...)
Helm Chart
scheduler values in the helm chart has changed. Instead of a simple toggle
on/off, we now require an explicit scheduler.type to specify usage of the
DagsterDaemonScheduler, K8sScheduler, or otherwise. If your specified scheduler.type has
required config, these fields must be specified under scheduler.config.snake_case fields have been changed to camelCase. Please update your values.yaml as follows:pipeline_run → pipelineRundagster_home → dagsterHomeenv_secrets → envSecretsenv_config_maps → envConfigMapscelery and k8sRunLauncher have now been consolidated under the Helm value
runLauncher for simplicity. Use the field runLauncher.type to specify usage of the
K8sRunLauncher, CeleryK8sRunLauncher, or otherwise. By default, the K8sRunLauncher is
enabled.CeleryK8sRunLauncher, you should explicitly enable your message broker of choice.userDeployments are now enabled by default.Event log messages streamed to stdout and stderr have been streamlined to be a single line
per event.
Experimental support for memoization and versioning lets you execute pipelines incrementally, selecting which solids need to be rerun based on runtime criteria and versioning their outputs with configurable identifiers that capture their upstream dependencies.
To set up memoized step selection, users can provide a MemoizableIOManager, whose has_output
function decides whether a given solid output needs to be computed or already exists. To execute
a pipeline with memoized step selection, users can supply the dagster/is_memoized_run run tag
to execute_pipeline.
To set the version on a solid or resource, users can supply the version field on the definition.
To access the derived version for a step output, users can access the version field on the
OutputContext passed to the handle_output and load_input methods of IOManager and the
has_output method of MemoizableIOManager.
Schedules that are executed using the new DagsterDaemonScheduler can now execute in any
timezone by adding an execution_timezone parameter to the schedule. Daylight Savings Time
transitions are also supported. See the
Schedules Overview for
more information and examples.
Helm
We've added schema validation to our Helm chart. You can now check that your values YAML file is correct by running:
helm lint helm/dagster -f helm/dagster/values.yaml
Added support for resource annotations throughout our Helm chart.
Added Helm deployment of the dagster daemon & daemon scheduler.
Added Helm support for configuring a compute log manager in your dagster instance.
User code deployments now include a user ConfigMap by default.
Changed the default liveness probe for Dagit to use httpGet "/dagit_info" instead of
tcpSocket:80
Dagster-K8s [Kubernetes]
Dagster-Celery-K8s
dagster-docker library with a DockerRunLauncher that launches each run in its own
Docker container. (See Deploying with Docker docs
for an example.)create_databricks_job_solid for creating solids that launch Databricks jobs.Bugfixes
New
Bugfixes
Community Contributions
Bugfixes
mher/flower:0.9.5 for the Flower pod.