1.6.0
may need to run dagster instance migrate
to enable.map_asset_specs
to enable modifying AssetSpec
s and AssetsDefinition
s in bulk.AssetSpec.replace_attributes
and AssetSpec.merge_attributes
to easily alter properties of an asset spec.PipesS3MessageReader
now has a new parameter include_stdio_in_messages
which enables log forwarding to Dagster via Pipes messages.log_external_stream
has been added. It can be used to forward external logs to Dagster via Pipes messages.load_powerbi_asset_specs(..., use_workspace_scan=False)
.dagster-sigma snapshot
command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.DagsterExecutionStepNotFoundError
errors when trying to execute an asset check step of a run launched by a backfill.owners
added to AssetOut
s when defining a @graph_multi_asset
were not added to the underlying AssetsDefinition
.&
or |
operators on AutomationCondition
s with labels would cause that label to be erased.SigmaFilter
no longer fetch lineage information.DagsterPowerBITranslator.get_asset_key
is deprecated in favor of DagsterPowerBITranslator.get_asset_spec().key
DagsterLookerApiTranslator.get_asset_key
is deprecated in favor of DagsterLookerApiTranslator.get_asset_spec().key
DagsterSigmaTranslator.get_asset_key
is deprecated in favor of DagsterSigmaTranslator.get_asset_spec().key
DagsterTableauTranslator.get_asset_key
is deprecated in favor of DagsterTableauTranslator.get_asset_spec().key
Added run_id
to the run_tags
index to improve database performance. Run dagster instance migrate
to update the index. (Thanks, @HynekBlaha!)
Added icons for kind
tags: Cassandra, ClickHouse, CockroachDB, Doris, Druid, Elasticsearch, Flink, Hadoop, Impala, Kafka, MariaDB, MinIO, Pinot, Presto, Pulsar, RabbitMQ, Redis, Redpanda, ScyllaDB, Starrocks, and Superset. (Thanks, @swrookie!)
Added a new icon for the Denodo kind tag. (Thanks, @tintamarre!)
Errors raised from defining more than one Definitions
object at module scope now include the object names so that the source of the error is easier to determine.
[ui] Asset metadata entries like dagster/row_count
now appear on the events page and are properly hidden on the overview page when they appear in the sidebar.
[dagster-aws] PipesGlueClient
now attaches AWS Glue metadata to Dagster results produced during Pipes invocation.
[dagster-aws] PipesEMRServerlessClient
now attaches AWS EMR Serverless metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the job run.
[dagster-aws] PipesECSClient
now attaches AWS ECS metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the ECS task.
[dagster-aws] PipesEMRClient
now attaches AWS EMR metadata to Dagster results produced during Pipes invocation.
[dagster-databricks] PipesDatabricksClient
now attaches Databricks metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the Databricks job.
[dagster-fivetran] Added load_fivetran_asset_specs
function. It can be used with the FivetranWorkspace
resource and DagsterFivetranTranslator
translator to load your Fivetran connector tables as external assets in Dagster.
[dagster-looker] Errors are now handled more gracefully when parsing derived tables.
[dagster-sigma] Sigma assets now contain extra metadata and kind tags.
[dagster-sigma] Added support for direct workbook to warehouse table dependencies.
[dagster-sigma] Added include_unused_datasets
field to SigmaFilter
to disable pulling datasets that aren't used by a downstream workbook.
[dagster-sigma] Added skip_fetch_column_data
option to skip loading Sigma column lineage. This can speed up loading large instances.
[dagster-sigma] Introduced an experimental dagster-sigma snapshot
command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.
dagster-airlift
(experimental)#dagster-airlift
is coming out of stealth. See the initial Airlift RFC here, and the following documentation to learn more:
More Airflow-related content is coming soon! We'd love for you to check it out, and post any comments / questions in the #airflow-migration
channel in the Dagster slack.
monitor_all_code_locations
and monitored_jobs
did not raise the expected error. (Thanks, @apetryla!)AutomationCondition.any_deps_match()
and AutomationCondition.all_deps_match()
to render incorrectly when allow_selection
or ignore_selection
were set.CacheableAssetsDefinitions
in code locations that contained AutomationConditions
Too many open files
errors for jobs with many steps.dagster-dingtalk
to the list of community supported libraries.dagster-wandb
(Weights and Biases) documentation. (Thanks, @matt-weingarten!)dagster-sigma
documentation.AssetOut.from_spec
, that will construct an AssetOut
from an AssetSpec
.Column name
section of the asset overview page.gcs
(Google Cloud Storage) kind tag.report
and semanticmodel
kind tags.EcsRunLauncher
. (Thanks, @zyd14!)DagsterDbtTranslator.get_code_version
to customize the code version for your dbt assets. (Thanks, @Grzyblon!)PipesClientCompletedInvocation
. This metadata will be attached to all materializations and asset checks stored during the pipes invocation.AutomationCondition.execution_in_progress
which would cause it to evaluate to True
for unpartitioned assets that were part of a run that was in progress, even if the asset itself had already been materialized.AutomationCondition.run_in_progress
that would cause it to ignore queued runs.default_automation_condition_sensor
to be constructed for user code servers running on dagster version < 1.9.0
even if the legacy auto_materialize: use_sensors
configuration setting was set to False
.0
in cases where all partitions were evaluated./
characters now work correctly with Dagster Pipes.dagster-airlift
.types-sqlalchemy
package is no longer included in the dagster[pyright]
extra package.dagster project scaffold
now has an option to create dagster projects from templates with excluded files/filepaths.parse_tableau_external_and_materializable_asset_specs
is now available to parse a list of Tableau asset specs into a list of external asset specs and materializable asset specs.DagsterFivetranTranslator
to customize assets loaded from Fivetran.dagster_snowflake.fetch_last_updated_timestamps
now supports ignoring tables not found in Snowflake instead of raising an error.default_automation_condition_sensor
to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy auto_materialize: use_sensors
configuration setting was set to False
.dagster instance migrate
on Dagster version 1.9.0 constructed a SQL query that exceeded the maximum allowed depth.ImportError
s are no longer raised when bigquery libraries are not installed [#25708]dagster/storage_kind
tag.dbt retry
within a try/except block to avoid unnecessary, duplicate work.AssetExecutionContext
now exposes a has_partition_key_range
property.owners
, metadata
, tags
, and deps
properties on AssetSpec
are no longer Optional
. The AssetSpec
constructor still accepts None
values, which are coerced to empty collections of the relevant type.docker_executor
and k8s_job_executor
now consider at most 1000 events at a time when loading events from the current run to determine which steps should be launched. This value can be tuned by setting the DAGSTER_EXECUTOR_POP_EVENTS_LIMIT
environment variable in the run process.dagster/retry_on_asset_or_op_failure
tag that can be added to jobs to override run retry behavior for runs of specific jobs. See the docs for more information.build_sensor_for_freshness_checks
to describe when/why it skips evaluating freshness checks.@multi_asset_sensor
.ScheduleDefinition
now properly supports being passed a RunConfig
object.MaterializeResult
, but the function has no type annotation, previously, the IO manager would still be invoked with a None
value. Now, the IO manager is not invoked.AssetSpec
constructor now raises an error if an invalid owner string is passed to it.graph_multi_asset
decorator, the code_version
property on AssetOut
s passed in used to be ignored. Now, they no longer are.dagster-cloud job launch
command did not support specifying asset keys with prefixes in the --asset-key
argument.group:
, code location:
, tag:
, owner:
.Definitions
now has a get_all_asset_specs
method, which allows iterating over properties of the defined assetsterminate_runs
method to the Python GraphQL Client. (thanks @baumann-t!)DbtCliInvocation
now has a .get_error()
method that can be useful when using dbt.cli(..., raise_on_error=False)
.DynamicPartitionsDefinition
(using partitions_fn
) that caused a crash during job backfills.build_metadata_bounds_checks
API creates asset checks which verify that numeric metadata values on asset materializations fall within min or max values. See the documentation for more information.build_sensor_for_freshness_checks
and Dagster Plus. This API should now work when used with Dagster Plus.dagsterCloudAgent.additionalPodSpecConfig
to the Kubernetes agent Helm chart allowing arbitrary pod configuration to be applied to the agent pod.datetime.utcfromtimestamp
(thanks @dbrtly!)dbt-core==1.8.*
.BackfillPolicy
s, each asset would get materialized in its own run, rather than grouping assets together into single run.dagster/row_count
.CloudwatchLogsHandler
, ECRPublicClient
, SecretsManagerResource
, SSMResource
thanks @jacob-white-simplisafe
!TableMetadataValue
, TableSchemaMetadataValue
, or TableColumnLineageMetadataValue
defined.BackfillPolicy
of the underlying assets in the job.databricks-sdk
version bumped to 0.17.0
, thanks @lamalex
!dagster code-server start
, thanks @SanjaySiddharth
!@JonathanLai2004
!MaterializeResult
, ObserveResult
, or Output
, you can now include tags that will be attached to the corresponding AssetMaterialization
or AssetObservation
event. These tags will be rendered on these events in the UI.build_last_update_freshness_checks
and build_time_partition_freshness_checks
APIs have been updated to be clearer.%
into the asset graph’s query selector no longer crashes the UI.MetadataValue
have been changed from NamedTuple
s to Pydantic models. NamedTuple
functionality on these classes was not part of Dagster’s stable public API, but usages relying on their tuple-ness may break. For example: calling json.dumps
on collections that include them.dbt-core==1.5.*
has been removed, as it has reached end of life in April 2024.dagster-cloud
CLI where the --deployment
argument was ignored when the DAGSTER_CLOUD_URL
environment variable was set.dagster-cloud-cli
package wouldn’t work unless the dagster-cloud
package was installed as well.