load_assets_from_x
functions where we began erroring when a spec and AssetsDefinition had the same key in a given module. We now only error in this case if include_specs=True
.show_url_only
option to True
to enable the URL showing behavior.@dbt_assets
decorator would be ignoredload_definitions_from_module
, which can load all the assets, checks, schedules, sensors, and job objects within a module scope into a single Definitions object. Check out the documentation to learn more: https://docs.dagster.io/_apidocs/definitions#dagster.load_definitions_from_module.BackfillPolicy
, or none of them did. Mixed selections are now supported.AssetSpecs
may now contain a partitions_def
. Different AssetSpecs
passed to the same invocation of @multi_asset
can now have different PartitionsDefinitions
, as long as can_subset=True
.title
for asset backfills (not just for job backfills).airbyte_assets
decorator has been added. It can be used with the AirbyteCloudWorkspace
resource and DagsterAirbyteTranslator
translator to load Airbyte tables for a given connection as assets in Dagster. The build_airbyte_assets_definitions
factory can be used to create assets for all the connections in your Airbyte workspace.AirbyteCloudWorkspace.sync_and_poll(…)
method in the definition of a @airbyte_assets
decorator.ecs_executor
which executes Dagster steps via AWS ECS tasks. This can be used in conjunction with ECSRunLauncher
.dbt-core>=1.9
is now supported.load_looker_asset_specs
and build_looker_pdt_assets_definitions
are updated to accept an instance of DagsterLookerApiTranslator
or custom subclass.DagsterLookerApiTranslator.get_asset_spec
have been updated - the parameter looker_structure
is now of type LookerApiTranslatorStructureData
instead of LookerStructureData
. Custom Looker API translators should be updated.load_powerbi_asset_specs
has been updated to accept an instance of DagsterPowerBITranslator
or custom subclass.DagsterPowerBITranslator.get_asset_spec
have been updated - the parameter data
is now of type PowerBITranslatorData
instead of PowerBIContentData
. Custom Power BI translators should be updated.load_sigma_asset_specs
has been updated to accept an instance of DagsterSigmaTranslator
or a custom subclass.DagsterLookerApiTranslator.get_asset_spec
have been updated - the parameter data
is now of type Union[SigmaDatasetTranslatorData, SigmaWorkbookTranslatorData]
instead of Union[SigmaDataset, SigmaWorkbook]
. Custom Looker API translators should be updated.load_tableau_asset_specs
has been updated to accept an instance of DagsterTableauTranslator
or custom subclass.DagsterTableauTranslator.get_asset_spec
have been updated - the parameter data
is now of type TableauTranslatorData
instead of TableauContentData
. Custom Tableau translators should be updated.dagster-airlift
package erroneously introduced a dependency on dagster
. This has been rectified - dagster
is only required for the dagster-airlift[core]
submodule.@multi_asset_sensor
has been rolled back.blobStorageSnapshotUploads
has been added which enables a new process for how definition snapshots are uploaded to Dagster Cloud.cronitor
pin to allow versions >= 5.0.1
to enable use of DayOfWeek
as 7. Cronitor 4.0.0
is still disallowed. (Thanks, @joshuataylor!)checkDbReadyInitContainer
to optionally disable db check initContainer.kind
tags. (Thanks, @dragos-pop!)Re-executions
.kind
tag for display in the UI.DataprocResource
now receives an optional parameter labels
to be attached to Dataproc clusters. (Thanks, @thiagoazcampos!)checkDbReadyInitContainer
flag to the Dagster Helm chart to allow disabling the default init container behavior. (Thanks, @easontm!)build_materialize_workbook_assets_definition
which can be used to build assets that run materialize schedules for a Sigma workbook.SnowflakeResource
and SnowflakeIOManager
both accept additional_snowflake_connection_args
config. This dictionary of arguments will be passed to the snowflake.connector.connect
method. This config will be ignored if you are using the sqlalchemy
connector.BackfillPolicy
are now evaluated correctly during backfills. Self dependent assets no longer result in serial partition submissions or disregarded upstream dependencies.PipesCloudWatchMessageReader
correctly identifies streams which are not ready yet and doesn't fail on ThrottlingException
. (Thanks, @jenkoian!)FivetranWorkspace.sync_and_poll(...).fetch_column_metadata()
.dlt_assets
API docs. (Thanks, @zilto!)dagster/run-id
, dagster/code-location
, dagster/user
and Dagster Cloud environment variables) typically attached to external resources are now available under DagsterRun.dagster_execution_info
.SensorReturnTypesUnion
is now exported for typing the output of sensor functions.@fivetran_assets
decorator.load_fivetran_asset_specs
has been updated to accept an instance of DagsterFivetranTranslator
or custom subclass.fivetran_assets
decorator was added. It can be used with the FivetranWorkspace
resource and DagsterFivetranTranslator
translator to load Fivetran tables for a given connector as assets in Dagster. The build_fivetran_assets_definitions
factory can be used to create assets for all the connectors in your Fivetran workspace.ECSPipesClient.run
now waits up to 70 days for tasks completion (waiter parameters are configurable) (Thanks @jenkoian!)load_airbyte_cloud_asset_specs
function has
been added. It can be used with the AirbyteCloudWorkspace
resource and DagsterAirbyteTranslator
translator to load your Airbyte Cloud connection streams as external assets in Dagster.icechunk
kind.ConsolidatedSqliteEventLogStorage
, which is mostly used for tests.1.6.0
may need to run dagster instance migrate
to enable.map_asset_specs
to enable modifying AssetSpec
s and AssetsDefinition
s in bulk.AssetSpec.replace_attributes
and AssetSpec.merge_attributes
to easily alter properties of an asset spec.PipesS3MessageReader
now has a new parameter include_stdio_in_messages
which enables log forwarding to Dagster via Pipes messages.log_external_stream
has been added. It can be used to forward external logs to Dagster via Pipes messages.load_powerbi_asset_specs(..., use_workspace_scan=False)
.dagster-sigma snapshot
command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.DagsterExecutionStepNotFoundError
errors when trying to execute an asset check step of a run launched by a backfill.owners
added to AssetOut
s when defining a @graph_multi_asset
were not added to the underlying AssetsDefinition
.&
or |
operators on AutomationCondition
s with labels would cause that label to be erased.SigmaFilter
no longer fetch lineage information.DagsterPowerBITranslator.get_asset_key
is deprecated in favor of DagsterPowerBITranslator.get_asset_spec().key
DagsterLookerApiTranslator.get_asset_key
is deprecated in favor of DagsterLookerApiTranslator.get_asset_spec().key
DagsterSigmaTranslator.get_asset_key
is deprecated in favor of DagsterSigmaTranslator.get_asset_spec().key
DagsterTableauTranslator.get_asset_key
is deprecated in favor of DagsterTableauTranslator.get_asset_spec().key
Added run_id
to the run_tags
index to improve database performance. Run dagster instance migrate
to update the index. (Thanks, @HynekBlaha!)
Added icons for kind
tags: Cassandra, ClickHouse, CockroachDB, Doris, Druid, Elasticsearch, Flink, Hadoop, Impala, Kafka, MariaDB, MinIO, Pinot, Presto, Pulsar, RabbitMQ, Redis, Redpanda, ScyllaDB, Starrocks, and Superset. (Thanks, @swrookie!)
Added a new icon for the Denodo kind tag. (Thanks, @tintamarre!)
Errors raised from defining more than one Definitions
object at module scope now include the object names so that the source of the error is easier to determine.
[ui] Asset metadata entries like dagster/row_count
now appear on the events page and are properly hidden on the overview page when they appear in the sidebar.
[dagster-aws] PipesGlueClient
now attaches AWS Glue metadata to Dagster results produced during Pipes invocation.
[dagster-aws] PipesEMRServerlessClient
now attaches AWS EMR Serverless metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the job run.
[dagster-aws] PipesECSClient
now attaches AWS ECS metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the ECS task.
[dagster-aws] PipesEMRClient
now attaches AWS EMR metadata to Dagster results produced during Pipes invocation.
[dagster-databricks] PipesDatabricksClient
now attaches Databricks metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the Databricks job.
[dagster-fivetran] Added load_fivetran_asset_specs
function. It can be used with the FivetranWorkspace
resource and DagsterFivetranTranslator
translator to load your Fivetran connector tables as external assets in Dagster.
[dagster-looker] Errors are now handled more gracefully when parsing derived tables.
[dagster-sigma] Sigma assets now contain extra metadata and kind tags.
[dagster-sigma] Added support for direct workbook to warehouse table dependencies.
[dagster-sigma] Added include_unused_datasets
field to SigmaFilter
to disable pulling datasets that aren't used by a downstream workbook.
[dagster-sigma] Added skip_fetch_column_data
option to skip loading Sigma column lineage. This can speed up loading large instances.
[dagster-sigma] Introduced an experimental dagster-sigma snapshot
command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.
dagster-airlift
(experimental)#dagster-airlift
is coming out of stealth. See the initial Airlift RFC here, and the following documentation to learn more:
More Airflow-related content is coming soon! We'd love for you to check it out, and post any comments / questions in the #airflow-migration
channel in the Dagster slack.
monitor_all_code_locations
and monitored_jobs
did not raise the expected error. (Thanks, @apetryla!)AutomationCondition.any_deps_match()
and AutomationCondition.all_deps_match()
to render incorrectly when allow_selection
or ignore_selection
were set.CacheableAssetsDefinitions
in code locations that contained AutomationConditions
Too many open files
errors for jobs with many steps.dagster-dingtalk
to the list of community supported libraries.dagster-wandb
(Weights and Biases) documentation. (Thanks, @matt-weingarten!)dagster-sigma
documentation.