1.6.0
may need to run dagster instance migrate
to enable.map_asset_specs
to enable modifying AssetSpec
s and AssetsDefinition
s in bulk.AssetSpec.replace_attributes
and AssetSpec.merge_attributes
to easily alter properties of an asset spec.PipesS3MessageReader
now has a new parameter include_stdio_in_messages
which enables log forwarding to Dagster via Pipes messages.log_external_stream
has been added. It can be used to forward external logs to Dagster via Pipes messages.load_powerbi_asset_specs(..., use_workspace_scan=False)
.dagster-sigma snapshot
command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.DagsterExecutionStepNotFoundError
errors when trying to execute an asset check step of a run launched by a backfill.owners
added to AssetOut
s when defining a @graph_multi_asset
were not added to the underlying AssetsDefinition
.&
or |
operators on AutomationCondition
s with labels would cause that label to be erased.SigmaFilter
no longer fetch lineage information.DagsterPowerBITranslator.get_asset_key
is deprecated in favor of DagsterPowerBITranslator.get_asset_spec().key
DagsterLookerApiTranslator.get_asset_key
is deprecated in favor of DagsterLookerApiTranslator.get_asset_spec().key
DagsterSigmaTranslator.get_asset_key
is deprecated in favor of DagsterSigmaTranslator.get_asset_spec().key
DagsterTableauTranslator.get_asset_key
is deprecated in favor of DagsterTableauTranslator.get_asset_spec().key
Added run_id
to the run_tags
index to improve database performance. Run dagster instance migrate
to update the index. (Thanks, @HynekBlaha!)
Added icons for kind
tags: Cassandra, ClickHouse, CockroachDB, Doris, Druid, Elasticsearch, Flink, Hadoop, Impala, Kafka, MariaDB, MinIO, Pinot, Presto, Pulsar, RabbitMQ, Redis, Redpanda, ScyllaDB, Starrocks, and Superset. (Thanks, @swrookie!)
Added a new icon for the Denodo kind tag. (Thanks, @tintamarre!)
Errors raised from defining more than one Definitions
object at module scope now include the object names so that the source of the error is easier to determine.
[ui] Asset metadata entries like dagster/row_count
now appear on the events page and are properly hidden on the overview page when they appear in the sidebar.
[dagster-aws] PipesGlueClient
now attaches AWS Glue metadata to Dagster results produced during Pipes invocation.
[dagster-aws] PipesEMRServerlessClient
now attaches AWS EMR Serverless metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the job run.
[dagster-aws] PipesECSClient
now attaches AWS ECS metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the ECS task.
[dagster-aws] PipesEMRClient
now attaches AWS EMR metadata to Dagster results produced during Pipes invocation.
[dagster-databricks] PipesDatabricksClient
now attaches Databricks metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the Databricks job.
[dagster-fivetran] Added load_fivetran_asset_specs
function. It can be used with the FivetranWorkspace
resource and DagsterFivetranTranslator
translator to load your Fivetran connector tables as external assets in Dagster.
[dagster-looker] Errors are now handled more gracefully when parsing derived tables.
[dagster-sigma] Sigma assets now contain extra metadata and kind tags.
[dagster-sigma] Added support for direct workbook to warehouse table dependencies.
[dagster-sigma] Added include_unused_datasets
field to SigmaFilter
to disable pulling datasets that aren't used by a downstream workbook.
[dagster-sigma] Added skip_fetch_column_data
option to skip loading Sigma column lineage. This can speed up loading large instances.
[dagster-sigma] Introduced an experimental dagster-sigma snapshot
command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.
dagster-airlift
(experimental)#dagster-airlift
is coming out of stealth. See the initial Airlift RFC here, and the following documentation to learn more:
More Airflow-related content is coming soon! We'd love for you to check it out, and post any comments / questions in the #airflow-migration
channel in the Dagster slack.
monitor_all_code_locations
and monitored_jobs
did not raise the expected error. (Thanks, @apetryla!)AutomationCondition.any_deps_match()
and AutomationCondition.all_deps_match()
to render incorrectly when allow_selection
or ignore_selection
were set.CacheableAssetsDefinitions
in code locations that contained AutomationConditions
Too many open files
errors for jobs with many steps.dagster-dingtalk
to the list of community supported libraries.dagster-wandb
(Weights and Biases) documentation. (Thanks, @matt-weingarten!)dagster-sigma
documentation.AssetOut.from_spec
, that will construct an AssetOut
from an AssetSpec
.Column name
section of the asset overview page.gcs
(Google Cloud Storage) kind tag.report
and semanticmodel
kind tags.EcsRunLauncher
. (Thanks, @zyd14!)DagsterDbtTranslator.get_code_version
to customize the code version for your dbt assets. (Thanks, @Grzyblon!)PipesClientCompletedInvocation
. This metadata will be attached to all materializations and asset checks stored during the pipes invocation.AutomationCondition.execution_in_progress
which would cause it to evaluate to True
for unpartitioned assets that were part of a run that was in progress, even if the asset itself had already been materialized.AutomationCondition.run_in_progress
that would cause it to ignore queued runs.default_automation_condition_sensor
to be constructed for user code servers running on dagster version < 1.9.0
even if the legacy auto_materialize: use_sensors
configuration setting was set to False
.0
in cases where all partitions were evaluated./
characters now work correctly with Dagster Pipes.dagster-airlift
.types-sqlalchemy
package is no longer included in the dagster[pyright]
extra package.dagster project scaffold
now has an option to create dagster projects from templates with excluded files/filepaths.parse_tableau_external_and_materializable_asset_specs
is now available to parse a list of Tableau asset specs into a list of external asset specs and materializable asset specs.DagsterFivetranTranslator
to customize assets loaded from Fivetran.dagster_snowflake.fetch_last_updated_timestamps
now supports ignoring tables not found in Snowflake instead of raising an error.default_automation_condition_sensor
to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy auto_materialize: use_sensors
configuration setting was set to False
.dagster instance migrate
on Dagster version 1.9.0 constructed a SQL query that exceeded the maximum allowed depth.ImportError
s are no longer raised when bigquery libraries are not installed [#25708]ExecutorDefinition
and @executor
APIs are used to define in-process, multiprocess, and Dask executors, and may be used by users to
define new executors. Like loggers and storage, executors may be added to a ModeDefinition
and
may be selected and configured through the execution
field in the environment dict or YAML,
including through Dagit. Executors may no longer be configured through the RunConfig
.execute_pipeline
API, and the Dask executor is configured through the environment.
(See the dagster-dask README for details.)PresetDefinition.from_files
API for constructing a preset from a list of environment
files (replacing the old usage of this class). PresetDefinition
may now be directly
instantiated with an environment dict.--raise-on-error
or --no-raise-on-error
flag. Set this
option in executor config.MarkdownMetadataEntryData
class, so events yielded from client code may now render
markdown in their metadata.DAGIT_
, e.g. DAGIT_PORT
.