1.6.0 may need to run dagster instance migrate to enable.map_asset_specs to enable modifying AssetSpecs and AssetsDefinitions in bulk.AssetSpec.replace_attributes and AssetSpec.merge_attributes to easily alter properties of an asset spec.PipesS3MessageReader now has a new parameter include_stdio_in_messages which enables log forwarding to Dagster via Pipes messages.log_external_stream has been added. It can be used to forward external logs to Dagster via Pipes messages.load_powerbi_asset_specs(..., use_workspace_scan=False).dagster-sigma snapshot command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.DagsterExecutionStepNotFoundError errors when trying to execute an asset check step of a run launched by a backfill.owners added to AssetOuts when defining a @graph_multi_asset were not added to the underlying AssetsDefinition.& or | operators on AutomationConditions with labels would cause that label to be erased.SigmaFilter no longer fetch lineage information.DagsterPowerBITranslator.get_asset_key is deprecated in favor of DagsterPowerBITranslator.get_asset_spec().keyDagsterLookerApiTranslator.get_asset_key is deprecated in favor of DagsterLookerApiTranslator.get_asset_spec().keyDagsterSigmaTranslator.get_asset_key is deprecated in favor of DagsterSigmaTranslator.get_asset_spec().keyDagsterTableauTranslator.get_asset_key is deprecated in favor of DagsterTableauTranslator.get_asset_spec().keyAdded run_id to the run_tags index to improve database performance. Run dagster instance migrate to update the index. (Thanks, @HynekBlaha!)
Added icons for kind tags: Cassandra, ClickHouse, CockroachDB, Doris, Druid, Elasticsearch, Flink, Hadoop, Impala, Kafka, MariaDB, MinIO, Pinot, Presto, Pulsar, RabbitMQ, Redis, Redpanda, ScyllaDB, Starrocks, and Superset. (Thanks, @swrookie!)
Added a new icon for the Denodo kind tag. (Thanks, @tintamarre!)
Errors raised from defining more than one Definitions object at module scope now include the object names so that the source of the error is easier to determine.
[ui] Asset metadata entries like dagster/row_count now appear on the events page and are properly hidden on the overview page when they appear in the sidebar.
[dagster-aws] PipesGlueClient now attaches AWS Glue metadata to Dagster results produced during Pipes invocation.
[dagster-aws] PipesEMRServerlessClient now attaches AWS EMR Serverless metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the job run.
[dagster-aws] PipesECSClient now attaches AWS ECS metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the ECS task.
[dagster-aws] PipesEMRClient now attaches AWS EMR metadata to Dagster results produced during Pipes invocation.
[dagster-databricks] PipesDatabricksClient now attaches Databricks metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the Databricks job.
[dagster-fivetran] Added load_fivetran_asset_specs function. It can be used with the FivetranWorkspace resource and DagsterFivetranTranslator translator to load your Fivetran connector tables as external assets in Dagster.
[dagster-looker] Errors are now handled more gracefully when parsing derived tables.
[dagster-sigma] Sigma assets now contain extra metadata and kind tags.
[dagster-sigma] Added support for direct workbook to warehouse table dependencies.
[dagster-sigma] Added include_unused_datasets field to SigmaFilter to disable pulling datasets that aren't used by a downstream workbook.
[dagster-sigma] Added skip_fetch_column_data option to skip loading Sigma column lineage. This can speed up loading large instances.
[dagster-sigma] Introduced an experimental dagster-sigma snapshot command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.
dagster-airlift (experimental)#dagster-airlift is coming out of stealth. See the initial Airlift RFC here, and the following documentation to learn more:
More Airflow-related content is coming soon! We'd love for you to check it out, and post any comments / questions in the #airflow-migration channel in the Dagster slack.
monitor_all_code_locations and monitored_jobs did not raise the expected error. (Thanks, @apetryla!)AutomationCondition.any_deps_match() and AutomationCondition.all_deps_match() to render incorrectly when allow_selection or ignore_selection were set.CacheableAssetsDefinitions in code locations that contained AutomationConditionsToo many open files errors for jobs with many steps.dagster-dingtalk to the list of community supported libraries.dagster-wandb (Weights and Biases) documentation. (Thanks, @matt-weingarten!)dagster-sigma documentation.AssetOut.from_spec, that will construct an AssetOut from an AssetSpec.Column name section of the asset overview page.gcs (Google Cloud Storage) kind tag.report and semanticmodel kind tags.EcsRunLauncher. (Thanks, @zyd14!)DagsterDbtTranslator.get_code_version to customize the code version for your dbt assets. (Thanks, @Grzyblon!)PipesClientCompletedInvocation. This metadata will be attached to all materializations and asset checks stored during the pipes invocation.AutomationCondition.execution_in_progress which would cause it to evaluate to True for unpartitioned assets that were part of a run that was in progress, even if the asset itself had already been materialized.AutomationCondition.run_in_progress that would cause it to ignore queued runs.default_automation_condition_sensor to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy auto_materialize: use_sensors configuration setting was set to False.0 in cases where all partitions were evaluated./ characters now work correctly with Dagster Pipes.dagster-airlift.types-sqlalchemy package is no longer included in the dagster[pyright] extra package.dagster project scaffold now has an option to create dagster projects from templates with excluded files/filepaths.parse_tableau_external_and_materializable_asset_specs is now available to parse a list of Tableau asset specs into a list of external asset specs and materializable asset specs.DagsterFivetranTranslator to customize assets loaded from Fivetran.dagster_snowflake.fetch_last_updated_timestamps now supports ignoring tables not found in Snowflake instead of raising an error.default_automation_condition_sensor to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy auto_materialize: use_sensors configuration setting was set to False.dagster instance migrate on Dagster version 1.9.0 constructed a SQL query that exceeded the maximum allowed depth.ImportErrors are no longer raised when bigquery libraries are not installed [#25708]ShellCommandBlueprint, you can now use slashes as a delimiter to generate an AssetKey with multiple path components.mlflow_run_id attribute (Thanks Joe Percivall!)dagster dev was logging unexpectedly without the grpcio<1.65.0 pin.ContextVar was created in a different context error was raised when executing an async asset.multi_asset type-checker fix from @aksestok, thanks!InputContext passed to an IOManager ’s load_input function when invoking the output_value or output_for_node methods on JobExecutionResult now has the name "dummy_input_name" instead of None.DbtProject is adopted and no longer experimental. Using DbtProject helps achieve a setup where the dbt manifest file and dbt dependencies are available and up-to-date, during development and in production. Check out the API docs for more: https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dagster_dbt.DbtProject.—use-dbt-project flag was introduced for the cli command dagster-dbt project scaffold. Creating a Dagster project wrapping a dbt project using that flag will include a DbtProject .DAGSTER_UI_EVENT_LOAD_CHUNK_SIZE environment variable on the Dagster webserver.RunFailureReason.START_TIMEOUT run monitoring failure reason. Thanks @jobicarter!ObserveResult objects to not be stored with the produced AssetObservation event.metadata defined on SourceAssets to be unavailable when accessed in an IOManager.@graph_asset decorator overload missing an owners argument, thanks @askvinni!execute_k8s_job when there was a transient failure while loading logs from the launched job. Thanks @piotrmarczydlo!dateutil package being installed in the default EMR python evnrionment.AutoMaterializeRule.skip_on_parent_missing rule when a parent asset had its PartitionsDefinition changed.AutomationConditions.AutomationCondition.newly_updated() would trigger on any ASSET_OBSERVATION event. Now, it only triggers when the data version on that event changes.dagster-dbt project prepare-for-deployment has been replaced by dagster-dbt project prepare-and-package.DbtProject no longer prepares the dbt manifest file and dbt dependencies in its constructor during initialization. This process has been moved to prepare_if_dev(), that can be called on the DbtProject instance after initialization. Check out the API docs for more: https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dagster_dbt.DbtProject.prepare_if_dev.GraphDefinition as the job argument to schedules and sensors is deprecated. Derive a job from the GraphDefinition using graph_def.to_job() and pass this instead.dagster-plus CLI in the sidenav to correctly be dagster-cloud CLI.AssetsDefinition construction, enforce single key per output namedagster/last_updated_timestamp.AutoMaterializeRule.skip_on_not_all_parents_updated_since_cron() rule gained a new dependency with a different PartitionsDefinition.source_key_prefix arg of load_assets_from_modules. (thanks @drjlin)!load_assets_from_airflow_dag no longer allows multiple tasks to materialize the same asset.dagster-cloud ci init CLI will now use the --deployment argument as the base deployment when creating a branch deployment. This base deployment will be used for Change Tracking.dbt_with_bigquery_insights now respects CLI arguments for profile configuration and also selects location / dataset from the profile when available.build_freshness_checks_for_dbt_assets which allows users to parameterize freshness checks entirely within dbt. Check out the API docs for more: https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dbt-dagster-dbt.max_partitions_per_run from the job’s constituent assets.asset_tags can now be specified when building dagstermill assetsDagsterSlingTranslatordagster/storage_kind tags attachedtags passed to outs in graph_multi_asset now get correctly propagated to the resulting assets.build_metadata_bounds_checks now no longer errors when targeting metadata keys that have special characters.--read-only flag to the dagster-cloud ci branch-deployment CLI command, which returns the current branch deployment name for the current code repository branch without update the status of the branch deployment.