cronitor pin to allow versions >= 5.0.1 to enable use of DayOfWeek as 7. Cronitor 4.0.0 is still disallowed. (Thanks, @joshuataylor!)checkDbReadyInitContainer to optionally disable db check initContainer.kind tags. (Thanks, @dragos-pop!)Re-executions.kind tag for display in the UI.DataprocResource now receives an optional parameter labels to be attached to Dataproc clusters. (Thanks, @thiagoazcampos!)checkDbReadyInitContainer flag to the Dagster Helm chart to allow disabling the default init container behavior. (Thanks, @easontm!)build_materialize_workbook_assets_definition which can be used to build assets that run materialize schedules for a Sigma workbook.SnowflakeResource and SnowflakeIOManager both accept additional_snowflake_connection_args config. This dictionary of arguments will be passed to the snowflake.connector.connect method. This config will be ignored if you are using the sqlalchemy connector.BackfillPolicy are now evaluated correctly during backfills. Self dependent assets no longer result in serial partition submissions or disregarded upstream dependencies.PipesCloudWatchMessageReader correctly identifies streams which are not ready yet and doesn't fail on ThrottlingException. (Thanks, @jenkoian!)FivetranWorkspace.sync_and_poll(...).fetch_column_metadata().dlt_assets API docs. (Thanks, @zilto!)dagster/run-id, dagster/code-location, dagster/user and Dagster Cloud environment variables) typically attached to external resources are now available under DagsterRun.dagster_execution_info.SensorReturnTypesUnion is now exported for typing the output of sensor functions.@fivetran_assets decorator.load_fivetran_asset_specs has been updated to accept an instance of DagsterFivetranTranslator or custom subclass.fivetran_assets decorator was added. It can be used with the FivetranWorkspace resource and DagsterFivetranTranslator translator to load Fivetran tables for a given connector as assets in Dagster. The build_fivetran_assets_definitions factory can be used to create assets for all the connectors in your Fivetran workspace.ECSPipesClient.run now waits up to 70 days for tasks completion (waiter parameters are configurable) (Thanks @jenkoian!)load_airbyte_cloud_asset_specs function has
been added. It can be used with the AirbyteCloudWorkspace resource and DagsterAirbyteTranslator translator to load your Airbyte Cloud connection streams as external assets in Dagster.icechunk kind.ConsolidatedSqliteEventLogStorage, which is mostly used for tests.1.6.0 may need to run dagster instance migrate to enable.map_asset_specs to enable modifying AssetSpecs and AssetsDefinitions in bulk.AssetSpec.replace_attributes and AssetSpec.merge_attributes to easily alter properties of an asset spec.PipesS3MessageReader now has a new parameter include_stdio_in_messages which enables log forwarding to Dagster via Pipes messages.log_external_stream has been added. It can be used to forward external logs to Dagster via Pipes messages.load_powerbi_asset_specs(..., use_workspace_scan=False).dagster-sigma snapshot command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.DagsterExecutionStepNotFoundError errors when trying to execute an asset check step of a run launched by a backfill.owners added to AssetOuts when defining a @graph_multi_asset were not added to the underlying AssetsDefinition.& or | operators on AutomationConditions with labels would cause that label to be erased.SigmaFilter no longer fetch lineage information.DagsterPowerBITranslator.get_asset_key is deprecated in favor of DagsterPowerBITranslator.get_asset_spec().keyDagsterLookerApiTranslator.get_asset_key is deprecated in favor of DagsterLookerApiTranslator.get_asset_spec().keyDagsterSigmaTranslator.get_asset_key is deprecated in favor of DagsterSigmaTranslator.get_asset_spec().keyDagsterTableauTranslator.get_asset_key is deprecated in favor of DagsterTableauTranslator.get_asset_spec().keyAdded run_id to the run_tags index to improve database performance. Run dagster instance migrate to update the index. (Thanks, @HynekBlaha!)
Added icons for kind tags: Cassandra, ClickHouse, CockroachDB, Doris, Druid, Elasticsearch, Flink, Hadoop, Impala, Kafka, MariaDB, MinIO, Pinot, Presto, Pulsar, RabbitMQ, Redis, Redpanda, ScyllaDB, Starrocks, and Superset. (Thanks, @swrookie!)
Added a new icon for the Denodo kind tag. (Thanks, @tintamarre!)
Errors raised from defining more than one Definitions object at module scope now include the object names so that the source of the error is easier to determine.
[ui] Asset metadata entries like dagster/row_count now appear on the events page and are properly hidden on the overview page when they appear in the sidebar.
[dagster-aws] PipesGlueClient now attaches AWS Glue metadata to Dagster results produced during Pipes invocation.
[dagster-aws] PipesEMRServerlessClient now attaches AWS EMR Serverless metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the job run.
[dagster-aws] PipesECSClient now attaches AWS ECS metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the ECS task.
[dagster-aws] PipesEMRClient now attaches AWS EMR metadata to Dagster results produced during Pipes invocation.
[dagster-databricks] PipesDatabricksClient now attaches Databricks metadata to Dagster results produced during Pipes invocation and adds Dagster tags to the Databricks job.
[dagster-fivetran] Added load_fivetran_asset_specs function. It can be used with the FivetranWorkspace resource and DagsterFivetranTranslator translator to load your Fivetran connector tables as external assets in Dagster.
[dagster-looker] Errors are now handled more gracefully when parsing derived tables.
[dagster-sigma] Sigma assets now contain extra metadata and kind tags.
[dagster-sigma] Added support for direct workbook to warehouse table dependencies.
[dagster-sigma] Added include_unused_datasets field to SigmaFilter to disable pulling datasets that aren't used by a downstream workbook.
[dagster-sigma] Added skip_fetch_column_data option to skip loading Sigma column lineage. This can speed up loading large instances.
[dagster-sigma] Introduced an experimental dagster-sigma snapshot command, allowing Sigma workspaces to be captured to a file for faster subsequent loading.
dagster-airlift (experimental)#dagster-airlift is coming out of stealth. See the initial Airlift RFC here, and the following documentation to learn more:
More Airflow-related content is coming soon! We'd love for you to check it out, and post any comments / questions in the #airflow-migration channel in the Dagster slack.
monitor_all_code_locations and monitored_jobs did not raise the expected error. (Thanks, @apetryla!)AutomationCondition.any_deps_match() and AutomationCondition.all_deps_match() to render incorrectly when allow_selection or ignore_selection were set.CacheableAssetsDefinitions in code locations that contained AutomationConditionsToo many open files errors for jobs with many steps.dagster-dingtalk to the list of community supported libraries.dagster-wandb (Weights and Biases) documentation. (Thanks, @matt-weingarten!)dagster-sigma documentation.AssetOut.from_spec, that will construct an AssetOut from an AssetSpec.Column name section of the asset overview page.gcs (Google Cloud Storage) kind tag.report and semanticmodel kind tags.EcsRunLauncher. (Thanks, @zyd14!)DagsterDbtTranslator.get_code_version to customize the code version for your dbt assets. (Thanks, @Grzyblon!)PipesClientCompletedInvocation. This metadata will be attached to all materializations and asset checks stored during the pipes invocation.AutomationCondition.execution_in_progress which would cause it to evaluate to True for unpartitioned assets that were part of a run that was in progress, even if the asset itself had already been materialized.AutomationCondition.run_in_progress that would cause it to ignore queued runs.default_automation_condition_sensor to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy auto_materialize: use_sensors configuration setting was set to False.0 in cases where all partitions were evaluated./ characters now work correctly with Dagster Pipes.dagster-airlift.types-sqlalchemy package is no longer included in the dagster[pyright] extra package.dagster project scaffold now has an option to create dagster projects from templates with excluded files/filepaths.parse_tableau_external_and_materializable_asset_specs is now available to parse a list of Tableau asset specs into a list of external asset specs and materializable asset specs.DagsterFivetranTranslator to customize assets loaded from Fivetran.dagster_snowflake.fetch_last_updated_timestamps now supports ignoring tables not found in Snowflake instead of raising an error.default_automation_condition_sensor to be constructed for user code servers running on dagster version < 1.9.0 even if the legacy auto_materialize: use_sensors configuration setting was set to False.dagster instance migrate on Dagster version 1.9.0 constructed a SQL query that exceeded the maximum allowed depth.ImportErrors are no longer raised when bigquery libraries are not installed [#25708]dagit.enableReadOnly . When enabled, a separate Dagit instance is deployed in —read-only mode. You can use this feature to serve Dagit to users who you do not want to able to kick off new runs or make other changes to application state.EventMetadata.asset.LOGS_CAPTURED, which explicitly links to the captured stdout/stderr logs for a given step, as determined by the configured ComputeLogManager on the Dagster instance. Previously, these links were available on the STEP_START event.network key on DockerRunLauncher config can now be sourced from an environment variable.get_execution_data method of SensorDefinition and ScheduleDefinition has been renamed to evaluate_tick. We expect few to no users of the previous name, and are renaming to prepare for improved testing support for schedules and sensors.build_sensor_context API. See Testing sensors.dagster.Int ) as type annotations on functions decorated with @solid have been resolved.K8sRunLauncher sometimes hanged while launching a run due to holding a stale Kubernetes client.context.update_cursor(str_value) that is persisted across evaluations to save unnecessary computation. This persisted string value is made available on the context as context.cursor. Previously, we encouraged cursor-like behavior by exposing last_run_key on the sensor context, to keep track of the last time the sensor successfully requested a run. This, however, was not useful for avoiding unnecessary computation when the sensor evaluation did not result in a run request.--read-only mode, which will disable mutations in the user interface and on the server. You can use this feature to run instances of Dagit that are visible to users who you do not want to able to kick off new runs or make other changes to application state.dagster-pandas, the event_metadata_fn parameter to the function create_dagster_pandas_dataframe_type may now return a dictionary of EventMetadata values, keyed by their string labels. This should now be consistent with the parameters accepted by Dagster events, including the TypeCheck event.# old MyDataFrame = create_dagster_pandas_dataframe_type( "MyDataFrame", event_metadata_fn=lambda df: [ EventMetadataEntry.int(len(df), "number of rows"), EventMetadataEntry.int(len(df.columns), "number of columns"), ] ) # new MyDataFrame = create_dagster_pandas_dataframe_type( "MyDataFrame", event_metadata_fn=lambda df: { "number of rows": len(df), "number of columns": len(dataframe.columns), }, )
PandasColumn.datetime_column() now has a new tz parameter, allowing you to constrain the column to a specific timezone (thanks @mrdavidlaing!)DagsterGraphQLClient now takes in an optional transport argument, which may be useful in cases where you need to authenticate your GQL requests:authed_client = DagsterGraphQLClient( "my_dagit_url.com", transport=RequestsHTTPTransport(..., auth=<some auth>), )
ecr_public_resource to get login credentials for the AWS ECR Public Gallery. This is useful if any of your pipelines need to push images.max_completion_wait_time_seconds configuration option, which controls how long it will wait for a Databricks job to complete before exiting.build_solid_context function can be used to provide a context to the invocation.from dagster import build_solid_context @solid def basic_solid(): return "foo" assert basic_solid() == 5 @solid def add_one(x): return x + 1 assert add_one(5) == 6 @solid(required_resource_keys={"foo_resource"}) def solid_reqs_resources(context): return context.resources.foo_resource + "bar" context = build_solid_context(resources={"foo_resource": "foo"}) assert solid_reqs_resources(context) == "foobar"
build_schedule_context allows you to build a ScheduleExecutionContext using a DagsterInstance. This can be used to test schedules.from dagster import build_schedule_context with DagsterInstance.get() as instance: context = build_schedule_context(instance) my_schedule.get_execution_data(context)
build_sensor_context allows you to build a SensorExecutionContext using a DagsterInstance. This can be used to test sensors.from dagster import build_sensor_context with DagsterInstance.get() as instance: context = build_sensor_context(instance) my_sensor.get_execution_data(context)
build_input_context and build_output_context allow you to construct InputContext and OutputContext respectively. This can be used to test IO managers.from dagster import build_input_context, build_output_context io_manager = MyIoManager() io_manager.load_input(build_input_context()) io_manager.handle_output(build_output_context(), val)
build_input_context/build_output_context must be used as a context manager.with build_input_context(resources={"cm_resource": my_cm_resource}) as context: io_manager.load_input(context)
validate_run_config can be used to validate a run config blob against a pipeline definition & mode. If the run config is invalid for the pipeline and mode, this function will throw an error, and if correct, this function will return a dictionary representing the validated run config that Dagster uses during execution.validate_run_config( {"solids": {"a": {"config": {"foo": "bar"}}}}, pipeline_contains_a ) # usage for pipeline that requires config validate_run_config( pipeline_no_required_config ) # usage for pipeline that has no required config
RetryPolicy has been added. This allows you to declare automatic retry behavior when exceptions occur during solid execution. You can set retry_policy on a solid invocation, @solid definition, or @pipeline definition.@solid(retry_policy=RetryPolicy(max_retries=3, delay=5)) def fickle_solid(): # ... @pipeline( # set a default policy for all solids solid_retry_policy=RetryPolicy() ) def my_pipeline(): # will use the pipelines policy by default some_solid() # solid definition takes precedence over pipeline default fickle_solid() # invocation setting takes precedence over definition fickle_solid.with_retry_policy(RetryPolicy(max_retries=2))
dagster/priority tag directly on pipeline definitions would cause an error. This has been fixed.create_dagster_pandas_dataframe_type() function would, in some scenarios, not use the specified materializer argument when provided. This has been fixed (thanks @drewsonne!)dagster-graphql --remote now sends the query and variables as post body data, avoiding uri length limit issues.sqlalchemy.Engine objects would be invalidated after 8 hours of idle time due to MySQL’s default configuration, resulting in an sqlalchemy.exc.OperationalError when attempting to view pages in Dagit in long-running deployments.asOf URL parameter, which shows a snapshot of the asset at the provided timestamp, including parent materializations as of that time.make dev_install has been fixed.reload_repository_location and submit_pipeline_execution have been fixed - the underlying GraphQL queries had a missing inline fragment case.@solid decorator can now wrap a function without a context argument, if no context information is required. For example, you can now do:@solid def basic_solid(): return 5 @solid def solid_with_inputs(x, y): return x + y
however, if your solid requires config or resources, then you will receive an error at definition time.
metadata_entries argument may now instead accept a metadata argument, which should allow for a more convenient API. The metadata argument takes a dictionary with string labels as keys and EventMetadata values. Some base types (str, int, float, and JSON-serializable list/dicts) are also accepted as values and will be automatically coerced to the appropriate EventMetadata value. For example:@solid def old_metadata_entries_solid(df): yield AssetMaterialization( "my_asset", metadata_entries=[ EventMetadataEntry.text("users_table", "table name"), EventMetadataEntry.int(len(df), "row count"), EventMetadataEntry.url("http://mysite/users_table", "data url") ] ) @solid def new_metadata_solid(df): yield AssetMaterialization( "my_asset", metadata={ "table name": "users_table", "row count": len(df), "data url": EventMetadata.url("http://mysite/users_table") } )
--heartbeat-tolerance argument that allows you to configure how long the process can run before shutting itself down due to a hanging thread. This parameter can be used to troubleshoot failures with the daemon process.PartitionSetDefinition.create_schedule_definition, the partition_selector function that determines which partition to use for a given schedule tick can now return a list of partitions or a single partition, allowing you to create schedules that create multiple runs for each schedule tick.KeyError.Dict and Set types for solid inputs/outputs now works as expected. Previously a structure like Dict[str, Dict[str, Dict[str, SomeClass]]] could result in confusing errors.solid_config.map and collect steps downstream of other map and collect steps to mysteriously not execute when using multiprocess executors has been resolved.solid_exception on HookContext which returns the actual exception object thrown in a failed solid. See the example “Accessing failure information in a failure hook“ for more details.solid_output_values on HookContext which returns the computed output values.make_values_resource helper for defining a resource that passes in user-defined values. This is useful when you want multiple solids to share values. See the example for more details.--path-prefix, our color-coded favicons denoting the success or failure of a run were not loading properly. This has been fixed.