Declarative Automation#

Dagster automatically tracks a large amount of information about events that impact the status of your assets, and the dependencies between them. Declarative Automation is a framework that lets you access this information to make intelligent decisions.

Pre-built conditions are provided to handle common use cases, such as executing on a periodic schedule or whenever an upstream dependency updates, but conditions can be customized in a fine-grained manner, allowing precise control over when work gets executed.

Benefits#

Using Declarative Automation helps you:

Ensure you're working with the most up-to-date data
Optimize resource usage by only materializing assets or executing checks when needed
Precisely define when specific assets should be updated based on the state of other assets

Prerequisites#

Before continuing, you should be familiar with:

How it works#

Declarative Automation has two main components:

An automation condition (AutomationCondition), which represents when an individual asset or check should be executed.
A sensor (AutomationConditionSensorDefinition), which evaluates each AutomationCondition and launches runs in response to their status.

Automation conditions#

Automation conditions describe the conditions under which work should be executed. Dagster provides three pre-built conditions:

Name	Description	Useful for
AutomationCondition.on_cron(cron_schedule)	This condition will materialize an asset on a provided cron schedule, after all of its parents have been updated	Regularly updating an asset without worrying about the specifics of how its parents update
AutomationCondition.on_missing()	This condition will materialize an asset if all its dependencies have been updated, but the asset itself has not.	Filling in partitioned assets as soon as upstream data is available.
AutomationCondition.eager()	This condition will materialize an asset: If the asset has never been materialized before, or When the asset's parents update, as long as none of the parents are currently missing or have an update in progress	Automatically propagating changes through the asset graph Ensuring assets remain up-to-date

With assets, automation conditions can be set on the @asset decorator or on an AssetSpec:


import dagster as dg

@dg.asset(automation_condition=dg.AutomationCondition.eager())
def my_eager_asset(): ...

AssetSpec("my_cron_asset", automation_condition=AutomationCondition.on_cron("@daily"))

The same is true for asset checks:


import dagster as dg

@dg.asset_check(asset=dg.AssetKey("orders"), automation_condition=dg.AutomationCondition.on_cron("@daily"))
def my_eager_check() -> dg.AssetCheckResult:
    return dg.AssetCheckResult(passed=True)


AssetCheckSpec(
    "my_cron_check",
    asset=dg.AssetKey("orders"),
    automation_condition=AutomationCondition.on_cron("@daily"),
)

The core AutomationCondition framework is extensible, allowing you to build conditions that fit specific needs. Refer to the Customizing automation conditions guide for more information.

Sensors#

When automation conditions for an asset or check are met, a sensor will execute a run in response. This sensor, named default_automation_condition_sensor, will be available for each code location and monitor all assets within that location. To use multiple sensors or change the properties of the default sensor, refer to the AutomationConditionSensorDefinition documentation.

For an automation condition sensor to run, it must be turned on and an active dagster-daemon process must be running. If you used dagster dev to start the Dagster UI/webserver, the daemon process will be automatically launched alongside the webserver.

After these criteria are met, the sensor's evaluation history will be visible in the UI:

Default automation sensor evaluations in the Dagster UI

You'll also be able to view a detailed history of each asset's evaluations on the asset's Asset Details page. This allows you to see why an asset was or wasn't materialized at different points in time:

Automation condition evaluations in the Asset Details page

Getting started#

To use Declarative Automation, you'll need to enable the automation condition sensor in the Dagster UI:

Navigate to Overview > Sensors
Locate the desired code location.
In the code location, toggle the default_automation_condition_sensor sensor to on.

From here, you can:

Define custom automation conditions
View a history of each evaluation for the sensor
Navigate to individual assets to see a history of their evaluations