Skip to main content

Automating jobs

Preview feature

This feature is considered in a preview stage, and is under active development, and not considered ready for production use. You may encounter feature gaps, and the APIs may change. For more information, see the API lifecycle stages documentation.

In addition to individual assets and asset checks, automation conditions can be used to automate the execution of asset jobs. This is useful when a set of assets should always be executed together in a single run, rather than each asset being evaluated and requested independently.

To automate a job, pass an automation_condition to define_asset_job. Job-scoped conditions are built by wrapping an asset-level condition (such as eager(), missing(), or on_cron()) with one of the following:

The job's root assets are the assets in the job's selection that have no parents within the job. The wrapped condition is only evaluated against these root assets; when the job's condition becomes true, a single run of the entire job is launched, executing the root assets and everything downstream of them within the job.

In the following example, whenever raw_data is updated, the eager() condition becomes true for processed_data (the job's only root asset), and a single run of analytics_job is launched that executes both processed_data and report:

src/<project_name>/defs/jobs.py
import dagster as dg


@dg.asset
def raw_data() -> None: ...


@dg.asset(deps=[raw_data])
def processed_data() -> None: ...


@dg.asset(deps=[processed_data])
def report() -> None: ...


analytics_job = dg.define_asset_job(
    "analytics_job",
    selection=[processed_data, report],
    automation_condition=dg.AutomationCondition.all_job_root_assets_match(
        dg.AutomationCondition.eager()
    ),
)

Behavior

  • The wrapped condition is evaluated against the job's root assets only. Assets downstream of the roots within the job are executed as part of the launched run without being evaluated separately.
  • Each time the job's condition becomes true, one run is launched covering the job's full selection.
  • Jobs with automation conditions are evaluated by the default automation condition sensor (default_automation_condition_sensor), which must be enabled in the UI. For more information, see automation condition sensors. Custom automation condition sensors cannot target jobs, so jobs are always handled by the default sensor.

Partitioned jobs​

If the assets in the job are partitioned, the job's condition is evaluated on a per-partition basis, and one run is launched for each partition for which the condition is true. Each run is tagged with its partition key.

The following restrictions apply to partitioned jobs with automation conditions:

  • All assets in the job must share the same partitions definition.
  • A job with an automation condition cannot also have partitioned run config. Providing both an automation_condition and a PartitionedConfig (or a config value on a partitioned job) will raise an error when the job is resolved.
  • Runs requested for a partitioned job are launched individually and are not grouped into backfills.