Jobs are the main unit of execution and monitoring in Dagster. They allow you to execute a portion of a graph of Software-defined Assets or ops based on a schedule or an external trigger.
When a job begins, it kicks off a run. A run is a single execution of a job in Dagster. Runs can be launched and viewed in the Dagster UI.
Using jobs provides the following benefits:
- Automation: With schedules and sensors, jobs can be used to automate the execution of your Dagster pipelines. Refer to the Automating your data pipelines guide for more info and examples.
- Control job run priority: If using a run queue, you can apply custom prioritization rules to how job runs are prioritized and executed.
- Potential for improved efficency: By applying concurrency limits to job runs, there may be benefits to your pipeline's efficiency. Refer to the Limiting run concurrency guide for more info and examples.
Jobs are supported for both Software-defined Assets and ops, but the usage for each concept is unique. Refer to the following documentation for more info:
With jobs, you can:
- Automate the execution of Software-defined Assets and ops
- Materialize a selection of assets based on a schedule or external trigger (sensor)
- Apply metadata about jobs and runs using tags
- View and launch runs of jobs in the Dagster UI