|Base class for run launchers.|
Runs instigated from the Dagit UI, the scheduler, or
dagster pipeline launch are "launched" in Dagster. This is a distinct operation from "executing" a pipeline using the
execute_pipeline python API or the CLI
execute command. A 'launch' operation allocates computational resources (e.g. a process, a container, a kubernetes pod, etc) to carry out a run execution and then instigates the execution.
The core abstraction in the launch process is the run launcher, which is configured as part of the Dagster Instance. The run launcher is the interface to the computational resources that will be used to actually execute Dagster runs. It receives the ID of a created run and a representation of the pipeline that is about to undergo execution.
The simplest run launcher is the built-in run launcher,
DefaultRunLauncher. This run launcher spawns a new process on the same node as the pipeline's repository location. It also provides the ability to terminate launched runs.
Other run launchers include:
K8sRunLauncher, which allocates a Kubernetes Job per run.
For more information, check out Kubernetes Deployment Guides.
DockerRunLauncher, which launches runs in a Docker container.
CeleryK8sRunLauncher, which launches runs as single Kubernetes Jobs.
For more information, check out Deploying Dagster on Helm, Advanced.
A few examples of when a custom run launcher is needed:
Colloquially we refer to the process or computational resource created by the run launcher as the run worker. The run launcher only determines the behavior of the run worker. Once execution starts within the run worker, it is the executor -- an in-memory abstraction in the run worker process -- that takes over management of computational resources.