Dagster provides several different ways to execute jobs.
|A method to execute a job synchronously, typically for running scripts or testing.|
You can also launch jobs in other ways:
from dagster import job, op @op def return_one(): return 1 @op def add_two(i: int): return i + 2 @op def multi_three(i: int): return i * 3 @job def my_job(): multi_three(add_two(return_one()))
Dagster comes with a web-based interface for viewing and interacting with jobs and other Dagster objects.
To view your job in Dagit, you can use the
dagit -f my_job.py
Then navigate to http://localhost:3000 to start using Dagit:
Click on the "Launchpad" tab, then press the "Launch Run" button to execute the job. You will then see Dagit launch a job run:
By default, Dagit will run the job using the multiprocess executor - that means each step in the job runs in its own process, and steps that don't depend on each other can run in parallel.
Dagit Launchpad also offers a configuration editor to let you interactively build up the configuration. See details in Dagit.
To execute your job directly, you can simply run:
dagster job execute -f my_job.py
Dagster includes Python APIs for execution that are useful when writing tests or scripts.
JobDefinition.execute_in_processexecutes a job and returns an
if __name__ == "__main__": result = my_job.execute_in_process()
Dagster supports ways to run a subset of a job, called Op Selection.
To specify op selection, Dagster supports a simple query syntax.
It works as follows:
*, in which case that op and all of its ancestors (upstream dependencies) are selected.
*, in which case that op and all of its descendents (downstream dependencies) are selected.
+s, in which case that op and descendents up to that many hops away are selected.
+s, in which case that op and ancestors up to that many hops away are selected.
some_op: select "some_op" itself
*some_op: select "some_op" and all ancestors (upstream dependencies).
some_op*: select "some_op" and all descendants (downstream dependencies).
*some_op*: select "some_op" and all of its ancestors and descendants.
+some_op: select "some_op" and its direct parents.
some_op+++: select "some_op" and its children, its children's children, and its children's children's children.
You can use this selection syntax in the
op_selection argument to the
Similarly, you can specify the same op selection in Dagit Launchpad: