Op graphs
Assets vs ops
If you are just getting started with Dagster, we strongly recommend you use assets rather than ops to build your data pipelines. The ops documentation is for Dagster users who need to manage existing ops, or who have complex use cases.
A graph is a set of interconnected ops or sub-graphs. While individual ops typically perform simple tasks, ops can be assembled into a graph to accomplish complex tasks.
Graphs can be used in three different ways:
- To back assets - Basic assets are computed using a single op, but if computing one of your assets requires multiple discrete steps, you can compute it using a graph instead.
- Directly inside a job - Each op job contains a graph.
- Inside other graphs - You can build complex graphs out of simpler graphs.
Relevant APIs
Name | Description |
---|---|
@dg.graph | The decorator used to define an op graph, which can form the basis for multiple jobs. |
GraphDefinition | An op graph definition, which is a set of ops (or sub-graphs) wired together. Forms the core of a job. Typically constructed using the @dg.job decorator. |
Creating op graphs
Using the @graph
decorator
To create an op graph, use the @dg.graph
decorator.
In the following example, we return one output from the root op (return_one
) and pass data along through single inputs and outputs:
from dagster import OpExecutionContext, graph, op
@op
def return_one(context: OpExecutionContext) -> int:
return 1
@op
def add_one(context: OpExecutionContext, number: int) -> int:
return number + 1
@graph
def linear():
add_one(add_one(add_one(return_one())))