Dependency DSL

You can find the code for this example on Github.

This demo example is to demonstrate that it is possible to construct the dependencies of a pipeline definition from a YAML file or similar. This is not the "blessed" file format nor is it our recommendation for doing things. However we've been asked a number of times if this is possible and this should serve as a reasonable template to build upon.

For example:

pipeline:
  name: some_example
  description: blah blah blah
  solids:
    - def: add_one
      alias: A
    - def: add_one
      alias: B
      deps:
        num:
          solid: A
    - def: add_two
      alias: C
      deps:
        num:
          solid: A
    - def: subtract
      deps:
        left:
          solid: B
        right:
          solid: C

Open in a playground

Open in Gitpod

Download

curl https://codeload.github.com/dagster-io/dagster/tar.gz/master | tar -xz --strip=3 dagster-master/examples/dep_dsl
cd dep_dsl