Airflow Ingest

You can find the code for this example on Github.

This example demonstrates how to use Airflow ingest to compile an Airflow DAG into a Dagster pipeline that can be executed (and explored) the same way as a Dagster-native pipeline. There are two pipelines in the repo, "airflow_simple_dag" and "airflow_complex_dag". In "airflow_simple_dag", we demonstrate the use of Airflow templates while in "airflow_complex_dag" we show the translation of a more complex dependency structure.

The "execution_date" for the Airflow DAG is specified through the pipeline tags. To specify tags, call to:

    tags={'airflow_execution_date': utc_execution_date_str}
from airflow_ingest.airflow_complex_dag import complex_dag
from airflow_ingest.airflow_simple_dag import simple_dag
from dagster import repository
from dagster_airflow.dagster_pipeline_factory import make_dagster_pipeline_from_airflow_dag

airflow_simple_dag = make_dagster_pipeline_from_airflow_dag(simple_dag)
airflow_complex_dag = make_dagster_pipeline_from_airflow_dag(complex_dag)

def airflow_ingest_example():
    return [airflow_complex_dag, airflow_simple_dag]

Open in a playground

Open in Gitpod


curl | tar -xz --strip=2 dagster-master/examples/airflow_ingest
cd airflow_ingest