Airflow Ingest

You can find the code for this example on Github.

This example demonstrates how to use Airflow ingest to compile an Airflow DAG into a Dagster pipeline that can be executed (and explored) the same way as a Dagster-native pipeline. There are two pipelines in the repo, "airflow_simple_dag" and "airflow_complex_dag". In "airflow_simple_dag", we demonstrate the use of Airflow templates while in "airflow_complex_dag" we show the translation of a more complex dependency structure.

The "execution_date" for the Airflow DAG is specified through the pipeline tags. To specify tags, call to:

make_dagster_pipeline_from_airflow_dag(
    dag=dag,
    tags={'airflow_execution_date': utc_execution_date_str}
)
repo.py
from airflow_ingest.airflow_complex_dag import complex_dag
from airflow_ingest.airflow_simple_dag import simple_dag
from dagster_airflow.dagster_pipeline_factory import make_dagster_pipeline_from_airflow_dag

from dagster import repository

airflow_simple_dag = make_dagster_pipeline_from_airflow_dag(simple_dag)
airflow_complex_dag = make_dagster_pipeline_from_airflow_dag(complex_dag)


@repository
def airflow_ingest_example():
    return [airflow_complex_dag, airflow_simple_dag]

Open in a playground

Open in Gitpod

Download

curl https://codeload.github.com/dagster-io/dagster/tar.gz/master | tar -xz --strip=2 dagster-master/examples/airflow_ingest
cd airflow_ingest