Ask AI

Airflow Migration Tutorial#

This tutorial demonstrates using dagster-airlift to migrate an Airflow DAG to Dagster.

Using dagster-airlift we can

  • Observe Airflow DAGs and their execution history with no changes to Airflow code
  • Model and observe assets orchestrated by Airflow with no changes to Airflow code
  • Enable a migration process that
    • Can be done task-by-task in any order with minimal coordination
    • Has task-by-task rollback to reduce risk
    • That retains Airflow DAG structure and execution history during the migration

Process#

This is a high level overview of the steps to migrate an Airflow DAG to Dagster:

  • Peer
    • Observe an Airflow instance from within a Dagster Deployment via the Airflow REST API.
    • This loads every Airflow DAG as an asset definition and creates a sensor that polls Airflow for execution history.
  • Observe
    • Add a mapping that maps the Airflow DAG and task id to a basket of definitions that you want to observe. (e.g. render the full lineage the dbt models an Airflow task orchestrates)
    • The sensor used for peering also polls for task execution history, and adds materializations to an observed asset when its corresponding task successfully executes
  • Migrate
    • Selectively move execution of Airflow tasks to Dagster Software Defined Assets

Pages#