Dagster is the data orchestration platform built for productivity.
Get started with Dagster in just three quick steps:
To install Dagster into an existing Python environment, run:
pip install dagster
This installs the latest stable version of the core Dagster packages in your current Python environment.
To get started, we'll define two simple data assets:
cerealsasset that represents a CSV dataset about breakfast cereals, and
nabisco_cerealsasset, which is a downstream dependency of
cerealsand only contains cereals manufactured by Nabisco
In the directory where you installed Dagster, copy this code and save it in a file named
import csv import requests from dagster import asset @asset def cereals(): response = requests.get("https://docs.dagster.io/assets/cereal.csv") lines = response.text.split("\n") return [row for row in csv.DictReader(lines)] @asset def nabisco_cereals(cereals): """Cereals manufactured by Nabisco""" return [row for row in cereals if row["mfr"] == "N"]
Next, you'll materialize the assets. Materialization computes an asset's contents and writes them to persistent storage. By default, this is a pickle file on the local system.
There are a few ways to materialize an asset:
Dagit is a web-based interface for viewing and interacting with Dagster objects.
To install Dagit, run:
pip install dagit
To launch Dagit, run:
dagit -f cereal.py
You should see output similar to:
Serving dagit on http://127.0.0.1:3000 in process 70635
Navigate to http://localhost:3000 in your web browser to view your assets.
Click the Materialize All button to launch a run that materializes the assets:
You can also use the Dagster Python API to materialize the assets as a script.
Add a few lines to
cereal.py, which executes a run within the Python process:
from dagster import materialize if __name__ == "__main__": materialize([cereals, nabisco_cereals])
Now you can run:
Congrats - you just created and materialized your first Dagster assets! Now that you've done that, what's next?
If you get stuck or have any other questions, we'd love to hear from you on Slack: