The easiest way to start building a Dagster project is by using the
dagster project CLI. This CLI tool helps generate files and folder structures that enable you to quickly get started with Dagster.
If you don't already have Dagster installed, verify you meet the installation requirements before continuing.
You can scaffold a new project using the default project skeleton, or start with one of the official Dagster examples.
To learn more about the default files in a Dagster project, refer to the Dagster project file reference.
To get started, run:
pip install dagster dagster project scaffold --name my-dagster-project
dagster project scaffold command generates a folder structure with a single Dagster code location and other files, such as
setup.py. This takes care of setting things up with an empty project, enabling you to quickly get started.
The newly generated
my-dagster-project directory is a fully functioning Python package and can be installed with
To install it as a package and its Python dependencies, run:
pip install -e ".[dev]"
-e) flag instructs
pip to install your code location as a Python package in "editable mode" so that as you develop, local code changes are automatically applied.
To start the Dagster UI, run:
Use your browser to open http://localhost:3000 to view the project.
You can specify new Python dependencies in
Environment variables, which are key-value pairs configured outside your source code, allow you to dynamically modify application behavior depending on environment.
Using environment variables, you can define various configuration options for your Dagster application and securely set up secrets. For example, instead of hard-coding database credentials - which is bad practice and cumbersome for development - you can use environment variables to supply user details. This allows you to parameterize your pipeline without modifying code or insecurely storing sensitive data.
Refer to the Using environment variables and secrets in Dagster code guide for more info and examples.
Tests can be added in the
my_dagster_project_tests directory and run using
Once your project is ready to move to production, check out our recommendations for transitioning data pipelines from development to production.
Check out the following resources to learn more about deployment options: