Ask AI

Creating a new Dagster project#

The easiest way to start building a Dagster project is by using the dagster project CLI. This CLI tool helps generate files and folder structures that enable you to quickly get started with Dagster.

Step 1: Bootstrap a new project#

If you don't already have Dagster installed, verify you meet the installation requirements before continuing.

You can scaffold a new project using the default project skeleton, or start with one of the official Dagster examples.

To learn more about the default files in a Dagster project, refer to the Dagster project file reference.

Using the default project skeleton#

To get started, run:

pip install dagster
dagster project scaffold --name my-dagster-project

The dagster project scaffold command generates a folder structure with a single Dagster code location and other files, such as pyproject.toml and setup.py. This takes care of setting things up with an empty project, enabling you to quickly get started.

Step 2: Install project dependencies#

The newly generated my-dagster-project directory is a fully functioning Python package and can be installed with pip.

To install it as a package and its Python dependencies, run:

pip install -e ".[dev]"
Using the --editable (-e) flag instructs pip to install your code location as a Python package in "editable mode" so that as you develop, local code changes are automatically applied.

Step 3: Start the Dagster UI#

To start the Dagster UI, run:

dagster dev

Note: This command also starts the Dagster daemon. Refer to the Running Dagster locally guide for more info.

Use your browser to open http://localhost:3000 to view the project.

Step 4: Development#

Adding new Python dependencies#

You can specify new Python dependencies in setup.py.

Using environment variables and secrets#

Environment variables, which are key-value pairs configured outside your source code, allow you to dynamically modify application behavior depending on environment.

Using environment variables, you can define various configuration options for your Dagster application and securely set up secrets. For example, instead of hard-coding database credentials - which is bad practice and cumbersome for development - you can use environment variables to supply user details. This allows you to parameterize your pipeline without modifying code or insecurely storing sensitive data.

Refer to the Using environment variables and secrets in Dagster code guide for more info and examples.

Adding and running unit tests#

Tests can be added in the my_dagster_project_tests directory and run using pytest:

pytest my_dagster_project_tests

Next steps#

Once your project is ready to move to production, check out our recommendations for transitioning data pipelines from development to production.

Check out the following resources to learn more about deployment options:

  • Dagster+ - Deploy using Dagster-managed infrastructure
  • Your own infrastructure - Deploy to your infrastructure, such as Docker, Kubernetes, Amazon Web Services, etc.