This section will show you how to a create a new Dagster project and organize your files as you build larger and larger pipelines. Dagster comes with a convenient CLI command for generating a project skeleton, but you can also choose to organize your files differently as your project evolves.
If you're completely new to Dagster, we recommend that you visit our Tutorial to learn all the basic concepts of Dagster.
If you're just starting a new Dagster project, the CLI command
dagster new-project will generate a project skeleton with boilerplate code for development and testing. If you have
dagster installed in your Python environment, then you can run the following shell command to generate a Dagster project called
dagster new-project PROJECT_NAME cd PROJECT_NAME
The newly generated
PROJECT_NAME directory is in fact a fully functioning Python package and can be installed with
pip. This Python package is considered a repository location in Dagster. A single Python file can also be a repository location, like in the Hello World example. For more details about different types of repository locations, visit the Repositories overview.
Here's a breakdown of the files and directories that are generated:
|A Python package that contains code for your new Dagster repository|
|A Python package that contains tests for |
|A file that specifies the location of the user code for Dagit and the Dagster CLI. Visit the Workspaces overview for more details.|
|A description and guide for your new code repository|
|A build script with Python package dependencies for your new code repository|
Inside of the directory
PROJECT_NAME/, the following files and directories are generated:
|A Python package that contains SolidDefinitions, which represent individual units of computation|
|A Python package that contains PipelineDefinitions, which are built up from solids|
|A Python package that contains ScheduleDefinitions, to trigger recurring pipeline runs based on time|
|A Python package that contains SensorDefinitions, to trigger pipeline runs based on external state|
|A Python module that contains a RepositoryDefinition, to specify which pipelines, schedules, and sensors are available in your repository|
This file structure is a good starting point and suitable for most Dagster projects. As you build more and more pipelines, you may eventually find your own way of structuring your code that works best for you.
pipwill install your repository in "editable mode" so that as you develop, local code changes will automatically apply.
pip install --editable .
The Dagit process automatically uses the file
workspace.yaml to find your repository location(s), from which Dagster will load your pipelines, schedules, and sensors. To see how you can customize the Dagit process, run
Once your Dagster Daemon process is running, you should be able to enable schedules and sensors for your Dagster pipelines in your repository.
Once you have created a new Dagster repository with the CLI command
dagster new-project, you can find tests in
PROJECT_NAME is the name of your project. You can run all of your tests with the following command:
As you create Dagster solids and pipelines, add tests in
PROJECT_NAME_tests/ to check that your code behaves as desired and does not break over time.
For hints on how to write tests for solids and pipelines in Dagster, see our documentation tutorial on Testing.
Once your Dagster project is ready, visit the Deployment Guides to learn how to run Dagster in production environments, such as Docker, Kubernetes, AWS EC2, etc.