Note: While this specific example uses a project created by scaffolding, projects created using an official example will also contain these files. In official examples, assets.py will be a subdirectory instead of a Python file.
Let's take a look at what each of these files and directories does:
A Python module that contains Dagster code.
This directory also contains the following:
The __init__.py file includes a Definitions object that contains all the definitions defined within your project. A definition can be an asset, a job, a schedule, a sensor, or a resource. This allows Dagster to load the definitions in a module.
Note: As your project grows, we recommend organizing assets in sub-modules. For example, you can put all analytics-related assets in a my_dagster_project/assets/analytics/folder and use load_assets_from_package_module in the top-level definitions to load them, rather than needing to manually add assets to the top-level definitions every time you define one.
A Python module that contains tests for my_dagster_project.
A description and starter guide for the Dagster project.
A file that specifies package core metadata in a static, tool-agnostic way.
This file includes a tool.dagster section which references the Python module with your Dagster definitions defined and discoverable at the top level. This allows you to use thedagster dev command to load your Dagster code without any parameters. Refer to the Code locations documentation to learn more.
Note:pyproject.toml was introduced in PEP-518 and meant to replace setup.py, but we may still include a setup.py for compatibility with tools that do not use this spec.
A build script with Python package dependencies for your new project as a package. Use this file to specify dependencies.
Note: If using Dagster Cloud, add dagster-cloud as a dependency.
An ini file that contains option defaults for setup.py commands.
Depending on your use case or if you're using Dagster Cloud, you may also need to add additional configuration files to your project. Refer to the Example project structures section for a look at how these files might fit into your projects.
Configures your Dagster instance, including defining storage locations, run launchers, sensors, and schedules. Refer to the dagster.yaml reference for more info, including a list of use cases and available options.
Using the default project skeleton, let's take a look at how some example Dagster projects would be structured in different scenarios.
Note: With the exception of dagster_cloud.yaml, it's not necessary for configuration files to be located with your project files. These files typically need to be located in DAGSTER_HOME. For example, in larger deployments, DAGSTER_HOME and Dagster infrastructure configuration can be managed separately from the code locations they support.