We love to see our community members get involved! If you are planning to contribute to Dagster, you will first need to set up a local development environment.
Install Python 3.8 or above.
Create and activate a virtualenv, using the tool of your choice. On macOS you can install
pyenv with Homebrew:
brew install pyenv pyenv-virtualenv
Then add the following commands to your shell profile:
eval "$(pyenv init -)" eval "$(pyenv virtualenv-init -)"
and finally create and activate the virtualenev:
pyenv install 3.9.16 pyenv virtualenv 3.9.16 dagster39 pyenv activate dagster39
Ensure that you have node installed by running
node -v, and that you have yarn installed. If you are on macOS, you can install yarn with Homebrew:
brew install yarn
Clone the Dagster repository to the destination of your choice:
git clone email@example.com:dagster-io/dagster.git
make dev_install at the root of the repository. This sets up a full Dagster developer environment with all modules and runs tests that do not require heavy external dependencies such as docker. This will take a few minutes. Note that certain sections of the makefile (sanity_check, which is part of
rebuild_ui) require POSIX compliant shells and will fail on CMD and powershell -- if developing on windows, using something like WSL or git-bash is recommended. Note also that if this command fails while installing python packages, the problem might be resolved by ensuring you are running an up-to-date version of
pip (upgrade with
pip install -U pip).
Note for Macs with an M1 or M2 chip: Some users have reported installation problems due to missing wheels for arm64 Macs when installing the
grpcio package. To install the
dagster development environment using our pre-built wheel of the
grpcio package for M1 and M2 machines, run
make dev_install_m1_grpcio_wheel instead of
Run some tests manually to make sure things are working:
python -m pytest python_modules/dagster/dagster_tests
Some notes on developing in Dagster:
make rufffrom the repo root to format, sort imports, and autofix some lint errors. It will also print out errors that need to be manually fixed.
make pyrightfrom the repo root to analyze the whole repo for type-correctness. Note that the first time you run this, it will take several minutes because a new virtualenv will be constructed.
make ruffin the root Dagster directory before submitting a pull request. If you're also using VS Code, you can see what we're using for our
For development, run an instance of the webserver providing GraphQL service on a different port than the webapp, with any pipeline. For example:
cd dagster/examples/docs_snippets/docs_snippets/intro_tutorial/basics/connecting_solids/ dagster-webserver -p 3333 -f complex_pipeline.py
Keep this running. Then, in another terminal, run the local development (autoreloading, etc.) version of the webapp:
cd dagster/js_modules/dagster-ui make dev_webapp
During development, you might find these commands useful. Run them from
yarn ts: Typescript typechecking
yarn lint: Linting with autofix
yarn jest: An interactive Jest test runner that runs only affected tests by default
To run all of them together, run
The docs style checklist includes a set of style guidelines to adhere to when adding or modifying docs.
To run the Dagster documentation website locally, run the following commands:
cd docs make next-watch-build # Serves the docs website on http://localhost:3001
Troubleshooting tip: You may need to run
make next-dev-install first to install dependencies. Also make sure that your Node version is >=12.13.0.
The API documentation is generated from ReStructured Text files (
.rst), which extracts Python docstrings from the library files. The
.rst files can be found in the
If you change any
.rst files, be sure to run the following command in the
The majority of our code snippets are pulled from real Python files. This allows us to test our code snippets and ensure they remain up-to-date.
.mdx files, you'll see a code snippet with
python file=/path/to/file.py startafter=start_marker endbefore=end_marker at the beginning of the block. For example:
You can find the corresponding Python file at
dagster/examples/docs_snippets/docs_snippets/concepts/asset/asset_dependency.py. The code included in each snippet is the code in the file between the
# start_marker and
# end_marker comments.
To change the code snippet, update the
.py file, then run the following from the
You can find more information about developing documentation in
When you are ready for more of a challenge, you can tackle issues with the most 👍 reactions. We factor engagement into prioritization of the issues. You can also explore other labels and pick any issue based on your interest.
In the PR template, please describe the change, including the motivation/context, test coverage, and any other relevant information. Please note if the PR is a breaking change or if it is related to an open GitHub issue.
A Core reviewer will review your PR in around one business day and provide feedback on any changes it requires to be approved. Once approved and all the tests (including Buildkite!) pass, the reviewer will click the Squash and merge button in Github 🥳.
Your PR is now merged into Dagster! We’ll shout out your contribution in the weekly release notes.