Contributing

If you are planning to contribute to Dagster, you will first need to set up a local development environment.

Environment Setup

  1. Install Python. Python 3.6 or above recommended, but our CI/CD pipeline currently tests against up-to-date patch versions of Python 2.7, 3.5, 3.6, and 3.7. Note that Dagster is a 2/3 compatible project: tests must pass against Python 2.7.
  2. Create and activate a virtualenv, using the tool of your choice. On macOS you can install pyenv with Homebrew:
brew install pyenv pyenv-virtualenv

       Then add the following commands to your shell profile:

eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

       and finally create and activate the virtualenev:

pyenv install 3.7.4
pyenv virtualenv 3.7.4 dagster37
pyenv activate dagster37
  1. Ensure that you have node installed by running node -v, and that you have yarn installed. If you are on macOS, you can install yarn with Homebrew:
brew install yarn
  1. Clone the Dagster repository to the destination of your choice:
git clone git@github.com:dagster-io/dagster.git
  1. Run make dev_install at the root of the repository. This sets up a full Dagster developer environment with all modules and runs tests that do not require heavy external dependencies such as docker. This will take a few minutes. Note that certain sections of the makefile (sanity_check, which is part of rebuild_dagit) require POSIX compliant shells and will fail on CMD and powershell -- if developing on windows, using something like WSL or git-bash is recommended.
make dev_install
  1. Run some tests manually to make sure things are working:
python -m pytest python_modules/dagster/dagster_tests

Developing Dagster

Some notes on developing in Dagster:

  • Python 2: We plan to continue supporting Python 2 for some time; it is worth testing your changes with Python 2 to ensure compatibility. You can use tox for this, from within any of the Python packages:
PYENV=py27 tox
  • Black/Pylint: We use black to enforce a consistent code style, along with pylint. We test these in our CI/CD pipeline.
  • Line Width: We use a line width of 100.
  • IDE: We recommend setting up your IDE to format with black on save and check pylint, but you can always run make black and make pylint in the root Dagster directory before submitting a pull request. If you're also using VS Code, you can see what we're using for our settings.jsonhere.

Developing Dagit

For development, run the Dagit GraphQL server on a different port than the webapp, with any pipeline. For example:

cd dagster/examples/docs_snippets/docs_snippets/intro_tutorial
dagit -p 3333 -f airflow.py

Keep this running. Then, in another terminal, run the local development (autoreloading, etc.) version of the webapp:

cd dagster/js_modules/dagit
make dev_webapp

To run JavaScript tests for the Dagit frontend, you can run:

cd dagster/js_modules/dagit
yarn test

In webapp development it's handy to run yarn run jest --watch to have an interactive test runner.

Some webapp tests use snapshots--auto-generated results to which the test render tree is compared. Those tests are supposed to break when you change something.

Check that the change is sensible and run yarn run jest -u to update the snapshot to the new result. You can also update snapshots interactively when you are in --watch mode.

Developing Docs

To run the Dagster documentation website locally, run the following commands:

cd docs
yarn --cwd next dev  # Serves the docs website on http://localhost:3001

The API documentation is generated from ReStructured Text files (.rst), which extracts Python docstrings from the library files. The .rst files can be found in the docs/sections/api/apidocs directory.

If you change any .rst files, be sure to run the following command in the docs directory:

make buildnext