Troubleshooting queued runs

Below are some possible issues that you can run in to with limiting run concurrency using the QueuedRunCoordinator.

Runs skip the QUEUED status, going straight to STARTED

This likely means that you are not using the QueuedRunCoordinator, which is responsible for run queuing. You can check the run coordinator on your instance in Dagit by going to Status -> Configuration.

Runs are not being dequeued

If your runs stay in the QUEUED status, the issue is likely either that the Dagster Daemon is not deployed on your instance, or that the queue configuration is blocking your runs.

Step 1: Check that the Dagster Daemon is running on your instance

In Dagit, go to Status -> HEALTH, and check that the Run queue is healthy. If not:

Is the Daemon running at all?

Run queuing depends on the Daemon. See the Daemon Overview for how to set it up on your instance.

Is the Daemon accessing the same storages as the Dagit process?

If you have started a Daemon process, you should make sure that the Daemon is accessing the same storages as the Dagit process, meaning both processes should be using the same dagster.yaml. Locally, this means both processes should have the same DAGSTER_HOME environment variable set. See the Dagster Instance docs for more information.

Step 2: Check your queue configuration

If the Daemon is running on your instance, it may be the case that runs are intentionally being left in the queue due to concurrency rules. It may be helpful to look at the logged output from the Daemon process, as this will indicate when runs are skipped.

The run queue is configured in the dagster.yaml under run_coordinator. This can be viewed in Dagit via Status -> Configuration. Runs may be staying in QUEUED due to rules here. The most basic rule that would block the queue would be max_concurrent_runs: 0. In more practical cases, the queue will be blocked by some number of in progress runs. To understand the state of your run queue, it's helpful to view both the 'Queued' and 'In Progress' tabs of the Dagit run page.

Unblocking the queue

If there are runs in progress that are blocking the queue, you have the option of terminating them via Dagit so that other runs can proceed.

Still stuck?

If these steps didn't help, reach out in Slack or file an issue and we'll be happy to help investigate.