Troubleshooting queued runs
Runs skip the
QUEUED status, going straight to
This likely means that you are not using the QueuedRunCoordinator, which is responsible for run queuing. You can check the run coordinator on your instance in Dagit by going to
Runs are not being dequeued¶
If your runs stay in the
QUEUED status, the issue is likely either that the Dagster Daemon is not deployed on your instance, or that the queue configuration is blocking your runs.
Step 1: Check that the Dagster Daemon is running on your instance¶
In Dagit, go to
HEALTH, and check that the
Run queue is healthy. If not:
Is the Daemon running at all?¶
Run queuing depends on the Daemon. See the Daemon Overview for how to set it up on your instance.
Is the Daemon accessing the same storages as the Dagit process?¶
If you have started a Daemon process, you should make sure that the Daemon is accessing the same storages as the Dagit process, meaning both processes should be using the same
dagster.yaml. Locally, this means both processes should have the same
DAGSTER_HOME environment variable set. See the Dagster Instance docs for more information.
Step 2: Check your queue configuration¶
If the Daemon is running on your instance, it may be the case that runs are intentionally being left in the queue due to concurrency rules. It may be helpful to look at the logged output from the Daemon process, as this will indicate when runs are skipped.
The run queue is configured in the
run_coordinator. This can be viewed in Dagit via
Configuration. Runs may be staying in
QUEUED due to rules here. The most basic rule that would block the queue would be
max_concurrent_runs: 0. In more practical cases, the queue will be blocked by some number of in progress runs. To understand the state of your run queue, it's helpful to view both the 'Queued' and 'In Progress' tabs of the Dagit run page.
Unblocking the queue¶
If there are runs in progress that are blocking the queue, you have the option of terminating them via Dagit so that other runs can proceed.
If these steps didn't help, reach out in Slack or file an issue and we'll be happy to help investigate.