run_queue where to use

The run queue in Dagster is used to manage the execution of pipeline runs. It is a sequence of Dagster runs waiting to be executed, and Dagster pulls runs from the queue and calls

launch_run

on submitted runs. The run queue operates as a first-in, first-out priority queue unless there are concurrency limits or prioritization rules in place. Here are some scenarios where you might use the run queue: 1. Managing Concurrent Execution: If you want to limit the number of pipeline runs that execute concurrently, you can use the run queue in conjunction with the

max_concurrent_runs

configuration setting. This setting determines the maximum number of runs that are allowed to be in progress at once. 2. Prioritizing Runs: You can define custom prioritization rules for runs in the queue. By default, all runs have a priority of

0

, but you can assign higher priority to certain runs using the

dagster/priority

tag. Runs with higher priority will be launched first. 3. Handling Backfills: When performing backfills, you might have a large number of runs that need to be executed. The run queue can help manage these runs and ensure they are executed in an orderly fashion. 4. Custom Run Coordination: You can implement a custom run coordinator that extends the functionality of the default run queue. For example, you might want to attach additional metadata or tags to runs as they are submitted to the queue. 5. Controlling Run Launching: The run queue can be configured to control how frequently the Dagster Daemon checks the run queue for new runs to launch, using settings like

dequeue_interval_seconds

. To use the run queue, you would typically configure the

QueuedRunCoordinator

in your Dagster instance's

dagster.yaml

file. This configuration allows you to set various parameters such as

max_concurrent_runs

,

tag_concurrency_limits

,

dequeue_interval_seconds

, and others to control the behavior of the run queue. Remember that the run queue is an integral part of managing and orchestrating pipeline runs in Dagster, especially in production environments where you need to control the execution order, concurrency, and prioritization of runs.