https://dagster.io/ logo
#ask-ai
Title
# ask-ai
a

Andreas

03/13/2024, 2:47 PM
QueuedRunCoordinator
s

Scout

03/13/2024, 2:47 PM
The QueuedRunCoordinator in Dagster is a system component used to manage the set of active runs on your deployment. It controls the policy that Dagster uses to manage these runs, including prioritization and concurrency limits. Here are some key points about the QueuedRunCoordinator based on the documentation: 1. Run Queue: The QueuedRunCoordinator operates a run queue, which is a sequence of Dagster runs waiting to be executed. It pulls runs from the queue and calls
launch_run
on submitted runs, operating as a first-in, first-out priority queue unless other rules or limits are applied. 2. Concurrency Limits: You can configure concurrency limits using the
max_concurrent_runs
parameter, which defines the maximum number of runs that are allowed to be in progress at once. By default, this is set to 10, but you can adjust it according to your needs or disable it by setting it to -1. 3. Tag Concurrency Limits: The QueuedRunCoordinator allows you to apply limits to runs with particular tags. This can be used to control concurrency based on specific criteria, such as limiting the number of runs for a particular pipeline or user. 4. Dequeue Interval: The
dequeue_interval_seconds
parameter controls the interval at which the Dagster Daemon should periodically check the run queue for new runs to launch. 5. Dequeue Workers: If you enable threading for dequeuing runs with
dequeue_use_threads
, you can also limit the number of concurrent worker threads with the
dequeue_num_workers
parameter. 6. Priority Rules: Runs can be assigned priorities using the
dagster/priority
tag. Higher priority runs are launched first, and if multiple runs have the same priority, they are launched in the order they were submitted. 7. Custom Run Coordinator: You can create a custom run coordinator by subclassing the QueuedRunCoordinator. For example, you might want to read HTTP headers and attach them as tags to runs, which can be done by overriding the
submit_run
method in your custom class. 8. Error Handling: The QueuedRunCoordinator has parameters like
max_user_code_failure_retries
and
user_code_failure_retry_delay
to handle errors when reaching a Dagster gRPC server while dequeuing the run. 9. Scheduling: The QueuedRunCoordinator works with the Dagster Daemon, which is responsible for scheduling and executing runs according to the defined policies. For more detailed information on configuring and using the QueuedRunCoordinator, you can refer to the Dagster documentation, which provides examples and explanations of the various configuration options available.
2 Views