Can Dagster help with mutual exclusivity across solids and w

Question

Can Dagster help with mutual exclusivity across solids and workflows My use case is preventing DBT from running concurrently with itself e g Both in response to a regular schedule and to a signal

rex · Answer

We have the concept of run concurrency to limit the number of concurrent pipeline runs of the same tag <https docs dagster io deployment run coordinator limiting run concurrency>

rex · Answer

At the solid level if you re using the celery run launcher you can enforce concurrency limits by tagging the solids with a celery queue you ve configured The celery queue can have a limited number of workers say 1 worker so that those solids only execute one at a time <https docs dagster io deployment guides kubernetes customizing your deployment configuring celery queues>

rex · Answer

cc < johann> if we have something similar in the works for our standalone K8sRunLauncher Executor

Daniel Carleton · Answer

These are the same tags set thusly <https docs dagster io concepts solids pipelines pipelines solid tags>

rex · Answer

yep

Daniel Carleton · Answer

Brilliant

Daniel Carleton · Answer

Looks like `QueuedRunCoordinator` can limit concurrency without needing to resort to celery

johann · Answer

Without celery we currently only support concurrency limits per run

Daniel Carleton · Answer

Ah hmm and the granularity of a run is at the workflow level

johann · Answer

^ yep with the `QueuedRunCoordinator` Yes or we use the term pipeline

Daniel Carleton · Answer

Yes `s workflow pipeline` white check mark

Daniel Carleton · Answer

Seems like for local dev we can just use the in process executor and switch to celery for prod where concurrency is a concern Thanks for the help Great project hoping to adopt it into our infra