19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Can Dagster help with mutual exclusivity across so...
# ask-communityd
Daniel Carleton
06/15/2021, 4:31 PMCan Dagster help with mutual exclusivity across solids and workflows? My use case is preventing DBT from running concurrently with itself. e.g. Both in response to a regular schedule and to a signal.
r
rex
06/15/2021, 4:34 PMWe have the concept of run concurrency, to limit the number of concurrent pipeline runs of the same tag https://docs.dagster.io/deployment/run-coordinator#limiting-run-concurrency
rex
06/15/2021, 4:38 PMAt the solid level, if you’re using the celery run launcher, you can enforce concurrency limits by tagging the solids with a celery queue you’ve configured. The celery queue can have a limited number of workers (say, 1 worker), so that those solids only execute one at a time
https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#configuring-celery-queues
rex
06/15/2021, 4:38 PMcc @johann if we have something similar in the works for our standalone K8sRunLauncher/Executor
d
Daniel Carleton
06/15/2021, 4:38 PMThese are the same tags set thusly? https://docs.dagster.io/concepts/solids-pipelines/pipelines#solid-tags
r
rex
06/15/2021, 4:39 PMyep
d
Daniel Carleton
06/15/2021, 4:40 PMBrilliant!
Daniel Carleton
06/15/2021, 4:40 PMLooks like
QueuedRunCoordinatorj
johann
06/15/2021, 4:40 PMWithout celery we currently only support concurrency limits per run
d
Daniel Carleton
06/15/2021, 4:40 PMAh hmm, and the granularity of a "run" is at the workflow level?
j
johann
06/15/2021, 4:41 PM^ yep, with the
QueuedRunCoordinatord
Daniel Carleton
06/15/2021, 4:41 PMYes
s/workflow/pipeline✅ 1
Daniel Carleton
06/15/2021, 4:45 PMSeems like for local dev we can just use the in-process executor, and switch to celery for prod where concurrency is a concern. Thanks for the help! Great project -- hoping to adopt it into our infra
2 Views