https://dagster.io/ logo
#ask-community
Title
# ask-community
b

Boaz Menuhin

09/06/2022, 12:28 PM
We are using dagster for a while now and are very satisfied 🙂 We have jobs with dozens of operators, some of these jobs takes 5 hours to complete. We would like to enjoy the parallelizable nature of the operators DAG, but cannot, since some of these jobs may never run in parallel (say, due to communication with a dedicated hardware), so we limit the entire job to execute a single operator each time. Is there a way to limit certain operators from running in parallel, while allowing others to do so? Many thanks!
e

Eegan K

09/06/2022, 12:52 PM
Look into custom RunCoordinators! Particularly QueuedRunCoordinators can do exactly that
b

Boaz Menuhin

09/06/2022, 1:03 PM
Hi @Eegan K, thank you for you reply! I see in the documentation that RunCoordinators can limit job concurrency. Can we use it to also limit operator concurrency?
e

Eegan K

09/06/2022, 1:05 PM
Yup! just add a tags field to your @op operator
🤯 1
b

Boaz Menuhin

09/06/2022, 1:06 PM
We will give it a shot! Many thanks!
e

Eegan K

09/06/2022, 1:06 PM
no problemo
d

daniel

09/06/2022, 1:16 PM
Hi boaz - we don't currently have op-level concurrency using tags unfortunately unless you're using celery, although we'd like to support a native concurrency limit like eegan described in the future
e

Eegan K

09/06/2022, 1:17 PM
This type of concurrency limiting has been working great for me
I don't think that I’ve specified celery but maybe I have?
d

daniel

09/06/2022, 1:18 PM
That's somewhat surprising to me since as far as I know no such feature exists :) The queuedruncoordinator only considers tags on a job, not on an op
e

Eegan K

09/06/2022, 1:19 PM
Wish I could send you my code to show you but it does work!
d

daniel

09/06/2022, 1:23 PM
Maybe we’re talking about slightly different things - do you have an example of a tag that you’re using to limit concurrency?
e

Eegan K

09/06/2022, 1:49 PM
the tag itself is irrelevant
my op definition looks like @op(tags={“tag”:”tag”})
and then I just limit that tag in the queuedRunCoordinator
d

daniel

09/06/2022, 1:50 PM
Got it - I'm nearly certain that that will only have any effect if you put it on the @job rather than the @op
e

Eegan K

09/06/2022, 1:51 PM
my job definitions have no tags
and I am certain that these are limited
3 Views