We are using dagster for a while now and are very satisfied 🙂 We have jobs with dozens of operators, some of these jobs takes 5 hours to complete. We would like to enjoy the parallelizable nature of the operators DAG, but cannot, since some of these jobs may never run in parallel (say, due to communication with a dedicated hardware), so we limit the entire job to execute a single operator each time. Is there a way to limit certain operators from running in parallel, while allowing others to do so? Many thanks!

Look into custom RunCoordinators! Particularly QueuedRunCoordinators can do exactly that

Hi @Eegan K, thank you for you reply! I see in the documentation that RunCoordinators can limit job concurrency. Can we use it to also limit operator concurrency?

Yup! just add a tags field to your @op operator

We will give it a shot! Many thanks!

no problemo

Hi boaz - we don't currently have op-level concurrency using tags unfortunately unless you're using celery, although we'd like to support a native concurrency limit like eegan described in the future

This type of concurrency limiting has been working great for me

That's somewhat surprising to me since as far as I know no such feature exists :) The queuedruncoordinator only considers tags on a job, not on an op

Wish I could send you my code to show you but it does work!

Maybe we’re talking about slightly different things - do you have an example of a tag that you’re using to limit concurrency?

the tag itself is irrelevant

Got it - I'm nearly certain that that will only have any effect if you put it on the @job rather than the @op

my job definitions have no tags