Alex Kan

02/17/2023, 7:52 PM
Do tag concurrency limits for QueuedRunCoordinator only apply at the
granularity? Or can they be applied at the
granularity too Say I have a simple job (op A >> op B), and I have max_concurrent_runs for the run_coordinator set to 10. Could I make it such that task B has a tag that separately limits its concurrency to 1? My use case here is to prevent spamming of the API that powers task B while letting task A run freely. And I want to see if that’s possible without refactoring things to put op B in a separate job.


02/17/2023, 9:36 PM
hi! unfortunately, those tags apply only at the job granularity. The multiprocess_executor has similar tag_concurrency_limits configuration, which will let you set tag based limits within a single job, but they don't work across separate jobs, so I don't think that would work for this situation
:dagsgiving-2: 1

Alex Kan

02/17/2023, 10:31 PM
Ah no worries! Thanks for the reply again Owen, refactoring the job shouldn’t be too difficult