https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
r

Rubén Lopez Lozoya

03/31/2022, 5:51 PM
Hey guys, quick question on max concurrent runs: If no value is set, and only tag_concurrency_limits are applied, how will the Daemon decide how many runs should happen at once to prevent resource starvation?
d

daniel

03/31/2022, 5:59 PM
It won't do anything to prevent resource starvation other than the limits that you set
r

Rubén Lopez Lozoya

03/31/2022, 5:59 PM
And I assume that tag_concurrency_limits do not take precedence over max_concurrent_runs right?
d

daniel

03/31/2022, 6:00 PM
It'll enforce both sets of rules, yeah. If you set max_concurrent_runs, there will never be more runs that that at a time, whether or not you set tag concurrency limits
r

Rubén Lopez Lozoya

03/31/2022, 6:01 PM
To give some context: I'd love to have two node pools (one for high load pipelines requiring higher capacity machines) and another for small load ones. For the high load ones, I am going to provide more resources so I want to have higher limits, so I assume to do this I will need to get rid of max_concurrent_runs and perhaps have a high_load and low_load tag to handle it
and then use tags to handle node affinity on a per pipeline basis
j

johann

03/31/2022, 6:14 PM
That sounds right to me!
2 Views