https://dagster.io/ logo
Title
c

Charlie Bini

01/18/2023, 10:09 PM
is there a k8s_executor configuration where you can set the max concurrent pods? There's runs on the deployment settings, but there may be multiple pods in one run, so a finer-grained control would be nice
d

daniel

01/18/2023, 10:13 PM
we don't have a global limit for this right now unfortunately - you kind of have to approximate with a run limit and then potentially an ops-per-run limit
c

Charlie Bini

01/19/2023, 8:44 PM
thanks @daniel -- if I tag a large group of assets with the same tag, I should be able to add limit their concurrency right?
I'm trying that and I think it's working but hard to see what's going on
d

daniel

01/19/2023, 8:45 PM
this is within a single run?
c

Charlie Bini

01/19/2023, 8:46 PM
across multiple runs
d

daniel

01/19/2023, 8:46 PM
we only support concurrency limits across runs with tags that are applied at the job level
c

Charlie Bini

01/19/2023, 8:46 PM
ok think I get it
so if I tag every asset in question with {"foo": "bar"}, this won't work unless they're in the same run?
run_queue:
  tag_concurrency_limits:
    - key: foo
      value: bar
      limit: 40
d

daniel

01/19/2023, 8:51 PM
those limits won't check tags on assets at all - they'll check tags on jobs
c

Charlie Bini

01/19/2023, 8:51 PM
ok good to know