19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
announcements
dagster-airbyte
dagster-airflow
dagster-bigquery
dagster-cloud
dagster-cube
dagster-dask
dagster-dbt
dagster-de
dagster-ecs
dagster-feedback
dagster-kubernetes
dagster-machine-learning
dagster-noteable
dagster-releases
dagster-serverless
dagster-showcase
dagster-snowflake
dagster-spatial
dagster-support
dagster-wandb
dagstereo
data-platform-design
events
faq-read-me-before-posting
gigs-freelance
github-discussions
introductions
jobs
random
tools
豆瓣酱帮
Title
a
Arun Kumar
02/11/2022, 7:33 PMHi team, is there any way we can set concurrency control on the number of pods that can be spawned at a time by
k8s_executord
daniel
02/11/2022, 8:18 PMcc @johann who will know for sure, but I don't believe this is currently available functionality on the k8s_job_executor
Using the celery_k8s_job_executor would give you a way to enable op limits though (but you would need to run celery queues in your cluster for that)
j
johann
02/11/2022, 8:27 PMArun, you're correct that the tag concurrency is only at the per run level. Are you looking to limit the global number of concurrent ops (in which case Dan’s suggestion is the current solution) or would the limit be how many pods get launched for each individual run?
a
Arun Kumar
02/11/2022, 8:30 PMWe wanted to limit to number of pods that can be spawned from a single repository (irrespective of whether it comes from k8sRunLauncher or K8sExecutor). But the current tag based control does not limit the K8sExecutor pods.
Do we have some thing that can limit the number of pods per individual run? I think this would also be good for us
j
johann
02/11/2022, 8:33 PMGot it. Limiting per individual run is not currently supported but would be easy to add. The global limit is something we’ve discussed but is definitely a longer term project.
Would you be able to file issues for both requests? I don't think there are existing issues for these exact requirements
a
Arun Kumar
02/13/2022, 12:03 PM