https://dagster.io/ logo
#ask-ai
Title
# ask-ai
b

BC A

03/08/2024, 1:21 PM
@Scout setup max concurrent pod run within each run to 30 using this configuration : runLauncher: type: K8sRunLauncher config: k8sRunLauncher: runK8sConfig: containerConfig: # raw config for the pod's main container (job container) resources: limits: cpu: 2 memory: 8Gi ephemeral-storage: 10Gi
s

Scout

03/08/2024, 1:21 PM
The configuration snippet you've provided sets resource limits for the main container of a job when using the
K8sRunLauncher
. However, it does not directly set a limit on the maximum number of concurrent pod runs within each run. The resource limits specified in the
containerConfig
section (
cpu
,
memory
, and
ephemeral-storage
) are constraints on the resources available to each individual pod, not the concurrency of pod execution. To limit the maximum number of concurrent pod runs within each run, you would need to configure concurrency controls at the job or asset level. The documentation provides information on how to set concurrency limits using
tag_concurrency_limits
, which can be applied to limit the number of ops with certain tags that can execute at once within a single run. Here is an example from the documentation on how to use `tag_concurrency_limits`:
Copy code
yaml run_queue:   tag_concurrency_limits:     - key: "my-concurrency-limiter"       value: "value1"       limit: 30
In this example, the
tag_concurrency_limits
configuration is set in the
dagster.yaml
file, which is the configuration file for the Dagster instance. It specifies that for any op tagged with
my-concurrency-limiter: value1
, at most 30 instances of such ops can run concurrently. To apply this to your scenario, you would need to ensure that the relevant ops or assets in your job are tagged appropriately and that the
dagster.yaml
file includes the
tag_concurrency_limits
configuration with the desired concurrency limit. Please note that the exact implementation details may vary based on your specific use case and the version of Dagster you are using. Always refer to the latest Dagster documentation for the most accurate and up-to-date guidance on configuring concurrency limits.
4 Views