Quick question regarding limiting the # of concurr...
# ask-community
a
Quick question regarding limiting the # of concurrent runs a job can execute: Scenario: (v0.14.19) • I have a Graph in which there is an op "A" that spawns DynamicOutputs to run "B". The way it's written in the Graph is as follows:
Copy code
output_list = OP_A()
output_list.map(OP_B)
• The
output_list
above is of a length N where 0 <= N < +inf • However, I'd only like for OP_B's concurrent runs to be limited to 5 at a time to avoid having N runs running concurrently I noticed there was a config mapping for this purpose so I tried adding the following in the Launchpad before running the graph that was turned into a job using `to_job()`:
Copy code
ops:
  OP_B:
    config:
      execution:
        config:
          multiprocess:
            max_concurrent: 3
            start_method: forkserver
• But when I tried doing so and launching the run of the job, it still tries to run them all at once anyway (this is on local development running
dagit
in my CLI) My question is whether my config is incorrect or whether there is a better strategy to limit number of spawned outputs that run at once?
Here's a screenshot of the scenario. It tries to run all of the spawned dynamicoutputs but due to my resource constraints it fails on some steps:
j
Hi Alex, the run config schema should be:
Copy code
ops:
  OP_B:
    config: {}
execution:
  config:
    multiprocess:
       max_concurrent: 3
       start_method: forkserver
This will apply for all ops, unfortunately there’s not currently a way to limit it per op
One exception is if you leverage celery, it’s possible to send some ops to a different queue than others with a different concurrency limit See: https://docs.dagster.io/deployment/guides/celery#execute-on-celery and https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm-advanced#configuring-celery-queues
a
Legend, thank you @johann. That did the trick!