https://dagster.io/ logo
#ask-community
Title
# ask-community
c

Chang Hai Bin

05/19/2023, 8:28 AM
Hi everyone, is it possible to set job-level concurrency limit? I know that I can set:
Copy code
execution:
  config:
    multiprocess:
      max_concurrent: 2
so that within one job, only two sub-task/ops run at the same time But, I am more interested in e.g. triggering 1000 job-runs/backfills, but only 2 of those are running, and 998 on queue until one of those 2 completes. I know that there are also Dagster-level job-limit, but I don't want e.g. those 1000 jobs clogging up/preventing other existing jobs
dagster bot answered by content 1
v

Vitaly Markov

05/19/2023, 8:32 AM
c

Chang Hai Bin

05/19/2023, 8:34 AM
Thanks! let me check...
j

jamie

05/19/2023, 4:40 PM
hey @Chang Hai Bin I think tag-based run concurrency limits will solve this for you. A run is just an execution of a job, so you would just add a specific tag to the job you want to limit and configure what that limit should be. Then dagster will ensure that no more than that many runs (ie executions of that job) are happening at once
2 Views