Anoop Sharma
08/03/2022, 5:19 PMexecution:
config:
multiprocess:
max_concurrent: 2
but where do I put this config. Does it go in dagster.yaml?
PS: I have tried putting it in dagster.yaml but it doesn't seem to be working.owen
08/03/2022, 5:55 PM{"execution": {"config": {"multiprocess": {"max_concurrent": 2}}}}
. Unfortunately, this would need to be applied to each individual job (i.e. @job(config={...})
) -- there's not currently a way to apply the same executor config to all jobs at once.
To be clear, this would set the max concurrent ops for each job to 2, but if there were multiple jobs running at once, then more than 2 ops total would be running at once (max 2 per job)Anoop Sharma
08/03/2022, 6:08 PM