Hi, I tried running a job with multiprocess executor having 10 ops which are supposed to start in parallel. When I run it locally all the 10 ops are starting in parallel but when I run the same job on the dagster cloud it is starting only 2 ops in parallel and once the 2 ops is successful then it starts another 2 ops. Could you please help how do I configure the dagster cloud so all the ops starts in parallel?

By default, we limit the concurrency to the number of cores available. To get more concurrency, you should do two things: • launch the run in an isolated environment by selecting the checkbox next to

Launch Run

in the launchpad. Isolated runs have more resources and will be able to support more concurrency • In the launchpad, add the following config:

execution:
  config:
    multiprocess:
      max_concurrent: 10 # or whatever you choose

@johann thanks, I updated the max_concurrent to 50 in deployment settings and checked the isolated environment. Now it is running four ops in parallel. Could you please help me what other settings I have to change so that all the 10 ops starts execution in parallel?

That’s odd, I’m not able to reproduce. Could you link me the run?

When I click

View tags and config

I don’t see the execution config, was it applied to this run?

Yes it was applied.

Oops, I see the misunderstanding. This is a per run setting that you can apply in the launchpad

How do I change the step concurrency?

You can also set run config in code https://docs.dagster.io/concepts/configuration/config-schema

Awesome it worked! 🙌

That’s an out of memory error (we should add custom reporting to clarify what -9 is). This is the risk of increasing concurrency- each step runs in its own process and if they’re doing a lot of processing it’s easy to hit an OOM

What is the memory limits for a step if it's isolated?

https://docs.dagster.io/dagster-cloud/deployment/serverless#isolated-runs 16GB shared across all steps

@johann Can we increase the isolated runs compute resources for a job?

Not currently. Hybrid deployments are always an option, where you can bring whatever compute resources

@johann When can we expect the option of increasing the compute resources for a job? It is important factor for us to adopt dagster cloud serverless for the upcoming projects in our organization?

We have no immediate plans to increase the limits here for isolated runs - hybrid deployment may be a good option if you’re looking to run more resource intensive workloads in dagster cloud