Hi dagster team, I am running into following error when I tried to increase concurrency to 2 so that two solids can run in parallel. I did backscroll search but found that this is unlikely to happen with dagit. Any clue why this can happen? blob thinking eyes • dagit: 0.8.2 • config:

"execution": {
        "multiprocess": {"config": {"max_concurrent": 4}}
    }

• Error

{"error": ["dagster.core.errors.DagsterUnmetExecutorRequirementsError: You have attempted to use an executor that uses multiple processes with the pipeline \"abc_pipeline\" that is not reconstructable. Pipelines must be loaded in a way that allows dagster to reconstruct them in a new process.

How are you doing this pipeline execution, in python directly? There is a utility function

reconstructable

that can help create a

ReconstructablePipeline

from a direct pipeline definition reference

I am using pyspark step launcher to invoke the pipeline.

ahhh I see that one - that I believe is just a bug that has since been fixed

so that means we cannot launch multiple solids in parallel with dagit as of now?

you should be able to update to

0.8.11

without breaking changes

oh nice. good to know. thanks for jumping in to help.

no problem, let us know how the update process goes