Hi dagster team, I am running into following error...
# announcements
Hi dagster team, I am running into following error when I tried to increase concurrency to 2 so that two solids can run in parallel. I did backscroll search but found that this is unlikely to happen with dagit. Any clue why this can happen? blob thinking eyes • dagit: 0.8.2 • config:
Copy code
"execution": {
        "multiprocess": {"config": {"max_concurrent": 4}}
• Error
Copy code
{"error": ["dagster.core.errors.DagsterUnmetExecutorRequirementsError: You have attempted to use an executor that uses multiple processes with the pipeline \"abc_pipeline\" that is not reconstructable. Pipelines must be loaded in a way that allows dagster to reconstruct them in a new process.
How are you doing this pipeline execution, in python directly? There is a utility function
that can help create a
from a direct pipeline definition reference
I am using pyspark step launcher to invoke the pipeline.
ahhh I see that one - that I believe is just a bug that has since been fixed
so that means we cannot launch multiple solids in parallel with dagit as of now?
you should be able to update to
without breaking changes
if you remain on
you will hit this bug in the pyspark step launcher which prevents you from setting a multiprocess executor so highly recommend updating your dagster version
oh nice. good to know. thanks for jumping in to help.
no problem, let us know how the update process goes
👍 1