Hi Dagsterians I switched to version 1 0 3 from 0 13 12 I ha

Question

Hi Dagsterians I switched to version 1 0 3 from 0 13 12 I had to rewrite my pipelines as jobs which was quite straightforward My issue is that now jobs take considerably more time 5x what it took previously in some cases It looks like main problem is that dagster stalls between ops for a long time where before it ran them sequentially So it looks like some kind of scheduling issue When it stalls i can see there are ops in preparing section in the ui I ll attach a few screenshots below

Jacek Chmielewski · Answer

Old pipeline running all ops sequentially this was considerably faster than new jobs

Jacek Chmielewski · Answer

New job takes a lot of time in preparing stage much slower than old approach

Jacek Chmielewski · Answer

Worst case scenario stalling for a very long time

Jacek Chmielewski · Answer

I d appreciate any help

Jacek Chmielewski · Answer

alright after digging some more I was able to solve the issue in new dagster default executor is the `multi process executor` that runs each step in separate process whereas old dagster used `in process executor`

Jacek Chmielewski · Answer

I was able to reconfigure it back to `in process executor` docs are here <https docs dagster io deployment executors>

sandy · Answer

glad you were able to figure this out that s exactly the recommendation I would have made