Hi Dagsterians! I switched to version 1.0.3 from 0.13.12 - I had to rewrite my pipelines as jobs (which was quite straightforward). My issue is that now jobs take considerably more time (5x what it took previously in some cases). It looks like main problem is that dagster stalls between ops for a long time where before it ran them sequentially. So it looks like some kind of scheduling issue. When it stalls i can see there are ops in "preparing" section in the ui. I'll attach a few screenshots below.
Old "pipeline" - running all ops sequentially, this was considerably faster than new "jobs"
New "job" - takes a lot of time in "preparing" stage, much slower than old approach
Worst case scenario - stalling for a very long time
I'd appreciate any help
alright, after digging some more I was able to solve the issue - in new dagster default executor is the
that runs each step in separate process, whereas old dagster used