https://dagster.io/ logo
Join Slack
Powered by
Hi Dagsterians! I switched to version 1.0.3 from 0...
# ask-community
j
Hi Dagsterians! I switched to version 1.0.3 from 0.13.12 - I had to rewrite my pipelines as jobs (which was quite straightforward). My issue is that now jobs take considerably more time (5x what it took previously in some cases). It looks like main problem is that dagster stalls between ops for a long time where before it ran them sequentially. So it looks like some kind of scheduling issue. When it stalls i can see there are ops in "preparing" section in the ui. I'll attach a few screenshots below.
🤖 1
Old "pipeline" - running all ops sequentially, this was considerably faster than new "jobs"
New "job" - takes a lot of time in "preparing" stage, much slower than old approach
Worst case scenario - stalling for a very long time
I'd appreciate any help
alright, after digging some more I was able to solve the issue - in new dagster default executor is the 
multi_process_executor
that runs each step in separate process, whereas old dagster used 
in_process_executor
I was able to reconfigure it back to 
in_process_executor
, docs are here: https://docs.dagster.io/deployment/executors
s
glad you were able to figure this out. that's exactly the recommendation I would have made
👍 1
4 Views
Open in Slack
PreviousNext
Powered by