https://dagster.io/ logo
#ask-community
Title
# ask-community
k

Kobroli

03/31/2022, 4:09 PM
Hello again 🙂 Testing the fixed fan-in example, I noticed significant overhead for the individual
return_one
ops (in addition to the subprocess launch overhead depicted in blue), some instantiations can take up to more than 2 seconds. Is there a way on the user's end to reduce that overhead? What's the cause of this? Is this reason to believe that, for longer-running tasks, the overhead increases as a function of task duration?
On a different (less performant, but still decent) machine, one op takes even 6 sec
o

owen

03/31/2022, 4:38 PM
hi @Kobroli -- generally, the only overhead for a given step is the subprocess launch (highlighted in blue, as you note). I'm guessing this delay is more due to the level of parallelism in this particular example. Each step does end up needing to write to the same database, so I could imagine things getting hung up there. I can't imagine this delay scaling with the task duration, and I'd guess if you tossed a time.sleep(3) in there, then the steps would take much closer to 3 seconds
👍 1
2 Views