https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
m

Mark Fickett

06/27/2022, 8:47 PM
For
start_method: forkserver
would it make sense to always preload the code location? For example if I run
dagit -m orchestration.main
=> always have
orchestration.main
in
preload_modules
. From advice in https://dagster.slack.com/archives/C01U954MEER/p1647575140840359 I played around with preloading different modules, and I get a big effect (1 minute => 23s per job run) by preloading my main module. Seems logical, since (a) it's always used, and (b) it indirectly imports all the other modules.
c

chris

06/27/2022, 10:02 PM
that's a pretty drastic performance increase... made an issue to track. https://github.com/dagster-io/dagster/issues/8638. The person who has been mainly working on our multiprocessing forkserver implementation is on PTO right now, so might not get addressed right away, but I suspect that it might have been just an oversight in the original implementation.
ty thankyou 1
13 Views