https://dagster.io/ logo
#ask-community
Title
# ask-community
v

Vinnie

01/05/2023, 9:48 AM
My guess is there’s a few resource initializations going on in the background, if I’m not mistaken they always get initialized before the green op bar. How are the startup times when you’re launching ops without resources?
p

Pavel Schudel

01/05/2023, 10:24 AM
there are no resources really. I only use _make_values_resource() for global config. tried to remove it and still there is op startup._ you use it without resources and you don’t have startup time?
z

Zach

01/05/2023, 3:13 PM
this has always been the case in my experience, at least with the MultiProcessExecutor. I'm happy to be corrected but I'm pretty sure that's just the startup penalty for creating a new process + GRPC calls. love Dagster but they don't ever seem to claim it's a low-latency orchestrator, there's just a lot going on behind the scenes (and it's written in Python).
v

Vinnie

01/05/2023, 3:14 PM
Maybe Zach is right, I don’t recall off the top of my mind since none of my processes require super low latency and I don’t think I have anything running that does not require resource initializations.
a

alex

01/05/2023, 3:58 PM
There is a trade off between isolation and init cost, controlled by choosing which executor you use.
I am running k8 executor
This means that each op is run in its own kubernetes container, so the spin up cost is container init and process init. The process init cost is generally just the time spent importing modules. Using the multiprocess executor would remove the container init cost with reduced isolation, with all ops run in the same container as separate processes. Using the
forkserver
option on the multiprocess executor can further reduce process init times after the first, but can cause issues in some libraries. In process executor removes all init times but now all ops execute serially (at this time) in the same process.
❤️ 2
p

Pavel Schudel

01/06/2023, 11:21 AM
when I run Dagit locally for debugging I run the process executor right? I still have a lot of startup time.
a

alex

01/06/2023, 3:33 PM
its multiprocess by default, you can use the config section to switch it to in process / enable forkserver on multiproc
👍 1
2 Views