https://dagster.io/ logo
#ask-community
Title
# ask-community
g

Gabriel Montañola

08/08/2022, 8:22 PM
Delay between launching subprocess to executing step (2-3min) when using
multiprocess_executor
EKS
K8sRunLauncher
0.15.7
Hi there folks, how you doing? We noticed that there is a delay for each step and this makes complex jobs really slow. In a job we specified
@job(config=config, executor_def=in_process_executor)
and it works just fine. Any tips on how to troubleshoot this?
👍 1
j

johann

08/10/2022, 4:38 PM
Are you able to check cpu/mem usage on the pod? Could be a culprit for delays spinning up a new subprocess
g

Gabriel Montañola

08/17/2022, 12:17 PM
hi there @johann, pods are indeed with limited resources. but since we're going to try CeleryExecutor (so we can create different priority/resources queues) we won't try to tune our resources for multiprocess execution. thanks!
👍 1
3 Views