hi all in relation to

Question

hi all in relation to <https dagster slack com archives C014N0PK37E p1655970710731319 thread ts=1655541804 905499 amp cid=C014N0PK37E> we are investigating slow spin up times for our k8s jobs sometimes taking more than 90 secs from execution to start of the job in k8s is there anything in particular we should look at we are running dagster helm 0 15 0 in AWS EKS 1 m4 large and 2 m4 medium nodes

Andrea Giardini · Answer

can you describe the pod and post here the list of pod events

Adam McCartney · Answer

```LAST SEEN TYPE REASON OBJECT MESSAGE 21m Normal Scheduled pod dagster run ab72cb46 74b0 4aeb b5cf ef870655b7d1 rj6jh Successfully assigned dagster dagster run ab72cb46 74b0 4aeb b5cf ef870655b7d1 rj6jh to ip 10 1 2 194 eu west 1 compute internal 21m Normal Pulling pod dagster run ab72cb46 74b0 4aeb b5cf ef870655b7d1 rj6jh Pulling image 21m Normal Pulled pod dagster run ab72cb46 74b0 4aeb b5cf ef870655b7d1 rj6jh Successfully pulled image in 108 165958ms 21m Normal Created pod dagster run ab72cb46 74b0 4aeb b5cf ef870655b7d1 rj6jh Created container dagster 21m Normal Started pod dagster run ab72cb46 74b0 4aeb b5cf ef870655b7d1 rj6jh Started container dagster```

Adam McCartney · Answer

the pod starts up pretty quickly but then theres a lag between the pod starting and the job executing

Andrea Giardini · Answer

pod logs

Adam McCartney · Answer

just checking for anything i need to redact

Andrea Giardini · Answer

take your time slightly smiling face

Adam McCartney · Answer

the pod is created at 2022 06 23T08 45 43+01 00 then the pod logs start at 2022 06 23T08 47 04+01 00

Adam McCartney · Answer

i d like to understand whats causing the gap inbetween

Andrea Giardini · Answer

all these events seem to be triggered in the same seconds correct me if i m wrong do you see a gap in the event list

Andrea Giardini · Answer

There is nothing before that

Adam McCartney · Answer

no perhaps a logging setting i could change if there is something missing

Andrea Giardini · Answer

what do you see happening during this gap does the pod get scheduled immediately and go in a Running state

Adam McCartney · Answer

yes thats correct

Andrea Giardini · Answer

can you describe the pod maybe you have very restrictive resource limits requests

Andrea Giardini · Answer

docker image size is usually a contributing factor but that does not seem to be a problem in your case

Adam McCartney · Answer

i m using the helm chart so any limits are most likely left as default

Andrea Giardini · Answer

Indeed the container starts quick ```Started Thu 23 Jun 2022 08 45 45 +0100 Finished Thu 23 Jun 2022 08 47 04 +0100``` Are you doing some pre processing pulling data from somewhere can you share your dockerfile maybe

Andrea Giardini · Answer

you could also try to start a run get into the pod with `k exec` as soon as it s running and run `ps aux` or any other tool to understand what it is doing

Adam McCartney · Answer

the image is prebuilt and in a registry there s no data being pulled from anywhere i ll try and get some more info from the pod when it spins up thanks for your help so far i may be back with more questions

Andrea Giardini · Answer

Sure thing happy to help

Binoy Shah · Answer

< Adam McCartney> was this issue resolved

Adam McCartney · Answer

not yet i ve rebuild our Kubernetes images which is based on python 3 9 slim with some extra logging so i m going to do some forensics soon its not top of my priority list right now and i might call on some pair programming with a colleague to figure it out