any idea what would cause this error I m on GKE and looking

Question

any idea what would cause this error I m on GKE and looking at Logs Explorer there aren t any OOM or storage exceeded errors ```2022 05 23 17 26 42 +0000 dagster DEBUG powerschool resync 51ab1aae c307 4df5 aebb d76373179284 1 ENGINE EVENT Multiprocess executor received termination signal forwarding to active child processes```

Charlie Bini · Answer

I m guessing it could be CPU if it s not just some freak occurrence but I m not sure what to look for Don t see any logs pointing to the CPU

daniel · Answer

Are there any clues if you describe the pod Sometimes this can happen if k8s is bringing down the whole node

daniel · Answer

that the pod is on

Charlie Bini · Answer

using kubectl you mean

daniel · Answer

yeah

Charlie Bini · Answer

i ll check

Liezl Puzon · Answer

i haven t looked deeply into errors like this but previously when I was exceeding CPU and my resource allocation wasn t configured properly my pods appear to get stuck jobs run way longer than they should be running but that was before when I was using xgboost without a limit on threads

Charlie Bini · Answer

so nothing suspect but I m now noticing that the initial run failed to start with ```Run b447e7fb 8a16 4d9a b80d a582c0bd7f12 has not started execution after 313 06851840019226 seconds which is longer than the timeout of 300 seconds Marking run as failed ```

Charlie Bini · Answer

gonna dig through the logs some more and see what I come up with

daniel · Answer

That timeout is configurable in Dagster Cloud if you d like it to give more time before deciding that the run is unlikely to ever start

daniel · Answer

typically there would be some clue in the pod k8s job about why it didn t start up is it possible your cluster doesn t have sufficient resources to schedule the pod

Charlie Bini · Answer

yeah looking something like that

Charlie Bini · Answer

```0 4 nodes are available 1 node s had taint ToBeDeletedByClusterAutoscaler 1653326797 that the pod didn t tolerate 2 Insufficient memory 3 Insufficient cpu ```

Charlie Bini · Answer

I m using Autopilot so it might have been taking longer than usual to find a node