Hi , we’ve got an issue when kubernetes pod (the dagster-run) killed unexpectedly. we’ve seen that the runs on dagit still keeps running although the pod were killed. could you help me understand why is that happens?

What was the “job termination reason”? can be seen when describing the K8s job itself My first suggestion would be some sort of out-of-memory issue for a complex Dagster job

it happened for us when using spot instance in our gke node pool, but even I just run manual jon in dagit and after minutes I delete the pod manually- I still see the dagit job run…

Do you have “run monitoring” enabled? https://docs.dagster.io/deployment/run-monitoring Re spot instances - you can look out for eviction requests events in the cluster

but why the dagit job wasn’t failed after the pod crashed? (although I still didn’t enabled run monitoring)

What happens exactly after the K8s job has crashed? Do you see logs written in Dagit? Do you see Ops running? What happens after ops finish their execution?

the ops keep running forever and we don’t see nothing on the ui

Do they “run” in Dagit or do their steps are running forever as well? What do you see on their logs (on the containers themselves)?

the steps are still running, and I don’t have any container to look at because the pod crashed.

what you’re seeing is the “hanging runs” described in the run monitoring doc - in short, dagster still sees the steps as running because nothing has communicated to it that the run is no longer running. let us know if you still see this after enabling run monitoring, but i suspect it’ll take care of the issue for you

and where should I add the run_moinitoring? in the helm chart? can you send me an example?