Hi team When we executed pipelines k8s jobs got stuck and th

Question

Hi team When we executed pipelines k8s jobs got stuck and there is no pod created how to debug can anyone help me on this issue

daniel · Answer

Does describing the job in kubectl give any clues kubectl describe job lt job name here gt

Mohammad Nazeeruddin · Answer

k8s pods not created to describe

daniel · Answer

This would be the job not the pod I updated the command I wrote above

Mohammad Nazeeruddin · Answer

` root e88ddc17ffaf centos kubectl describe job dagster run 8d02d3ff 56ca 4240 9b97 3bed5f170e0d n dagster` Error from server NotFound jobs batch dagster run 8d02d3ff 56ca 4240 9b97 3bed5f170e0d not found

Mohammad Nazeeruddin · Answer

jobs not created to check it s showing in dagit UI only

daniel · Answer

Does kubectl get jobs show a job with that run ID in its name somewhere

Mohammad Nazeeruddin · Answer

`Run ID` 8d02d3ff 56ca 4240 9b97 3bed5f170e0d are you talking about this ^ run Id

daniel · Answer

That s right

Mohammad Nazeeruddin · Answer

Okay this id will helpful to debug

daniel · Answer

If you describe the job it might explain why it didn t create a pod

daniel · Answer

Dagster created a Kubernetes job <https kubernetes io docs concepts workloads controllers job |https kubernetes io docs concepts workloads controllers job > which should normally create a pod

Mohammad Nazeeruddin · Answer

But the problem is it s not creating any job to describe got this error because job pod not created Error from server NotFound jobs batch dagster run 8d02d3ff 56ca 4240 9b97 3bed5f170e0d not found I had only dagit daemon and repos pods

daniel · Answer

What command did you run to make that list

daniel · Answer

Jobs are different than pods you would need to run kubectl get jobs to generate the list of jobs

daniel · Answer

It would be surprising if there were no job given the event log message that says that it created a job successfully

Mohammad Nazeeruddin · Answer

Yeah showing in dagit UI job created but do t know what happening not showing in terminal I ran kubectl describe job dagster run 8d02d3ff 56ca 4240 9b97 3bed5f170e0d n dagster and kubectl get jobs n dagster i got gt No resources found

Mohammad Nazeeruddin · Answer

some time getting this error gt `dagster core errors DagsterExecutionInterruptedError` `File usr local lib python3 8 site packages dagster core execution api py line 748 in pipeline execution iterator` `for event in pipeline context executor execute pipeline context execution plan ` `File usr local lib python3 8 site packages dagster core executor in process py line 38 in execute` `yield from iter ` `File usr local lib python3 8 site packages dagster core execution api py line 822 in iter ` `yield from self execution context manager prepare context ` `File usr local lib python3 8 site packages dagster utils init py line 447 in generate setup events` `obj = next self generator ` `File usr local lib python3 8 site packages dagster core execution context creation pipeline py line 282 in execution context event generator` `yield from resources manager generate setup events ` `File usr local lib python3 8 site packages dagster utils init py line 447 in generate setup events` `obj = next self generator ` `File usr local lib python3 8 site packages dagster core execution resources init py line 227 in resource initialization event generator` `yield from core resource initialization event generator ` `File usr local lib python3 8 site packages dagster core execution resources init py line 160 in core resource initialization event generator` `for event in manager generate setup events ` `File usr local lib python3 8 site packages dagster utils init py line 447 in generate setup events` `obj = next self generator ` `File usr local lib python3 8 site packages dagster core execution resources init py line 285 in single resource event generator` `with user code error boundary ` `File usr local lib python3 8 contextlib py line 113 in enter ` `return next self gen ` `File usr local lib python3 8 site packages dagster core errors py line 181 in user code error boundary` `with raise execution interrupts ` `File usr local lib python3 8 contextlib py line 113 in enter ` `return next self gen ` `File usr local lib python3 8 site packages dagster core errors py line 151 in raise execution interrupts` `with raise interrupts as DagsterExecutionInterruptedError ` `File usr local lib python3 8 contextlib py line 113 in enter ` `return next self gen ` `File usr local lib python3 8 site packages dagster utils interrupts py line 73 in raise interrupts as` `raise error cls `

daniel · Answer

Sorry where is that error appearing

Mohammad Nazeeruddin · Answer

pipeline execution time got this gt `dagster core errors DagsterExecutionInterruptedError`

daniel · Answer

That would only happen inside the pod I thought there was no pod

Mohammad Nazeeruddin · Answer

Yes no pod

daniel · Answer

Something doesn t add up here the only thing that would have logged that message would have been from inside the pod

daniel · Answer

It seems like your cluster might be overloaded or otherwise having trouble spinning up pods or keeping them up That error would happen if the cluster terminated your pod

Mohammad Nazeeruddin · Answer

Okay