Hi Team We are running `Dagster 0 11 12` in our main k8s clu

Question

Hi Team We are running `Dagster 0 11 12` in our main k8s cluster and we are having some problems with node shutdowns You can find the error logs in the thread Basically the k8s cluster sends a `SIGTERM` to the Dagster job pod and Dagster behaves in an unexpected way Raises an `ENGINE EVENT` exception when receiving the `SIGTERM` Returns 0 to the pod marking it as completed when it actually failed Stops any other process e g running a failure hook to get notified That means that if a pod is sigtermed because of a node shutdown The pipeline immediately fails telling k8s that the pod terminated successfully We don t get any notification from the failure hook There is no automatic retry Dagster doesn t support re running failed pipelines Unfortunately we were not able to find a solution for this issue are we missing something slightly smiling face

Leonardo Cotti · Answer

Error logs `STEP FAILURE` ```dagster core errors DagsterExecutionInterruptedError File usr local lib python3 8 site packages dagster core execution plan execute plan py line 193 in dagster event sequence for step for step event in check generator step events File usr local lib python3 8 site packages dagster core execution plan execute step py line 305 in core dagster event sequence for step for user event in check generator File usr local lib python3 8 site packages dagster core execution plan execute step py line 64 in step output error checked user event sequence for user event in user event sequence File usr local lib python3 8 site packages dagster core execution plan execute step py line 599 in user event sequence for step compute fn for event in iterate with context File usr local lib python3 8 site packages dagster utils init py line 382 in iterate with context next output = next iterator File usr local lib python3 8 site packages dagster core execution plan compute py line 126 in execute core compute for step output in yield compute results step context inputs compute fn File usr local lib python3 8 site packages dagster core execution plan compute py line 109 in yield compute results for event in user event generator File usr local lib python3 8 site packages dagster dbt cli solids py line 193 in dbt cli run cli output = execute cli File usr local lib python3 8 site packages dagster dbt cli utils py line 65 in execute cli for raw line in process stdout File usr local lib python3 8 site packages dagster utils interrupts py line 78 in new signal handler raise error cls ``` `PIPELINE FAILURE`

Leonardo Cotti · Answer

```dagster core errors DagsterExecutionInterruptedError File usr local lib python3 8 site packages dagster core execution api py line 762 in pipeline execution iterator for event in pipeline context executor execute pipeline context execution plan File usr local lib python3 8 site packages dagster core executor in process py line 38 in execute yield from iter File usr local lib python3 8 site packages dagster core execution api py line 841 in iter yield from self iterator File usr local lib python3 8 site packages dagster core execution plan execute plan py line 69 in inner plan execution iterator for step event in check generator dagster event sequence for step step context File usr local lib python3 8 site packages dagster core execution plan execute plan py line 270 in dagster event sequence for step raise interrupt error File usr local lib python3 8 site packages dagster core execution plan execute plan py line 193 in dagster event sequence for step for step event in check generator step events File usr local lib python3 8 site packages dagster core execution plan execute step py line 305 in core dagster event sequence for step for user event in check generator File usr local lib python3 8 site packages dagster core execution plan execute step py line 64 in step output error checked user event sequence for user event in user event sequence File usr local lib python3 8 site packages dagster core execution plan execute step py line 599 in user event sequence for step compute fn for event in iterate with context File usr local lib python3 8 site packages dagster utils init py line 382 in iterate with context next output = next iterator File usr local lib python3 8 site packages dagster core execution plan compute py line 126 in execute core compute for step output in yield compute results step context inputs compute fn File usr local lib python3 8 site packages dagster core execution plan compute py line 109 in yield compute results for event in user event generator File usr local lib python3 8 site packages dagster dbt cli solids py line 193 in dbt cli run cli output = execute cli File usr local lib python3 8 site packages dagster dbt cli utils py line 65 in execute cli for raw line in process stdout File usr local lib python3 8 site packages dagster utils interrupts py line 78 in new signal handler raise error cls ``` `ENGINE EVENT` ```An exception was thrown during execution that is likely a framework error rather than an error in user code dagster check CheckError Invariant failed Description Pipeline run dbt pipeline 348a5c19 3973 4d1c afe5 12f926e42bb4 in state PipelineRunStatus FAILURE expected NOT STARTED or STARTING Stack Trace File usr local lib python3 8 site packages dagster grpc impl py line 86 in core execute run yield from execute run iterator recon pipeline pipeline run instance File usr local lib python3 8 site packages dagster core execution api py line 74 in execute run iterator check invariant File usr local lib python3 8 site packages dagster check init py line 167 in invariant raise CheckError f Invariant failed Description desc ```

max · Answer

< prha>

Alessandro Marrella · Answer

Hi < Leonardo Cotti> not sure if this entirely helps your case and SIGTERM should probably be handled more gracefully but as a workaround if you set PodDisruptionBudgets for run pods the node will wait to shutdown until your job is completed unless it s a spot pre emptible instance example this is what i do in my cluster at least

Leonardo Cotti · Answer

I had a catch up with our internal team that managed the k8s cluster and we found that using the annotation ``` false ``` in the pipeline tags worked

Leonardo Cotti · Answer

They also mentioned that if there is a force shutdown of the node the annotation won t work and we might get the same issue Although it is a rare event is there any plan to support pod eviction The main two problems are No automatic retry of the pipeline The failure hooks don t work no notifications

johann · Answer

Hi Leonardo late followup here but we just released automatic retries <https docs dagster io deployment run retries>