I have a dagster deployment on AWS EKS which works quite well. I run long running dagster jobs or steps: several hours to several days. I have been having trouble with kubernetes vertical auto-scaling evicting nodes on which these long running jobs are still running. basically ruining my job. Does anybody have similar problems and if yes any mitigation? Discussing with ChatGPT on the subject, the options I see are 1) implement some kubernetes finalizers that would prevent pods running jobs to be deleted 2) implement a custom autoscaler or controller.

It can probably be fixed with this annotation: https://kubernetes.io/docs/reference/labels-annotations-taints/#cluster-autoscaler-kubernetes-io-safe-to-evict

Unfortunately, it does not seem to be enough. I already have this annotation.

Are you sure the annotation is applied on the pod and not only on the job?

I am not sure I understand your question. I only see the annotation on the pod, not on the job which would be correct because this is a pod annotation

a job creates a pod, but the job’s annotation are not necessarily the same as its pod annotations, that’s why i was asking

So I see the annotation on the pod which it should help. I was surprised myself that it would not be enough. ChatGPT says there is no guarantee:

https://aws.github.io/aws-eks-best-practices/cluster-autoscaling/#prevent-scale-down-eviction aws EKS docs

ok it seems that I am victim of the known problem of EKS which is the Availability Zone Rebalance which does not take into account the annotation. I have observed that my pods were terminated on nodes which were evicted because of AZrebalance. I will try to suspend the availability zone rebalance and see if it helps.