My pipelines are failing frequently with a

KeyboardInterrupt

error. They are running with the K8sRunLauncher on EKS with Spot instances. The failure occurs at different solids throughout the pipeline and it appears Dagster is unable to restart from this failed state:

Invariant failed, Description: Pipeline run ... in state PipelineRunStatus.FAILUER, expected NOT_STARTED or STARTING

so that once a pod is evicted and a new one created, the job fails to run in the new pod. Our pipelines run for hours, some even for days, and pod eviction is not uncommon. I have seen the suggestions for

cluster_autoscaler

, which we are not using, and

podDisruptionBudgets

, also not an option as we are on k8s v1.19. But these are workarounds and do not account for all possible failures. It would be ideal if a pipeline could restart automatically should its pod be evicted. If there is a solution I am missing please let me know.

With IO managers you can do manual retries but I think automatic stuff is still WIP.

Yes, your IOManager works great and the manual retries are really nice, the Dagit UI for pipeline running is beautifully done, but automatic retries for pipelines which fail due to resource issues is key. If we need manual intervention it defeats the purpose of automating workflows with Dagster. We’re looking at On Demand instances instead of spot but they are more expensive and still not a great solution. Can you give me any idea on when Dagster may support this?

I'm just a fellow user so that's not my department 😄

oh my goodness 🤣, my bad, sorry about that and thanks @Noah K for your suggestion

I'm sure some Elementl folks will be along soon and can give you a better idea of stuff from their side

Looks like this also came up in https://dagster.slack.com/archives/C014N0PK37E/p1624523418142400?thread_ts=1624264314.137600&cid=C014N0PK37E. Mind filing a feature request for this?

The second of those issues has a ticket filed 🙂

You’re right - here’s a similar issue you can upvote https://github.com/dagster-io/dagster/issues/3705

Thank you @rex and @Noah K. I upvoted and added to both of those requests. 🤞