Good morning everyone! I've got a question: I am using preemptible instances in my GCP cluster. What is the default behavior of dagster if, during a pipeline run, the target node is effectively terminated? I saw the release notes of 0.10 version mentioning support for spot instances but cannot find any specifics.
04/12/2021, 1:55 PM
Hi @Rubén Lopez Lozoya - I think the answer is we currently don’t have a great story here. If a step worker is killed (doing solid compute) it may retry, depending on the executor’s retry configuration. If the run worker (doing the pipeline orchestration) is killed, the run should be marked as failed. We are looking to add retries/fault tolerance here in the future, if you’d like to create an issue it helps with prioritization