Have you tried cleaning involved cache? like docker caching, etc?

I haven't but would that be somewhat related when running Dagster in k8s?

I think you should try this out

Hi Caio - that "Sending run termination request" should only appear when somebody presses the "Terminate" button in Dagit or terminates a run over the GraphQL API. Are you certain that nobody did that just before that line was logged?

@daniel my apologies, the log message is actually

Ignoring a duplicate run that was started from somewhere other than the run monitor daemon

I see - what that sounds to me is that your k8s cluster may be spinning down the pod on which your run is happening and then restarting it

What I mean by that is the subsequent step in the run won't resume. It's an intermittent issue which happens in the middle of a run

and run retries useful for kicking off a new run to try to recover: https://docs.dagster.io/deployment/run-retries#run-retries

I have enabled runMonitor and was going to give that a try. Let's say a node scale down event happens in the cluster and hence the pod get's killed. Will the `

max_resume_run_attempts > 0

try to recover the run?

that's this bit here: https://docs.dagster.io/deployment/run-monitoring#resuming-runs-after-run-worker-crashes-experimental - which only works in certain situations, I'd recommend using the "Run retries" 2nd link i posted to make it retry

it's this config here right? https://github.com/dagster-io/dagster/blob/master/helm/dagster/values.yaml#L1078-L1080

yeah

would that retry the entire run or resume where it was left off?

Believe this last bit addresses that: https://docs.dagster.io/deployment/run-retries#retry-strategy

ty! We'll give it a try