Hey all, ever since updating to 0.11.13 I've been getting lots of these errors:

An exception was thrown during execution that is likely a framework error, rather than an error in user code.
dagster.check.CheckError: Invariant failed. Description: Pipeline run cleanup (e44bf39c-e57e-4318-ac8f-1f7c62667a40) in state PipelineRunStatus.STARTED, expected NOT_STARTED or STARTING.

Any idea what would be causing this? I'm running this on a k8s instance that's comprised of spot instances and from what I've seen it looks like that might be a cause if the nodes are going down frequently. But before on 0.11.12 it was failing less. I also switched from celery_k8s_exec to just the k8s_job_exec during this time too.

@Kirk Stennett we’re aware of this issue and I’m actively working on improving our tolerance here. We currently break if the run worker process (which manages the run) goes down, and starting a new one logs the error you’re seeing

Any issue that I could follow along with / provide background on? @johann

I believe it would be a coincidence that this picked up in frequency with the upgrade

Agreed, I think this is just randomness with bidding. I'm also don't have full access over the cluster so it could be a few things. I'm pretty sure it's just coincidence

Any issue that I could follow along with

There is now! https://github.com/dagster-io/dagster/issues/4993