Hey all ever since updating to 0 11 13 I ve been getting lot

Question

Hey all ever since updating to 0 11 13 I ve been getting lots of these errors ```An exception was thrown during execution that is likely a framework error rather than an error in user code dagster check CheckError Invariant failed Description Pipeline run cleanup e44bf39c e57e 4318 ac8f 1f7c62667a40 in state PipelineRunStatus STARTED expected NOT STARTED or STARTING ``` Any idea what would be causing this I m running this on a k8s instance that s comprised of spot instances and from what I ve seen it looks like that might be a cause if the nodes are going down frequently But before on 0 11 12 it was failing less I also switched from celery k8s exec to just the k8s job exec during this time too

Kirk Stennett · Answer

It also looks like the scheduler spins up new pods when this happens to compensate for the old ones being removed Is there any tooling in future versions that would support dagster recognizing these new pods

johann · Answer

< Kirk Stennett> we re aware of this issue and I m actively working on improving our tolerance here We currently break if the run worker process which manages the run goes down and starting a new one logs the error you re seeing

Kirk Stennett · Answer

Any issue that I could follow along with provide background on < johann>

johann · Answer

I believe it would be a coincidence that this picked up in frequency with the upgrade

Kirk Stennett · Answer

Agreed I think this is just randomness with bidding I m also don t have full access over the cluster so it could be a few things I m pretty sure it s just coincidence

johann · Answer

gt Any issue that I could follow along with There is now <https github com dagster io dagster issues 4993>