Here s a new one `dagster check CheckError Invariant failed

Question

Here s a new one `dagster check CheckError Invariant failed Description Pipeline run condition asset 20b0921f 2547 4e10 a237 05b83b27f696 in state PipelineRunStatus STARTED expected PipelineRunStatus NOT STARTED`

johann · Answer

Is the stack trace from a launcher Are you instigating the run from dagit

Noah K · Answer

It s from this <https github com geomagical dagster geomagical blob main dagster geomagical launcher worker py>

Noah K · Answer

Combined a bunch of the existing launchers to make a new celery based one

johann · Answer

Do you have retries configured by chance

johann · Answer

Ah `task acks late` is on A possible cause is that a worker got far enough with a pipeline to mark it as started then crashed

Noah K · Answer

Hmm fair Is there any way to reset things slightly smiling face

Noah K · Answer

Since if the worker did crash I would want to try and restart as much as possible

Noah K · Answer

For the short term I can just disable that

johann · Answer

I think that is the best approach in the short term Long term there s some work on our end to make retries easier for you here Either the call sites that instigate runs e g scheduler should resubmit a run after a fail or we should make it so that if you retry one of your tasks with the same run id we can pick up from the failure

johann · Answer

Creating an issue is helpful if you d like

Noah K · Answer

+1 Thanks