Here s a new one `dagster check CheckError Invariant failed dagster #announcements

Here's a new one, `dagster.check.CheckError: Invar...

Noah K

12/02/2020, 3:43 PM

Here's a new one,

dagster.check.CheckError: Invariant failed. Description: Pipeline run condition_asset (20b0921f-2547-4e10-a237-05b83b27f696) in state PipelineRunStatus.STARTED, expected PipelineRunStatus.NOT_STARTED

johann

12/02/2020, 3:50 PM

Is the stack trace from a launcher? Are you instigating the run from dagit?

Noah K

12/02/2020, 3:51 PM

It's from this https://github.com/geomagical/dagster-geomagical/blob/main/dagster_geomagical/launcher_worker.py

Noah K

12/02/2020, 3:54 PM

Combined a bunch of the existing launchers to make a new celery-based one

johann

12/02/2020, 3:55 PM

Do you have retries configured by chance?

johann

12/02/2020, 3:57 PM

task_acks_late

is on. A possible cause is that a worker got far enough with a pipeline to mark it as started, then crashed

Noah K

12/02/2020, 4:05 PM

Hmm, fair. Is there any way to reset things 🙂

Noah K

12/02/2020, 4:06 PM

Since if the worker did crash, I would want to try and restart as much as possible

Noah K

12/02/2020, 4:07 PM

For the short term I can just disable that

johann

12/02/2020, 5:54 PM

I think that is the best approach in the short term. Long term there’s some work on our end to make retries easier for you here. Either the call sites that instigate runs (e.g. scheduler) should resubmit a run after a fail, or we should make it so that if you retry one of your tasks with the same run_id we can pick up from the failure

johann

12/02/2020, 5:56 PM

Creating an issue is helpful if you’d like!

Noah K

12/02/2020, 5:58 PM

👍 Thanks

14 Views

Open in Slack

Previous Next