https://dagster.io/ logo
#announcements
Title
# announcements
n

Noah K

12/02/2020, 3:43 PM
Here's a new one,
dagster.check.CheckError: Invariant failed. Description: Pipeline run condition_asset (20b0921f-2547-4e10-a237-05b83b27f696) in state PipelineRunStatus.STARTED, expected PipelineRunStatus.NOT_STARTED
j

johann

12/02/2020, 3:50 PM
Is the stack trace from a launcher? Are you instigating the run from dagit?
Combined a bunch of the existing launchers to make a new celery-based one
j

johann

12/02/2020, 3:55 PM
Do you have retries configured by chance?
Ah
task_acks_late
is on. A possible cause is that a worker got far enough with a pipeline to mark it as started, then crashed
n

Noah K

12/02/2020, 4:05 PM
Hmm, fair. Is there any way to reset things 🙂
Since if the worker did crash, I would want to try and restart as much as possible
For the short term I can just disable that
j

johann

12/02/2020, 5:54 PM
I think that is the best approach in the short term. Long term there’s some work on our end to make retries easier for you here. Either the call sites that instigate runs (e.g. scheduler) should resubmit a run after a fail, or we should make it so that if you retry one of your tasks with the same run_id we can pick up from the failure
Creating an issue is helpful if you’d like!
n

Noah K

12/02/2020, 5:58 PM
👍 Thanks
9 Views