https://dagster.io/ logo
#ask-community
Title
# ask-community
c

Charlie Bini

09/19/2022, 2:43 PM
got this error today, is this a bug? all steps completed successfully, but this caused the job to be marked as a failure and restart
Copy code
dagster._check.CheckError: Invariant failed. Description: Attempted to mark step load_destination[gsheets_survey_completion_6] as complete that was not known to be in flight

  File "/root/app/__pypackages__/3.10/lib/dagster/_core/execution/api.py", line 1035, in pipeline_execution_iterator
    for event in pipeline_context.executor.execute(pipeline_context, execution_plan):
  File "/root/app/__pypackages__/3.10/lib/dagster/_core/executor/step_delegating/step_delegating_executor.py", line 220, in execute
    active_execution.handle_event(dagster_event)
  File "/root/app/__pypackages__/3.10/lib/dagster/_core/execution/plan/active.py", line 402, in handle_event
    self.mark_success(step_key)
  File "/root/app/__pypackages__/3.10/lib/dagster/_core/execution/plan/active.py", line 346, in mark_success
    self._mark_complete(step_key)
  File "/root/app/__pypackages__/3.10/lib/dagster/_core/execution/plan/active.py", line 387, in _mark_complete
    check.invariant(
  File "/root/app/__pypackages__/3.10/lib/dagster/_check/__init__.py", line 1470, in invariant
    raise CheckError(f"Invariant failed. Description: {desc}")
o

owen

09/19/2022, 4:44 PM
hi @Charlie Bini! For some extra context, did this error happen on a run that was a re-execution of a previous run (e.g. re-executing from a failure), or on a regular run from the start of the job?
c

Charlie Bini

09/19/2022, 4:48 PM
it happened in the root run
o

owen

09/19/2022, 4:51 PM
got it, looking into this
hey @Charlie Bini would it be possible to share a debug file (or a link to the run, if you're using cloud?) also, is this the first/only time that this job has failed in this way?
c

Charlie Bini

09/19/2022, 6:08 PM
yup I can share, where should I send it?
it's the first time I recall seeing that error
o

owen

09/19/2022, 6:09 PM
dms work!
j

johann

09/19/2022, 6:35 PM
Hi Charlie- could you also DM one of us the result of
kubectl get -o yaml dagster-run-af7970df-09ba-4bcf-97b7-d0f8de1d90ee
I’m wondering if you set
completions
in the Job spec? It seems like a bunch of pods started for that step in serial
c

Charlie Bini

09/19/2022, 6:45 PM
just sent DM'd it to @owen. That's not something I'm setting myself, but perhaps Autopilot is setting it?
j

johann

09/19/2022, 7:44 PM
Nothing out of ordinary in the Job spec. Wondering if it is related to this prefect autopilot issue https://github.com/PrefectHQ/prefect/issues/5485
5 Views