got this error today, is this a bug? all steps completed successfully, but this caused the job to be marked as a failure and restart

dagster._check.CheckError: Invariant failed. Description: Attempted to mark step load_destination[gsheets_survey_completion_6] as complete that was not known to be in flight

  File "/root/app/__pypackages__/3.10/lib/dagster/_core/execution/api.py", line 1035, in pipeline_execution_iterator
    for event in pipeline_context.executor.execute(pipeline_context, execution_plan):
  File "/root/app/__pypackages__/3.10/lib/dagster/_core/executor/step_delegating/step_delegating_executor.py", line 220, in execute
    active_execution.handle_event(dagster_event)
  File "/root/app/__pypackages__/3.10/lib/dagster/_core/execution/plan/active.py", line 402, in handle_event
    self.mark_success(step_key)
  File "/root/app/__pypackages__/3.10/lib/dagster/_core/execution/plan/active.py", line 346, in mark_success
    self._mark_complete(step_key)
  File "/root/app/__pypackages__/3.10/lib/dagster/_core/execution/plan/active.py", line 387, in _mark_complete
    check.invariant(
  File "/root/app/__pypackages__/3.10/lib/dagster/_check/__init__.py", line 1470, in invariant
    raise CheckError(f"Invariant failed. Description: {desc}")

hi @Charlie Bini! For some extra context, did this error happen on a run that was a re-execution of a previous run (e.g. re-executing from a failure), or on a regular run from the start of the job?

it happened in the root run

got it, looking into this

yup I can share, where should I send it?

dms work!

Hi Charlie- could you also DM one of us the result of

kubectl get -o yaml dagster-run-af7970df-09ba-4bcf-97b7-d0f8de1d90ee

just sent DM'd it to @owen. That's not something I'm setting myself, but perhaps Autopilot is setting it?

Nothing out of ordinary in the Job spec. Wondering if it is related to this prefect autopilot issue https://github.com/PrefectHQ/prefect/issues/5485