https://dagster.io/ logo
#ask-community
Title
# ask-community
n

Navya Krishna Ramuni

08/11/2022, 10:47 AM
Hi Team, I have this issue coming up when running the pieplines in multi-processor mode on kubernetes. I have dynamic graphs in my job. But the pipeline runs fine if executed in a single process mode({'execution':{'config': {in_process: ''}}})
Copy code
dagster._check.CheckError: Invariant failed. Description: Attempted to mark step import_uds_session[0] as complete that was not known to be in flight
  File "/home/dagster/python/dagster/core/execution/api.py", line 822, in pipeline_execution_iterator
    for event in pipeline_context.executor.execute(pipeline_context, execution_plan):
  File "/home/dagster/python/dagster/core/executor/multiprocess.py", line 239, in execute
    active_execution.handle_event(step_failure_event)
  File "/home/dagster/python/dagster/core/execution/plan/active.py", line 400, in handle_event
    self.mark_failed(step_key)
  File "/home/dagster/python/dagster/core/execution/plan/active.py", line 342, in mark_failed
    self._mark_complete(step_key)
  File "/home/dagster/python/dagster/core/execution/plan/active.py", line 387, in _mark_complete
    check.invariant(
  File "/home/dagster/python/dagster/_check/__init__.py", line 1455, in invariant
    raise CheckError(f"Invariant failed. Description: {desc}")
Also I notice that Kubernetes pod running this job is killed when this error is appeared. Does not have any idea on this issue ?
5 Views