Hey, I am having this error while re-executing an EMR job from Failure:

Finished initialization of resources [adls2, emr_snowflake, pyspark_step_launcher, secrets, spark_io_manager].
Traceback (most recent call last):
  File "/mnt/tmp/spark-757ec306-1f9b-4a4d-beff-c87b932d5b67/emr_step_main.py", line 87, in <module>
    main(sys.argv[1], sys.argv[2])
  File "/mnt/tmp/spark-757ec306-1f9b-4a4d-beff-c87b932d5b67/emr_step_main.py", line 44, in main
    for event in run_step_from_ref(step_run_ref, instance):
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/plan/external_step.py", line 279, in run_step_from_ref
    step_context = step_run_ref_to_step_context(step_run_ref, instance)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/plan/external_step.py", line 258, in step_run_ref_to_step_context
    for _ in initialization_manager.prepare_context():
  File "/usr/local/lib/python3.7/site-packages/dagster/utils/__init__.py", line 462, in generate_setup_events
    obj = next(self.generator)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/context_creation_pipeline.py", line 269, in execution_context_event_generator
    _validate_plan_with_context(execution_context, execution_plan)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/context_creation_pipeline.py", line 411, in _validate_plan_with_context
    validate_reexecution_memoization(pipeline_context, execution_plan)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/memoization.py", line 20, in validate_reexecution_memoization
    invalid_run_id=parent_run_id,
dagster.core.errors.DagsterRunNotFoundError: Run id 32fd1987-f2c2-4b3f-8b24-f1b7067a67b3 set as parent run id was not found in instance

Hi Oluwashina… Are you re-executing from

dagit

? If so, I guess we can assume that the instance does know about the run

32fd1987-f2c2-4b3f-8b24-f1b7067a67b3

.

Yes, I am re-executing from dagit

And just to confirm:

32fd1987-f2c2-4b3f-8b24-f1b7067a67b3

is the failed run you are re-executing?

Yes

Any chance you can send over the gzipped debug file?