I've had a lot of job failures with this: ```An ex...
# dagster-plus
I've had a lot of job failures with this:
Copy code
An exception was thrown during execution that is likely a framework error, rather than an error in user code.
dagster._core.errors.DagsterInvariantViolationError: __ASSET_JOB_0 has no op named run_dbt_7869a_c792f.
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_grpc/impl.py", line 137, in core_execute_run
    yield from execute_run_iterator(
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/execution/api.py", line 866, in __iter__
    yield from self.execution_context_manager.prepare_context()
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_utils/__init__.py", line 491, in generate_setup_events
    obj = next(self.generator)
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/execution/context_creation_job.py", line 291, in orchestration_context_event_generator
    context_creation_data = create_context_creation_data(
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/execution/context_creation_job.py", line 116, in create_context_creation_data
    resource_keys_to_init=get_required_resource_keys_to_init(execution_plan, job_def),
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/execution/resources_init.py", line 375, in get_required_resource_keys_to_init
    get_required_resource_keys_for_step(job_def, step, execution_plan)
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/execution/resources_init.py", line 427, in get_required_resource_keys_for_step
    source_manager_key = execution_plan.get_manager_key(source_handle, job_def)
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/execution/plan/plan.py", line 651, in get_manager_key
    return _get_manager_key(self.step_dict_by_key, step_output_handle, job_def)
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/execution/plan/plan.py", line 1341, in _get_manager_key
    output_def = pipeline_def.get_node(node_handle).output_def_named(step_output.name)
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/definitions/job_definition.py", line 407, in get_node
    return self._graph_def.get_node(handle)
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/definitions/graph_definition.py", line 340, in get_node
    node = self.node_named(name)
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/definitions/graph_definition.py", line 327, in node_named
    raise DagsterInvariantViolationError(f"{self._name} has no op named {name}.")
Hi Brendan - was this in cloud? If so would you be able to link to a run where this happened?
a couple more questions -- what
version are you on and are you using
Yup, in the cloud. Using
Here's a link to the run: https://mtm-data-research.dagster.cloud/prod/runs/5b29a12e-fcce-4161-b26e-6c27226a0ac5
version 1.3.9
It's possible this happened kicking off a 're-execute' from the gui, but I honestly don't recall exactly how it happened.
gotcha, thanks for that info. just confirming that we're still looking into why this might be happening
👍 1
@Brendan Jackson this issue has been quite challenging to replicate, so still looking into the root cause, but one user reported that explicitly setting the
parameter when calling
helped with the issue (still unclear if that's related, or just a red herring).
@owen Well from my perspective it's no big deal - it hasn't happened again!
I'll watch for it and if it happens again try to get more exact steps for reproducing. I only flagged it here because the stack trace seemed to indicate something internal was amiss.
gotcha, thanks for the update blob salute
necromancy, but i see this happening on my non-cloud k8s deployment with the latest version.