I ve had a lot of job failures with this ```An exception was dagster #dagster-plus

I've had a lot of job failures with this: ```An ex...

Brendan Jackson

06/13/2023, 1:14 PM

I've had a lot of job failures with this:

Copy code

An exception was thrown during execution that is likely a framework error, rather than an error in user code.
dagster._core.errors.DagsterInvariantViolationError: __ASSET_JOB_0 has no op named run_dbt_7869a_c792f.
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_grpc/impl.py", line 137, in core_execute_run
    yield from execute_run_iterator(
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/execution/api.py", line 866, in __iter__
    yield from self.execution_context_manager.prepare_context()
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_utils/__init__.py", line 491, in generate_setup_events
    obj = next(self.generator)
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/execution/context_creation_job.py", line 291, in orchestration_context_event_generator
    context_creation_data = create_context_creation_data(
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/execution/context_creation_job.py", line 116, in create_context_creation_data
    resource_keys_to_init=get_required_resource_keys_to_init(execution_plan, job_def),
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/execution/resources_init.py", line 375, in get_required_resource_keys_to_init
    get_required_resource_keys_for_step(job_def, step, execution_plan)
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/execution/resources_init.py", line 427, in get_required_resource_keys_for_step
    source_manager_key = execution_plan.get_manager_key(source_handle, job_def)
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/execution/plan/plan.py", line 651, in get_manager_key
    return _get_manager_key(self.step_dict_by_key, step_output_handle, job_def)
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/execution/plan/plan.py", line 1341, in _get_manager_key
    output_def = pipeline_def.get_node(node_handle).output_def_named(step_output.name)
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/definitions/job_definition.py", line 407, in get_node
    return self._graph_def.get_node(handle)
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/definitions/graph_definition.py", line 340, in get_node
    node = self.node_named(name)
  File "/venvs/fb460baabe2f/lib/python3.8/site-packages/dagster/_core/definitions/graph_definition.py", line 327, in node_named
    raise DagsterInvariantViolationError(f"{self._name} has no op named {name}.")

daniel

06/13/2023, 2:49 PM

Hi Brendan - was this in cloud? If so would you be able to link to a run where this happened?

owen

06/13/2023, 4:28 PM

a couple more questions -- what

dagster

version are you on and are you using

load_assets_from_dbt_manifest

load_assets_from_dbt_project

Brendan Jackson

06/13/2023, 4:31 PM

Yup, in the cloud. Using

load_assets_from_dbt_project

Here's a link to the run: https://mtm-data-research.dagster.cloud/prod/runs/5b29a12e-fcce-4161-b26e-6c27226a0ac5

Brendan Jackson

06/13/2023, 4:32 PM

dagster

version 1.3.9

Brendan Jackson

06/13/2023, 4:33 PM

It's possible this happened kicking off a 're-execute' from the gui, but I honestly don't recall exactly how it happened.

owen

06/13/2023, 10:22 PM

gotcha, thanks for that info. just confirming that we're still looking into why this might be happening

👍 1

owen

06/16/2023, 10:41 PM

@Brendan Jackson this issue has been quite challenging to replicate, so still looking into the root cause, but one user reported that explicitly setting the

op_name

parameter when calling

load_assets_from_dbt...

helped with the issue (still unclear if that's related, or just a red herring).

Brendan Jackson

06/20/2023, 10:05 AM

@owen Well from my perspective it's no big deal - it hasn't happened again!

Brendan Jackson

06/20/2023, 10:06 AM

I'll watch for it and if it happens again try to get more exact steps for reproducing. I only flagged it here because the stack trace seemed to indicate something internal was amiss.

owen

06/21/2023, 6:10 PM

gotcha, thanks for the update blob salute

Christian Hollinger

09/01/2023, 8:05 PM

necromancy, but i see this happening on my non-cloud k8s deployment with the latest version.

76 Views

Open in Slack

Previous Next