https://dagster.io/ logo
Join Slack
Powered by
I'm getting "Error occurred while loading input" t...
# ask-community
m
I'm getting "Error occurred while loading input" that just says "Invariant failed." I think this worked with 0.14.7 and started failing with 0.14.9. Here's the stack trace:
Copy code
battery_data_job_local - c49da2c4-6a0f-41b2-a477-f9baa34afc1f - 1741477 - data_pipe_graph_sasquatch_anode.raw_data_graph._publish_raw_data_trace_context - STEP_FAILURE - Execution of step "data_pipe_graph_sasquatch_anode.raw_data_graph._publish_raw_data_trace_context" failed.

dagster.core.errors.DagsterExecutionLoadInputError: Error occurred while loading input "pipe" of step "data_pipe_graph_sasquatch_anode.raw_data_graph._publish_raw_data_trace_context"::

dagster.check.CheckError: Invariant failed.

Stack Trace:
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/core/execution/plan/utils.py", line 47, in solid_execution_error_boundary
    yield
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/core/execution/plan/inputs.py", line 607, in _load_input_with_input_manager
    value = input_manager.load_input(context)
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/core/storage/fs_io_manager.py", line 152, in load_input
    context.add_input_metadata({"path": MetadataValue.path(os.path.abspath(filepath))})
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/core/execution/context/input.py", line 325, in add_input_metadata
    if self.asset_key:
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/core/execution/context/input.py", line 216, in asset_key
    check.invariant(len(matching_input_defs) == 1)
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/check/__init__.py", line 1167, in invariant
    raise CheckError("Invariant failed.")
The op in question is:
Copy code
@op(
    ins={"start": In(Nothing)},
)
def _publish_raw_data_trace_context(context):
    publish_current_trace_context(context)
And the 
pipe
parameter it mentions is an enum value (produced as a constant form a different op, passed through a @graph. I'm not really sure where to look next, any suggestions?
Confirmed that the same graph works if I drop back to 
0.14.8
. Happy to pull a debug file if that's helpful.
j
We have a fix going out today in 0.14.10
🎉 1
m
Thanks, confirmed I'm not hitting that error after upgrading to 
0.14.10
.
I'm seeing another invariant error though, when retrying from a failed step w/in a subgraph:
Copy code
2022-04-21 21:57:06 -0400 - dagster - ERROR - battery_data_job_local - cd139594-8283-49d9-9e36-24504be680f3 - 2375675 - data_pipe_graph_sasquatch_cathode.raw_data_graph._normalize_arbin4_task[arbin_f_N5MJH] - STEP_FAILURE - Execution of step "data_pipe_graph_sasquatch_cathode.raw_data_graph._normalize_arbin4_task[arbin_f_N5MJH]" failed.

dagster.core.errors.DagsterInvariantViolationError: Attempting to access run_id, but it was not provided when constructing the OutputContext

Stack Trace:
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/core/execution/plan/execute_plan.py", line 232, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/core/execution/plan/execute_step.py", line 360, in core_dagster_event_sequence_for_step
    for evt in _type_check_and_store_output(step_context, user_event, input_lineage):
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/core/execution/plan/execute_step.py", line 417, in _type_check_and_store_output
    for evt in _store_output(step_context, step_output_handle, output, input_lineage):
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/core/execution/plan/execute_step.py", line 552, in _store_output
    for elt in iterate_with_context(
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/utils/__init__.py", line 401, in iterate_with_context
    next_output = next(iterator)
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/core/execution/plan/execute_step.py", line 542, in _gen_fn
    gen_output = output_manager.handle_output(output_context, output.value)
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/core/storage/fs_io_manager.py", line 116, in handle_output
    filepath = self._get_path(context)
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/core/storage/fs_io_manager.py", line 99, in _get_path
    keys = context.get_output_identifier()
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/core/execution/context/output.py", line 380, in get_output_identifier
    run_id = self.run_id
  File "/home/mfickett/Documents/mfickett/dev/data-pipeline/.direnv/python-3.9.5/lib/python3.9/site-packages/dagster/core/execution/context/output.py", line 157, in run_id
    raise DagsterInvariantViolationError(
j
Was this a dynamic out?
m
Yes, the 
_normalize_arbin4_task
op is processing a dynamic out.
j
I think this is related to this issue. Fix planned for first week of May https://github.com/dagster-io/dagster/issues/6044
ty thankyou 1
9 Views
Open in Slack
PreviousNext
Powered by