I am not sure what I am missing in the output if it needs to dagster #ask-community

I am not sure what I am missing in the output, if ...

Jeremy Lyman

04/03/2023, 10:01 PM

I am not sure what I am missing in the output, if it needs to be of a specific format but I keep getting the following error during a backfill operation.

Copy code

dagster._core.errors.DagsterExecutionHandleOutputError: Error occurred while handling output "result" of step "ods_components":

Jeremy Lyman

04/03/2023, 10:01 PM

Copy code

@asset(required_resource_keys={"ods"},
       partitions_def=hourly_partitions_def)
def ods_components(context) -> List[dict]:
    """Components from the ODS."""
    start, end = context.asset_partitions_time_window_for_output()
    # partition_date_str = context.asset_partition_key_for_output()
    get_dagster_logger().info(
        f"Materializing ODS assets between: {start} {end}"
    )
    return context.resources.ods.extract_between(start, end)

Copy code

dagster._core.errors.DagsterExecutionHandleOutputError: Error occurred while handling output "result" of step "ods_components":

The above exception was caused by the following exception:
AssertionError
  File ".pyenv/versions/3.11.2/lib/python3.11/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
  File ".pyenv/versions/3.11.2/lib/python3.11/site-packages/dagster/_utils/__init__.py", line 439, in iterate_with_context
    next_output = next(iterator)
                  ^^^^^^^^^^^^^^
  File ".pyenv/versions/3.11.2/lib/python3.11/site-packages/dagster/_core/execution/plan/execute_step.py", line 594, in _gen_fn
    gen_output = output_manager.handle_output(output_context, output.value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.11.2/lib/python3.11/site-packages/dagster/_core/storage/upath_io_manager.py", line 235, in handle_output
    assert len(paths) == 1
           ^^^^^^^^^^^^^^^

sandy

04/04/2023, 12:25 AM

Hi @Jeremy Lyman - are you launching a run that covers a range of asset partitions (rather than a single partition?) The default IO manager currently doesn't support this kind of run, so you would need to write your own IO manager or use one of the DB IO managers for this to work. Here's an issue where we're tracking this: https://github.com/dagster-io/dagster/issues/10490

Jeremy Lyman

04/04/2023, 3:02 AM

Thank you. I was already playing around with IO Managers so this will give me all the more reason to explore. 👍

Open in Slack

Previous Next