Hi all I ve mostly been using ```return Output value metadat dagster #ask-community

Hi all, I've mostly been using ```return Output(va...

moritz

04/10/2023, 11:37 AM

Hi all, I've mostly been using

Copy code

return Output(value, metadata={...})

for my assets and I'm very happy with that. Now, I wanted to use the same for a dynamic graph. I cannot use the same pattern or I get an

AttributeError: 'InvokedNodeOutputHandle' object has no attribute

because I'm expecting it to be a pandas DataFrame but it's not. The result is the output of running

pd.concat

on the collected dynamic value. Any ideas?

Charles

04/10/2023, 11:40 AM

I'm also using a dynamic output

-> DynamicOutput[dict]

and returning a

dict

with all the information needed. I will try your suggestion of adding metadata as part of

Output

Charles

04/10/2023, 11:42 AM

Copy code

return Output(
        value=output_df,
        metadata={
            "schema": "some schema",
             "table": "some table"
        }
    )

In my case the above still returns `None`inside my io_manager.

moritz

04/10/2023, 11:43 AM

Yes, I don't think it's a solution to your problem. I think you need to configure your I/O manager.

claire

04/10/2023, 10:21 PM

Hi Moritz. Not sure how you're running into the error above, but the following worked locally for me:

Copy code

@op(out=DynamicOut())
def load_pieces():
    df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
    large_data = [df, df, df]
    for idx, piece in enumerate(large_data):
        yield DynamicOutput(piece, mapping_key=str(idx), metadata={"idx": idx})


@op
def concat_pieces(context, pieces: List[pd.DataFrame]):
    <http://context.log.info|context.log.info>(pd.concat(pieces))


@job
def dynamic_graph():
    pieces = load_pieces()
    concat_pieces(pieces.collect())

21 Views

Open in Slack

Previous Next