Hey folks, new here 👋 I have a question about multiple dynamic op outputs: If I have a single dynamic output, I would decorate with e.g.

@op(out=DynamicOut(str))
def create_inputs():
    for i in range(3):
        yield DynamicOutput(value=f"value{i}", mapping_key=f"output{i}")

@graph
def process(inp: str):
    res = another_op(inp)
    ...

@job
def my_job():
    inputs = create_inputs()
    inputs.map(process)

now suppose I want to map to a graph that takes several input arguments, which might get passed to different ops inside

@graph
def process(inp1: str, inp2: int, inp3: float):
    res1 = another_op1(inp1)
    res2 = another_op2(inp2)
    res3 = another_op3(inp3)
    ...

how would I annotated and implement a

create_inputs

op that would yield "dynamic Tuple[str, int, float]s", s.t. I can still use

inputs = create_inputs()
    inputs.map(process)

in my job, together with correct annotations on the graph in dagit (with names and all)? Thanks a bunch 🙏

There may not be a way to achieve exactly what you desire. if you have the dynamic output be

Tuple[str, int, float]

you could just have

process

take the tuple and then add an

op

that splits it out in to multiple outputs.

@graph
def process(tuple_val: Tuple[str, int float]):
    inp1, inp2, inp3 = split(tuple_val)
    res1 = another_op1(inp1)
    res2 = another_op2(inp2)
    res3 = another_op3(inp3)
    ...

The

@graph

/

@job

functions dont execute at runtime, only at init time to build the dependency structure, so data manipulation has to happen in an

@op

I see, thanks for the quick reply! I might go with the splitting op, thanks for the advice!

omg I think this thread solved something that I've been banging my head against the wall about

@alex just as a follow up, for python builtin or multiprocessing, besides the

map

function, that passes single parameters unaltered to the eval function, there is also starmap which passes unpacked tuples to the eval function to be able to use multiple input parameters. Would it be worthwhile to consider adding a

starmap

method to

DynamicOutput

?

Hmm, it would likely be a big lift to support something like that. The data values that are passed between ops are opaque to the orchestration process and subject to a few user pluggable systems like

IOManager

. I encourage you to file an issue if you think it would be useful. Including some context on use case / motivation always helpful.

I see. Thanks, I'll consider it 🙂

@Dagster Bot discussion Multiple dynamic op outputs

Question in the thread has been surfaced to GitHub Discussions for future discoverability: https://github.com/dagster-io/dagster/discussions/8485