I wonder if someone might help point me in the right direction. I have on op that produces a list of strings. I'd like to setup another op that processes that list of strings in parallel but I'm having trouble understanding how to do this from the documentation. Something like this:

@op()
def op1():
    return list_of_my_strings

@op()
def op2(my_string):
    # Write string to db table

@graph()
def string_processor():
    string_list = op1()
    for my_string in string_list:
        op2(my_string)

where op2 is what I'd like to run in parallel. Apologies if my code is bad.

I think you’re after this: https://docs.dagster.io/concepts/ops-jobs-graphs/dynamic-graphs

@Ben Gatewood thanks very much for the guidance here. This looks to be the right path! Do you happen to know when using DynamicOutput do I have to wait for the iteration to complete before I can the parallel op? If my list is large parallel ops doesn't save as much as I'd hoped because it just transfers the time savings to building the dynamic output

I haven’t actually tested this but am fairly certain the dynamic op needs to fully complete first - I don’t think you can stream chunks out of it as they’re ready (if I understand the question correctly)

thanks @Ben Gatewood that was the question

No probs (happy to be corrected on that last point if someone knows different)

@chris hope you don't mind if I tag you here in this thread, but I saw you answer another question about dynamic ops so thought you might have insight into this question. In the case above, if the list I"m using to build my DynamicOutput is quite large (say over a million) is it possible to chunk so that some ops can start processing or do you have to wait for the full DynamicOutput to build before proceeding to the fan out?

Hey there - not possible to chunk as of now unfortunately. Waits for step calling the fan-out to complete before actually beginning fan out. If you had multi-layer dynamic ops, this would be possible, but alas we don’t support that yet

@chris Thanks for the help here. So just curious is this the only way to fan out ops at present. I should be a bit more clear here. Say I had a list of files and I have an op that processes those files. The only way to fan out on the op is to build the dynamic output first?