https://dagster.io/ logo
#ask-community
Title
# ask-community
k

Katja D'haeyer

09/16/2022, 5:21 PM
Hello team! I have 1 op that extracts data from several sources and writes the result to a database. Via parameters I can reuse the same op, but extract data from like a 100 sources. Since each op gets an alias I have a bunch of that op running in parallel. After all those are done, I want to start another op. How can I detect that all are done? For another case I use Nothing dependencies so although an op doesn't return a result, I can create a chain of ops waiting for each other. But in the later case I have separated ops which I can call by name like this: av =update_pt(waiting_for=update_cf(waiting_for=ap)). I don't see how a similar approach can work for my 100 parallel ops so any help would be appreciated. Thanks!
z

Zach

09/16/2022, 5:33 PM
could you turn your parallel data extraction into a dynamic fan-out, then collect Nothing dependencies from all the extraction ops into a list that the downstream op takes as a single input?
k

Katja D'haeyer

09/20/2022, 7:47 AM
This seems to work out 🙂 thx!
2 Views