https://dagster.io/ logo
t

Tobias Macey

04/21/2020, 4:34 PM
Working on the first stage of a pipeline and I believe I recall that Dagster doesn't currently have a means for running a fan-out/fan-in topology. The context being that I have a solid that generates a list of values and I want to execute the subsequent stages in sequence for each of those outputs. e.g. generate_list_of_strings fans out to do_something_with_string, do_something_else_with_string, etc. so that I have 5 solids running in parallel for item 1 in that list, and then repeat for each of the other strings.
m

max

04/21/2020, 4:46 PM
that's right, we don't yet have a map/collect abstraction
you may be able to get what you want by using
List[]
types
please feel free to weigh in on https://github.com/dagster-io/dagster/issues/462, where we are collecting rationales and ideas around this kind of behavior
t

Tobias Macey

04/21/2020, 4:57 PM
When you say to use
List[]
types, are you referring to generating that as an
Output
? Or something else?
m

max

04/21/2020, 5:01 PM
yes, each of your
do_something_with_string
solids could operate on a
List[str]
t

Tobias Macey

04/21/2020, 5:35 PM
Roger that, thanks. I'll give that a shot.