The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Working on the first stage of a pipeline and I believe I recall that Dagster doesn't currently have a means for running a fan-out/fan-in topology. The context being that I have a solid that generates a list of values and I want to execute the subsequent stages in sequence for each of those outputs. e.g. generate_list_of_strings fans out to do_something_with_string, do_something_else_with_string, etc. so that I have 5 solids running in parallel for item 1 in that list, and then repeat for each of the other strings.

that's right, we don't yet have a map/collect abstraction

you may be able to get what you want by using `List[]` types

please feel free to weigh in on <https://github.com/dagster-io/dagster/issues/462>, where we are collecting rationales and ideas around this kind of behavior

When you say to use `List[]` types, are you referring to generating that as an `Output`? Or something else?

yes, each of your `do_something_with_string` solids could operate on a `List[str]`

Roger that, thanks. I'll give that a shot.