Has there ever been a follow-up to https://github.com/dagster-io/dagster/pull/699? What's your current view on the issue (fanning in and out of subgraphs / handling Sequence[T])?

Still an area of the system that is not fully fleshed out yet. We can handle fan in pretty well , but haven't added anything nice for doing fan out. In the system as it stands today - the best thing to do for fan out is a fixed width division of work.

Sure. Basically, we have a list of texts that we process through a bunch of solids. In the end, a few of them are just maps over lists. i.e.

@lambda_solid
def word_counts(xs):
    return [word_count(x) for x in xs]

so each of them does

[a] -> [b]

i'd prefer to define the solids in terms of

a -> b

and let the system take care of the lifting. So something like

@pipeline
def mypipeline():
    xs: List[String] = extract_texts()
    scores = dagster_map(mycomposed, xs)
    load_scores(scores)

with something like:

@composed_solid
def mycomposed(text: String) -> Score:
    tokens = tokeniz(text)
    word_counts = word_count(tokens)
    return scoring(word_counts)

does that make sense?

Definitely makes sense, thanks for the concise example.

Thanks for looking into it 🙂 It probably doesn't get said often enough: dagster is already really awesome! Since we started trying it out at my company, I miss Flume (https://storage.googleapis.com/pub-tools-public-publication-data/pdf/35650.pdf) much less

We appreciated the kind words 😄 Keep the feedback/bug reports/ideas coming!