https://dagster.io/ logo
#announcements
Title
# announcements
k

Kevin

09/26/2019, 10:23 AM
Has there ever been a follow-up to https://github.com/dagster-io/dagster/pull/699? What's your current view on the issue (fanning in and out of subgraphs / handling Sequence[T])?
a

alex

09/26/2019, 3:16 PM
Still an area of the system that is not fully fleshed out yet. We can handle fan in pretty well , but haven't added anything nice for doing fan out. In the system as it stands today - the best thing to do for fan out is a fixed width division of work.
I would be curious to hear more about the details of the problem you are looking to solve if you have the time to share.
k

Kevin

09/26/2019, 4:57 PM
Sure. Basically, we have a list of texts that we process through a bunch of solids. In the end, a few of them are just maps over lists. i.e.
Copy code
@lambda_solid
def word_counts(xs):
    return [word_count(x) for x in xs]
so each of them does
[a] -> [b]
i'd prefer to define the solids in terms of
a -> b
and let the system take care of the lifting. So something like
Copy code
@pipeline
def mypipeline():
    xs: List[String] = extract_texts()
    scores = dagster_map(mycomposed, xs)
    load_scores(scores)
with something like:
Copy code
@composed_solid
def mycomposed(text: String) -> Score:
    tokens = tokeniz(text)
    word_counts = word_count(tokens)
    return scoring(word_counts)
does that make sense?
a

alex

09/26/2019, 5:03 PM
Definitely makes sense, thanks for the concise example.
k

Kevin

09/26/2019, 5:10 PM
Thanks for looking into it 🙂 It probably doesn't get said often enough: dagster is already really awesome! Since we started trying it out at my company, I miss Flume (https://storage.googleapis.com/pub-tools-public-publication-data/pdf/35650.pdf) much less
❤️ 1
a

alex

09/26/2019, 5:20 PM
We appreciated the kind words 😄 Keep the feedback/bug reports/ideas coming!
👍 1