https://dagster.io/ logo
#announcements
Title
# announcements
a

Andrew Madonna

05/24/2019, 6:24 PM
Ok, new (sub) use case. Solids that produce solids? I have hundreds of thousands of files that need to be processed. A would normally have a workflow node batch the files as a list of paths, and emit a file processing node per file batch. Each file batch could then be processed in subsequent steps independently in parallel.