hi, is there a way to create multiple assets from a single config? I want to pass a text in a config and split the text into sentences, then I want to generate a different downstream asset for every sentence, what is the best way to implement that?
emm I will refine the question, I need to materialize the assets dynamically at run time. not creating multiple definitions of an asset. may be I can use Partitions for it?
z
Zach
08/26/2023, 9:04 PM
Yeah seems like a dynamically-partitioned asset could work. Or if you don't necessarily need to track each sentence as an individual asset you could use a graph-backed asset with a fan-out via a dynamic graph. That's probably how I would do it.
p
Pavel Schudel
08/26/2023, 9:20 PM
yeah, but I want to create a downstream assets with the sentences, and I just can figure out how to do it, basically if I have a text=(s1, s2,s3) I want to create 3 image generation prompts (p1,p2,p3) and then 3 images (im1, im2, im3) and then aggregate them into a single context, so: text => (s1, s2, s3) => (p1, p2, p3) => (im1, im2, im3) => text_images.
and I am not sure partitions support that kind of operations.
z
Zach
08/26/2023, 9:25 PM
Sounds like a dynamic graph-backed asset would be much more straightforward. You could fan-out across your sentences in your first step, chain the image generation and image steps, then aggregate them in an op that handles the collected results of the chained fan-out. Alternatively you might be able to chain dynamically-partitioned assets with partition mappings. You can't really dynamically create assets at runtime as Dagster needs the assets to be defined before a graph is constructed