https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Adam Ward

12/20/2022, 1:34 PM
Hello - I need to load information into a DB from x number of upstream assets. I thought I might be able to do this by using a multi-asset and passing some recursive function to dependencies that materialises the downstream assets once the parents are reconciled, but I can't seem to figure out how to do that without repeating logic. In other words, I need to create x number of assets given x number of dependencies. Anyone tried something like this before?
dagster bot responded by community 1
This might have some pointers for you
🙏🏻 1
s

sean

12/20/2022, 8:37 PM
Hi Adam, I don’t quite understand what you’re trying to do-- could you say a little more or post a simplified code example?
a

Adam Ward

12/21/2022, 2:39 PM
Hi @Sven Lito - That was a great resource! Thank you! Love the asset_factory approach and will probably use some version of this. thank you box @sean - I basically want to have a producer asset group and a consumer asset group. The producer pulls from one data store and serializes to S3. I then want the consumer assets (which should be dependent on their respective source assets) to pull from S3 and create a one-for-one reproduction of those tables in a different data store. Note: I know this is a basic migration right now but the idea is that many consumers would then be able to consume from a single producer, reducing upstream cycles. I was initially looking towards multi-assets for the producer but, thinking on the asset_factory approach, I could perhaps create another factory that recurses over the asset keys of the first factory and produces assets with "destination" versions of the source asset keys.... At least, I think that would work..... 🤔 Open to suggestions here too...
7 Views