https://dagster.io/ logo
#ask-community
Title
# ask-community
g

Guillaume Onfroy

04/24/2023, 5:41 PM
Hi. Is there a recommended way to dynamically build multiple software-defined assets based on an upstream asset (e.g. looping through the upstream data)? I haven't found a clear way to achieve that from the docs.
c

chris

04/25/2023, 5:28 PM
sounds like you could maybe do this with dynamic partitions? - everything would use the same asset key, but you would generate which partition keys are processed dynamically. Instead of an upstream asset, you might want to use a sensor.
g

Guillaume Onfroy

04/25/2023, 10:05 PM
Hmm, that doesn't seem ideal in our case. We need each of the dynamically generated assets to share the same partition key (the date). Furthermore, our IO manager materialise assets based the asset key (used as the SQL table name). This seemed like it could be a solution at first: https://github.com/dagster-io/dagster/issues/9559#issuecomment-1270755525 However, the first op
find_files_to_process
here is also triggering the IO manager, which is a problem in our case. We would only want to process with the IO Manager the final output after the mapping.
c

chris

04/25/2023, 10:54 PM
I see - I think the main problem is that we don’t really support dynamically generated asset keys super well within the product. You could dynamically process assets by using a graph backed asset with dynamic mappings, and then consolidate into a fixed set of asset keys - but as it stands not sure how you would model a dynamically generated set of asset keys based on upstream data blob sad
g

Guillaume Onfroy

04/26/2023, 7:44 AM
Thanks for answering 🙂 Maybe it's something to keep in mind for future development. It feels, to me at least, that this is a valid use case in general.
f

Félix Tremblay

04/26/2023, 2:18 PM
Just an idea: maybe you could use MultiPartitionsDefinition. One partition for the date, and a partition for the different outputs. The second partition could be static if you already know the possible options, or dynamic otherwise
👍 1
Another idea: If the list of possible options is finite, you could use Multi Assets
g

Guillaume Onfroy

04/26/2023, 2:25 PM
Multi Assets isn't really an option, the list is indeed dynamic based on the content on an upstream asset. Maybe I can explore your first option. I need to wrap my mind around how that would be implemented.
Thanks for the input 🙂
5 Views