Hi. Is there a recommended way to dynamically build multiple software-defined assets based on an upstream asset (e.g. looping through the upstream data)? I haven't found a clear way to achieve that from the docs.

sounds like you could maybe do this with dynamic partitions? - everything would use the same asset key, but you would generate which partition keys are processed dynamically. Instead of an upstream asset, you might want to use a sensor.

Hmm, that doesn't seem ideal in our case. We need each of the dynamically generated assets to share the same partition key (the date). Furthermore, our IO manager materialise assets based the asset key (used as the SQL table name). This seemed like it could be a solution at first: https://github.com/dagster-io/dagster/issues/9559#issuecomment-1270755525 However, the first op

find_files_to_process

here is also triggering the IO manager, which is a problem in our case. We would only want to process with the IO Manager the final output after the mapping.

I see - I think the main problem is that we don’t really support dynamically generated asset keys super well within the product. You could dynamically process assets by using a graph backed asset with dynamic mappings, and then consolidate into a fixed set of asset keys - but as it stands not sure how you would model a dynamically generated set of asset keys based on upstream data :blob_sad:

Thanks for answering 🙂 Maybe it's something to keep in mind for future development. It feels, to me at least, that this is a valid use case in general.

Just an idea: maybe you could use MultiPartitionsDefinition. One partition for the date, and a partition for the different outputs. The second partition could be static if you already know the possible options, or dynamic otherwise

Multi Assets isn't really an option, the list is indeed dynamic based on the content on an upstream asset. Maybe I can explore your first option. I need to wrap my mind around how that would be implemented.