Howdy, folks! Newbie here. I’m crawling a basic paginated API with Dagster. Now I’m wondering if it makes sense to make every page a separate asset, and if yes, then how to do it. Basically I’d like to spawn new assets from another asset: e.g. fetch the total page count, then spawn an asset for each page in parallel, and once all of them have completed spawn another asset that is going to merge all the pages together. Wdyt?

Hi Dmitri. This sounds like a good use case for partitions. You could consider making each different page a partition of the same asset, so you could then have a downstream asset that read in all partitions and combines them.

Thanks so much for the pointer! Seems to be exactly what I need, will give it a try!