Is there a built in

IOManager

,

input_manager

or way to specify an

AssetIn

that provides a partitioned asset input as a generator that yields over the loaded inputs? Our partitioned assets are much too large to handle all at once in memory, but there are necessary steps where we need to index together into files or map/reduce data. It seems like the default manager will give you a

dict[partition_key] = partition_value

. Happy to write one, but want to make sure I"m not doing something idiomatically wrong.

Hi Alec - we don't currently have that, but it would be a reasonable thing to build

I'll be testing out an implementation this upcoming week. Happy to offer it available as a contrib.

Hi, @Alec Koumjian. I'm curious if you ever ended up implementing something. I have a similar use case.

I did, seems to be working well! I'll ask the team if we should open source it. We already have a few dagster utilities including another IOManager that handles working with files and directories rather than directly with objects in memory.