What do people use for an IO manager when they have time windowed partitions? I was trying to use the s3_pickle_io_manager, but it does not support time window partitions because the number of partitions != 1 according to the error.

Use the

UPathIOManager

, it supports s3 and multiple partitions

Cool, thank you. Will try it out

Hey, actually if you are just pickling stuff you can sue the default fs_io_manager with S3. It’s using the UPathIOManager under the hood. So you don’t need to write your own. Just make sure the credentials are set though environment variables or the ~/.aws/credentials file when passing the s://… base path to configure the fs_io_manager.

Interesting. That's good to know. We are using the fs_io_manager locally, but then switched to the s3 pickle manager. Did a variation where my class will inherit from the UPathIOManager

Yeah, it’s a little weird how the fs_io_managed supports remote storage but the old library io_managers are still present as duplicates, usually offering less functionality (regarding partitions stuff). Maybe something to think about @sandy