王昊
05/16/2023, 9:12 AMdagster_aws.s3_pickle_io_manager
as the io_manager
, the return values of each operation will be stored in S3. Over a long period of usage, this might occupy a significant amount of storage. Is there a conventional solution from the community for this issue, or should I manage it myself?Tim Castillo
05/16/2023, 3:05 PMmem_io_manager
instead of the S3 one. which would store the data in-memory if it fits.