If I use `dagster_aws.s3_pickle_io_manager` as the...
# ask-community
If I use
as the
, the return values of each operation will be stored in S3. Over a long period of usage, this might occupy a significant amount of storage. Is there a conventional solution from the community for this issue, or should I manage it myself?
You would have to manage this yourself. What I've done before is add a TTL via S3's Object Expiration feature. If the data you're passing between assets is temporary/ephemeral, another option would be to use the
instead of the S3 one. which would store the data in-memory if it fits.