Hi people. Do you know if when using persistence storage for outputs it's possible to have some things not be saved forever? In a certain node I have a model that outputs a bunch of things, mostly are used only for subsequent calculations in the pipeline. On top of that some are heavy and we wouldn't like our storage to explode. Is there a way to handle this without deleting the runs?

also interested

An idea from @sashank was if using s3 to use a policy to remove buckets with certain lifetime. And if using local storage to use a cron to remove old jobs. Still it'd be nice to perhaps specify when certain outputs are ethereal, such that they save only in memory. This could complicate the whole data layer though.

we're having a distributed setup so in-memory is not a choice - we have to use s3. but it would be nice to have ethereal outputs in a sense that their lifetime is limited to a complete pipeline cycle. So they could all be purged after it succeeded

Hmmm, that makes a lot of sense. Then the idea would be to specify in some way when the output is ethereal so pipeline data can be cleaned up on pipeline success. I wonder how difficult is to implement this