We have a job that exports data every 30 seconds. We want to find if the data has been diffed. To do this I'm thinking of storing the results of the previous run in a new asset and then in the export asset diff the previous results with the new results, then update the new asset to store the current run as the previous run for the next one. However, I'm not sure if this is the best way to do this. Does someone have any thoughts/feedback on this approach?

Have you thought about using data versions? https://docs.dagster.io/guides/dagster/asset-versioning-and-caching#step-three-computing-your-own-data-versions

Would that be useful if we only want to store one version of previous results, the result immediately before the current one?

I believe there is an API to retrieve the existing data version before it is updated — cc @sean

Would you recommend using an asset to store this previous results data, given that we need to read/write it from an asset?