I have a somewhat interesting FreshnessPolicy-esque use case which I think is hard to accommodate today, but which I think could be, and might be pretty general. https://ztf.uw.edu/alerts/public/ has many years of scientific data, bundled into tarballs per day. New tarballs are created each night. I am writing ETL jobs that scan those tarballs, pull out a subset of interesting features, and then march off to do other work with those features. Sometimes, old tarballs are modified. This can be because the archiving code was buggy, or there was an intermittent network failure, or whatever. MD5 checksums of the tarballs are in https://ztf.uw.edu/alerts/public/MD5SUMS. I would like to be able to say something like “Our asset is fresh if its MD5 checksum hasn’t changed since we last materialized it.” When the MD5SUMS file reports a difference (maybe polled once a day), I would like to kick off a new job.

@sean Seems like we could model this using observable source assets yeah? https://docs.dagster.io/guides/dagster/asset-versioning-and-caching#step-three-staleness-with-source-assets

Whoa, how did i not find this part of the docs. Yes

It currently does not work very well with partitioning, but that's on our near-term roadmap

Thanks for opening that. It would be nice if I could write a function that returns a map of partition-key -> logical version value.