Hi there — I’ve got two assets, one depending on the other, with the same (static) partitioning, and I’d like some help building a sensor to update them. I can always identify when a given partition of the downstream asset needs to be updated, but I’d like to only update the upstream asset if it has never been materialized before or if it is a certain age old (The upstream asset is only useful to compute the downstream one, so it’s wasteful for it to be always kept up to date across all partitions). I’ve read docs on
FreshnessPolicy-based reconciliation and on
asset caching, but I’m having a hard time adapting either (or both?) of these APIs to my use case.
The reconciliation sensor method seems like it would always (eagerly) keep the upstream asset up to date, but I would like this process to be lazy, only materializing a given partition of the upstream asset if my sensor identifies that the same partition downstream also needs to be updated.
The asset caching method, on the other hand, seems like I’d have to be able to compute a data version based on the time since last materialization of the asset, but I don’t see an API with which the asset could get its most recent materialization timestamp.
Am I going about this the right way, or is there another path I should be taking for this?