Hi there — I’ve got two assets, one depending on t...
# ask-community
j
Hi there — I’ve got two assets, one depending on the other, with the same (static) partitioning, and I’d like some help building a sensor to update them. I can always identify when a given partition of the downstream asset needs to be updated, but I’d like to only update the upstream asset if it has never been materialized before or if it is a certain age old (The upstream asset is only useful to compute the downstream one, so it’s wasteful for it to be always kept up to date across all partitions). I’ve read docs on FreshnessPolicy-based reconciliation and on asset caching, but I’m having a hard time adapting either (or both?) of these APIs to my use case. The reconciliation sensor method seems like it would always (eagerly) keep the upstream asset up to date, but I would like this process to be lazy, only materializing a given partition of the upstream asset if my sensor identifies that the same partition downstream also needs to be updated. The asset caching method, on the other hand, seems like I’d have to be able to compute a data version based on the time since last materialization of the asset, but I don’t see an API with which the asset could get its most recent materialization timestamp. Am I going about this the right way, or is there another path I should be taking for this?
Well, I’ve been made aware now that our requirements are less strict than I thought they were, so I’m gonna be able to get away with always materializing both the upstream and downstream together when the downstream one needs an update, which is much easier to do. I’m still curious about how one might “memoize” the upstream asset like I asked, but now it’s just a technical interest and not a necessity!