Hey all, I don't think I understand how Freshness ...
# ask-community
n
Hey all, I don't think I understand how Freshness Policies from upstream assest affect downstream assets (or vice versa). If there are two assets, A -> B, and A has a freshness policy of 10 minutes and B has a freshness policy of 1 minute, I would expect that A would run every 10 minutes and B would follow a minute after A gets loaded. Instead, I'm seeing that B will cause A to run every minute. So my question, how do I get the behaviour I expect?
The use case I'm thinking is this: Asset C depends on A and B. A takes a long time to run, so we only have it run once a week overnight and never ever want it to accidentally run again during the week. B runs every hour. C just needs to run once a day to load data for a dashboard by some time. How do I set this up? I though freshness policies would be the way, but judging from my test above, this would trigger the once-a-week asset to run every day
For reference, here's my test code:
Copy code
@asset(freshness_policy=FreshnessPolicy(maximum_lag_minutes=1))
def lag_1():
    return 1

@asset(freshness_policy=FreshnessPolicy(maximum_lag_minutes=5))
def lag_5():
    return 1

@asset(freshness_policy=FreshnessPolicy(maximum_lag_minutes=3))
def lag_3(lag_1, lag_5):
    return lag_1 + lag_5
And I see that lag_3 triggers lag_5 to run every 3 minutes
g
are you using the reconciliation sensor? If I understand correctly, the default behavior for that sensor is upstream updates causes downstream updates to follow. freshness policies town down that behavior to "it's fine to wait X minutes before I respond to an upstream materialization" in other words, I believe it's more useful when your upstream updates are more frequent than downstream. you seem to have the inverse case, upstream is "slower" and downstream is faster/more-frequent
n
Yeah, I have a sensor on all assets in this case
g
i think I came across this too when experimenting with reconciliation sensors (which I currently don't use in production)
but I believe the "correct" usage is to select downstream only, otherwise they all end up being updated on the faster schedule
n
Ohh, I see
And then put the upstream assets on a plain schedule, I suppose
g
the reconciliation-monitored assets will respond to upstream updates, which should be scheduled normally (unless they also depend on other upstreams)
as I understand it, freshness is basically "i dont need this asset to update as much as their upstreams"
n
Ah, yeah I'm still wrapping my head around some of the finer details of freshness policies
Thanks for your help!
g
np!
but since your upstream is "slower", your downstream reconciliated assets will end up being updated weekly too, only with a couple of minutes delay