Mycchaka Kleinbort
12/06/2022, 12:33 PMmostActiveUserId
that run on a daily schedule (pulls the latest user activity data from snowflake, returns the userId of the most active user - call this ). Most days the most active user is the same as yesterday. In this specific scenario, I only want to mark the downstream tasks as stale if the mostActiveUserId
asset value has changed.owen
12/06/2022, 9:57 PMobservable_source_asset
. Documentation for this feature is actually rolling out today, but for now I'll just send a link to the source, and (example in a test)
The basic idea is that, by default, it's assumed that when you run the most upstream asset, the data that it's reading from will have changed, but as you not this is not always the case. The observable source asset can go out and inspect the state of the source and return a version. Then, only if the version changes will downstream assets be considered stale. I think for your case, you can do something like return the most recent record id or timestamp etc.