Prratek Ramchandani
05/02/2023, 1:00 PMINFORMATION_SCHEMA
query to check when those tables were updated, we could check multiple tables in one query and yield any run requests needed. there isn't a way to do this with observation functions and it feels like we're missing some notion of an observable multi source asset?
• observation functions being run as jobs feels a little unintuitive - i'm using them to tell me if the asset in question has changed and think of it as a background process that evaluates every few seconds along with typical sensor evaluations
i like the benefit of then being able to use auto materializations for downstream assets but would like the process of "observing" a source asset to feel more lightweightDagster Jarred
05/02/2023, 3:45 PMSean Lopp
05/02/2023, 3:53 PMsandy
05/02/2023, 4:55 PMsandy
05/02/2023, 4:57 PM@observable_source_asset(
auto_observe_frequency_minutes=60
)
def raw_data():
...
then, similar to with `AutoMaterializePolicy`s, the daemon would automatically kick off runs to observe the asset
would this work for you?Prratek Ramchandani
05/02/2023, 9:06 PMsandy
05/04/2023, 10:45 PM