i guess i'm way out in experimental territory here. But i have a usecase for FreshnessPolicy and observable_source_asset - and i dont think they are playing together as i expected.
We have a client that is supposed to upload files to our s3, but the upload task fails more often than not. As such i want an asset that is updated when a new version of the file is detected and i want downstream assets to show if the file has not been updated in a while (say 24 hours)
So my current setup is: raw_invoices_asset (observable_source_asset that returns timestamp from the file on s3) -> invoices_asset (uses raw_invoices_asset as non_argument_deps AND has a freshness_policy of X max min lag).
What im seeing is that FrehnessPolicy is triggering on every run of observable_source_asset - no matter if the LogicalVersion has changed or not - what i was hoping for was FreshnessPolicy only triggering if the LogicalVersion has changed. Am i just missunderstanding these functions?
01/20/2023, 10:20 PM
Hi @Espen Jütte! We're working towards making things work this way, but at the current moment that functionality is not there. Sorry about the confusion, your mental model is accurate, it's the code that's lagging behind here 🙂
01/20/2023, 10:21 PM
no problem, seems i was able to patch things up with some sensoring, coursors and freshness policy. It's working about as well as expected - except the code is a bit more messy
Looking forward to the code catching up- the asset and freshness stuff is super exciting (even though the documentation is a byzantine mess of information). I think i got the process working as i wanted: setup a sensor, keep a cursor of the last changed date, if last changed date is different kick off a run, the run just materializes the non existing asset. And to that asset i attached a freshness policy so i can complain if the asset is not there
but man - when dagster works it is just miles ahead. Keep up the fantastic work!