Darija Viaznikova
06/17/2023, 4:27 PM{{
config(
materialized='incremental',
unique_key='id',
incremental_strategy='delete+insert',
dagster_auto_materialize_policy={'type': 'lazy'},
dagster_freshness_policy={'cron_schedule':'0 19 * * *', 'maximum_lag_minutes': 24*60}
)
}}
As I understand, with the current setup this asset should be updated by 19h. However, instead, this asset’s update is launched exactly at 19h every day. It takes hours to update, so the asset isn’t ready on time. Am I doing something wrong? The downstream assets do not have any freshness policies.Darija Viaznikova
06/20/2023, 3:57 PMowen
06/20/2023, 10:23 PMdagster_freshness_policy={'cron_schedule':'0 19 * * *', 'maximum_lag_minutes': 3*60}
where 3*60
represents the maximum number of minutes that you'd expect the computation to take (I just guessed at 3 hours). Right now, the freshness policy you have is telling Dagster "By 19:00 every day, make sure the data in this asset isn't more than 1 day old", so that run you're observing in that screenshot is actually intended to make your asset up to date for tomorrow's checkDarija Viaznikova
06/21/2023, 7:26 AM