https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
d

Darija Viaznikova

06/17/2023, 4:27 PM
Hi team, I am trying to schedule my DBT assets with auto-materialize + freshness policy, however, it does not work as expected. Here is my current setup for upstream assets:
Copy code
{{
  config(
    materialized='incremental',
    unique_key='id',
    incremental_strategy='delete+insert',
    dagster_auto_materialize_policy={'type': 'lazy'},
    dagster_freshness_policy={'cron_schedule':'0 19 * * *', 'maximum_lag_minutes': 24*60}
  )
}}
As I understand, with the current setup this asset should be updated by 19h. However, instead, this asset’s update is launched exactly at 19h every day. It takes hours to update, so the asset isn’t ready on time. Am I doing something wrong? The downstream assets do not have any freshness policies.
I am adding what the UI shows. As you can see, the run was started at 19:00 although it should have been already refreshed by that time. Am I missing something?
o

owen

06/20/2023, 10:23 PM
hi @Darija Viaznikova! what you'd want here is something along the lines of
Copy code
dagster_freshness_policy={'cron_schedule':'0 19 * * *', 'maximum_lag_minutes': 3*60}
where
3*60
represents the maximum number of minutes that you'd expect the computation to take (I just guessed at 3 hours). Right now, the freshness policy you have is telling Dagster "By 19:00 every day, make sure the data in this asset isn't more than 1 day old", so that run you're observing in that screenshot is actually intended to make your asset up to date for tomorrow's check
d

Darija Viaznikova

06/21/2023, 7:26 AM
Ohh, got it, thank you!
3 Views