Jason
12/12/2022, 7:01 PMFreshnessPolicy
& build_asset_reconciliation_sensor
as outlined apply to dbt Cloud assets? I'm aware that if you select individual assets to materialize, Dagster will override the dbt cloud job with that selection, so I'm wondering Dagster takes the same approach with freshness and reconciliation.
E.g. a non-dbt cloud asset, A, is stale and dbt cloud asset, B, is dependent on A. Asset B has dbt children C & D but dbt cloud job also has E, F and G which has no dependencies). Will the reconciliation sensor and FreshnessPolicy work and result in Dagster triggering a dbt build for only B, C and D?
https://docs.dagster.io/guides/dagster/scheduling-assetsowen
12/12/2022, 7:10 PMJason
12/12/2022, 7:21 PMload_assets_from_dbt_cloud_job
Example:
@asset(freshness_policy=FreshnessPolicy(maximum_lag_minutes=20))
owen
12/12/2022, 7:42 PMload_assets_from...
command, which will allow you to either supply a constant freshness policy (which will be applied to all loaded models), or a function that takes in a dictionary of node metadata and returns a freshness policy (i.e. lambda node_info: FreshnessPolicy(maximum_lag_minutes=20) if node_info["name"] == "model_i_care_about" else None
)Jason
12/12/2022, 7:43 PMJonathan Neo
01/02/2023, 7:43 AMload_assets_from_dbt_project
too?load_assets_from...
, is there a way to do it by applying freshnessPolicy directly on the assets returned from dbt_assets = load_assets_from_dbt_project(…)
?1.1.7
.
To enable freshness policy for a selected dbt model, you’ll just have to add the following line of config to your dbt model:
# file: my_model.sql
{{
config(
dagster_freshness_policy={"maximum_lag_minutes": 60, "cron_schedule": "0 9 * * *"}
)
}}
select * from {{ ref('my_table') }}
More here: https://github.com/dagster-io/dagster/issues/10924