https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
j

Jason

12/12/2022, 7:01 PM
Hi...Does the
FreshnessPolicy
&
build_asset_reconciliation_sensor
as outlined apply to dbt Cloud assets? I'm aware that if you select individual assets to materialize, Dagster will override the dbt cloud job with that selection, so I'm wondering Dagster takes the same approach with freshness and reconciliation. E.g. a non-dbt cloud asset, A, is stale and dbt cloud asset, B, is dependent on A. Asset B has dbt children C & D but dbt cloud job also has E, F and G which has no dependencies). Will the reconciliation sensor and FreshnessPolicy work and result in Dagster triggering a dbt build for only B, C and D? https://docs.dagster.io/guides/dagster/scheduling-assets
o

owen

12/12/2022, 7:10 PM
Hi @Jason! because dbt Cloud assets now support subsetting, the sensor should work as you're expecting (i.e. only kicking off runs of the assets required to meet the freshness policy) 🙂
j

Jason

12/12/2022, 7:21 PM
Awesome! Thanks for the clarification
@owen Am I correct to assume that although the sensor would work via other criteria, I can't actually apply a FreshnessPolicy to dbt Cloud assets as there's doesn't seem to be support for this in
load_assets_from_dbt_cloud_job
Example:
Copy code
@asset(freshness_policy=FreshnessPolicy(maximum_lag_minutes=20))
o

owen

12/12/2022, 7:42 PM
ah yep that's right -- working on adding support for that either this week or next! the current plan is to add a new parameter to the
load_assets_from...
command, which will allow you to either supply a constant freshness policy (which will be applied to all loaded models), or a function that takes in a dictionary of node metadata and returns a freshness policy (i.e.
lambda node_info: FreshnessPolicy(maximum_lag_minutes=20) if node_info["name"] == "model_i_care_about" else None
)
would that work for your use case?
j

Jason

12/12/2022, 7:43 PM
Yup, that would work!
j

Jonathan Neo

01/02/2023, 7:43 AM
@owen would this be the same for
load_assets_from_dbt_project
too?
Also, whilst waiting for this feature to be available in
load_assets_from...
, is there a way to do it by applying freshnessPolicy directly on the assets returned from
dbt_assets = load_assets_from_dbt_project(…)
?
Heya, for anyone wondering, this feature has been implemented in dagster version
1.1.7
. To enable freshness policy for a selected dbt model, you’ll just have to add the following line of config to your dbt model:
Copy code
# file: my_model.sql
{{
    config(
        dagster_freshness_policy={"maximum_lag_minutes": 60, "cron_schedule": "0 9 * * *"}
    )
}}
select * from {{ ref('my_table') }}
More here: https://github.com/dagster-io/dagster/issues/10924
2 Views