Hi wave anim we are currently experimenting with Dagster’s

FreshnessPolicy

and

AutoMaterializationPolicy

and so far it seems pretty cool that it can auto run a dbt model outside of a traditional “pipeline” 😎 However, before we consider moving away from the traditional pipeline, we would like to know if there’s a way that we can also have `dbt test`s running for each model that is materialised? Our current traditional pipeline uses a custom op that runs a

dbt run

and

dbt test

in sequence. The purpose of this is because we want to surface data quality issues as early as possible. Is there a way this can be done in the

AutoMaterliazation

world with the dbt Assets we generated from

load_assets_from_dbt_project(…)

?

Would the use_build_command work for you? If not, completely fair game. the limitations with it are part of why we're making changes to our dbt integration to give users more control (so you can explicitly run

["run", "test"]

per model).

I’m not aware of the side effect of using the

use_build_command

. If we did do this, can we still achieve the ability to do a dbt

run + test

? From the link you shared for your new dbt integration, it suggests that the

run + test

can only be achieved with low effort by utilising Example Number 4.

It would run

dbt build

rather than just run, which is a utility for dbt seed, snapshot, run, test. and yes.

Ah I see. Thanks for that Tim! That’s very helpful. Great food for thought moving forwards 🙂

@Manish Khatri I'd be hesitant to swap

dbt run

with

dbt build

. I'm not a DBT snapshots expert so I'm happy to be corrected, but my understanding of snapshots is that there's only one schema for them, so unlike models (which can be saved to a different schema like

dbt_cloud

, during local development), every time you do

dbt snapshot

, you are updating your production snapshots. That means that if you do a

dbt build

while testing stuff locally, you would be updating your prod snapshots! That's a big (and mostly silent) side effect. As such, I've generally asked my team to avoid the

dbt build

command entirely. See here for more details: https://discourse.getdbt.com/t/using-dynamic-schemas-for-snapshots/1070.

Hi Muhammad, thanks for the heads up. We don’t use any snapshots so luckily we can dodge that bullet. But I think the better approach for us would be to utilise the new dbt integration changes that Tim linked to. However, it’s something I’ll need to play around with before introducing it into our

main

branch as I’m doing some exploration on ways we can move from a traditional defined “pipeline” to leveraging Dagster’s unique automaterialization features.

Hey Manish! Do you mind starting a new thread? Our automation doesn't work in threads, so it'd be easier for us to track this in a separate Slack message.