https://dagster.io/ logo
#ask-community
Title
# ask-community
m

Manish Khatri

07/10/2023, 2:38 PM
Hi wave anim we are currently experimenting with Dagster’s
FreshnessPolicy
and
AutoMaterializationPolicy
and so far it seems pretty cool that it can auto run a dbt model outside of a traditional “pipeline” 😎 However, before we consider moving away from the traditional pipeline, we would like to know if there’s a way that we can also have `dbt test`s running for each model that is materialised? Our current traditional pipeline uses a custom op that runs a
dbt run
and
dbt test
in sequence. The purpose of this is because we want to surface data quality issues as early as possible. Is there a way this can be done in the
AutoMaterliazation
world with the dbt Assets we generated from
load_assets_from_dbt_project(…)
?
t

Tim Castillo

07/10/2023, 2:58 PM
Would the use_build_command work for you? If not, completely fair game. the limitations with it are part of why we're making changes to our dbt integration to give users more control (so you can explicitly run
["run", "test"]
per model).
m

Manish Khatri

07/10/2023, 3:32 PM
I’m not aware of the side effect of using the
use_build_command
. If we did do this, can we still achieve the ability to do a dbt
run + test
? From the link you shared for your new dbt integration, it suggests that the
run + test
can only be achieved with low effort by utilising Example Number 4.
t

Tim Castillo

07/10/2023, 3:34 PM
It would run
dbt build
rather than just run, which is a utility for dbt seed, snapshot, run, test. and yes.
m

Manish Khatri

07/10/2023, 4:21 PM
Ah I see. Thanks for that Tim! That’s very helpful. Great food for thought moving forwards 🙂
m

Muhammad Jarir Kanji

07/11/2023, 4:02 PM
@Manish Khatri I'd be hesitant to swap
dbt run
with
dbt build
. I'm not a DBT snapshots expert so I'm happy to be corrected, but my understanding of snapshots is that there's only one schema for them, so unlike models (which can be saved to a different schema like
dbt_cloud
, during local development), every time you do
dbt snapshot
, you are updating your production snapshots. That means that if you do a
dbt build
while testing stuff locally, you would be updating your prod snapshots! That's a big (and mostly silent) side effect. As such, I've generally asked my team to avoid the
dbt build
command entirely. See here for more details: https://discourse.getdbt.com/t/using-dynamic-schemas-for-snapshots/1070.
m

Manish Khatri

07/11/2023, 4:07 PM
Hi Muhammad, thanks for the heads up. We don’t use any snapshots so luckily we can dodge that bullet. But I think the better approach for us would be to utilise the new dbt integration changes that Tim linked to. However, it’s something I’ll need to play around with before introducing it into our
main
branch as I’m doing some exploration on ways we can move from a traditional defined “pipeline” to leveraging Dagster’s unique automaterialization features.
👍 1
Hi @Tim Castillo i’m just re-visiting this as I’m looking at how this can be done in
1.4.4
and using this tutorial which leverages the
dagster-dbt project scaffold
to output the new way of loading assets in dbt as my playground. I am unable to see how I can tell dagster to to a
dbt run
and
dbt test
using the new way. The scaffolded code runs a
build
command, and Example 4 in the original link you shared isn’t valid in the final implementation (typing error). Any input on this would be greatly appreciated 🙂
t

Tim Castillo

08/04/2023, 3:23 PM
Hey Manish! Do you mind starting a new thread? Our automation doesn't work in threads, so it'd be easier for us to track this in a separate Slack message.
👍 1
2 Views