https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
d

Djordje Velickovic

05/16/2023, 1:05 PM
Hello, I need help with dbt runs and model dependencies. We are building all mart models and one intermediate model (possibly more in the future) as tables in one dagster job (selected assets). Thing is, marts are depending on that intermediate model, but it’s running in parallel with them, so marts will not potentially have fresh data during that run. I have some thoughts about resolving this issue: • Creating separate job for building specific int models • Running whole DBT project (reason why we avoid this is that it’s much slower than just building marts, almost all intermediate models are views) • Should we switch to ephemeral? How will DBT assets behave if int models are ephemeral? • Something else? Does anyone have similar issue? Thanks folks!
🤖 1
f

Félix Tremblay

05/16/2023, 1:33 PM
The dependency between your dbt models should be respected. Have you double checked that the marts are indeed running "in parallel" with older data and it's not just a false impression?
d

Djordje Velickovic

05/16/2023, 1:38 PM
let me check on dummy example in local, since dagster is just executing
dbt run -s {list of models}
f

Félix Tremblay

05/16/2023, 1:43 PM
You could try defining the selection of models using the syntax for "everything downstream of" e.g. (
select="your_int_model+"
)
d

Djordje Velickovic

05/16/2023, 1:47 PM
Actually you’re correct, I executed the same in local, just on the smaller model set. I got false impression since it started run with bunch of models at first, but none of them were actual dependencies of that int model. DBT respects model dependencies even if specific models were selected.
Thanks!
f

Félix Tremblay

05/16/2023, 1:48 PM
Great!
You might like to check out this article to learn more about ways to select models, other than listing them
d

Djordje Velickovic

05/16/2023, 2:00 PM
Thank you very much! This is a great stuff
f

Félix Tremblay

05/16/2023, 2:29 PM
You're welcome!