Hello I ve previously used dagster s dbt cloud API and looki dagster #integration-dbt

Hello... I've previously used dagster's dbt-cloud ...

Jason

08/25/2023, 6:30 PM

Hello... I've previously used dagster's dbt-cloud API and looking to switch but we have a separate dbt github repo. Does Dagster only require the

manifest.json

file from dbt in order to not only visualize assets but also run/build them? So in theory, I can keep dbt separate (if i want to) and just sync the manifest file into the Dagster repo via CI?

dagster bot responded by community 1

Jacob Marcil

08/25/2023, 6:43 PM

Yes exactly, you can use something like this code. This will generate the manifest on the fly of you have

DAGSTER_DBT_PARSE_PROJECT_ON_LOAD

configured, or will select from an already generated

manifest

file if you don’t. Then you can configure you CI/CD like this in both repo. https://github.com/dagster-io/dagster/discussions/13767

Jason

08/25/2023, 6:55 PM

Ah thanks Jacob! Hadn't seen the GH discussion threads

Jason

08/25/2023, 8:07 PM

Just to circle back here after reading through above and messing around with the jaffle_shop demo... I do need both

manifest.json

and

dbt models folder

in the dagster project for this to work. The manifest file alone allows me to visualize the assets but when attempting to run models dbt finds no models to run.

Copy code

[WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 2 unused configuration paths:
- models.jaffle_shop.staging
- models.jaffle_shop

I thought having the

manifest

file alone would be enough to run dbt

Pedram Navid

08/26/2023, 11:15 PM

the manifest is required for rendering the UI and knowing what assets exist when loading your code, but the full dbt project with seeds and models and macros is required in order to run dbt, hope that helps

👍🏾 1

2 Views

Open in Slack

Previous Next