https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
e

Ethan Fuerst

08/10/2023, 9:30 PM
hey all, getting an error when setting up dagster locally for a dbt project. running
dagster-dbt project scaffold --project-name local_dagster
from the setup guide and getting a
FileNotFoundError
because my
profiles.yml
is not in the same directory as my
dbt_project.yml
. as dbt recommends, I have it in a directory in my top level directory. is there a way I can specify a
profiles.yml
location on initialization of a new dagster project?
r

rex

08/10/2023, 10:12 PM
I can change this behavior so that it doesn’t error on a missing
profiles.yml
. The reason that we expect the
profiles.yml
to be in the dbt project directory is because you need a durable reference to the
profiles.yml
when you use Dagster + dbt on an external platform (e.g. K8s, ECS, Docker compose, Dagster Cloud). Otherwise, dbt will be unable to connect to the data warehouse, since it will not have access to your personal machine’s
~/.dbt
, but only your Dagster code and your dbt project. The reason why dbt recommends
~/.dbt/
is because they take care of
profiles.yml
for you in dbt Cloud. But when deploying
dbt Core
in an open source capacity, you will need to manage
profiles.yml
yourself.
e

Ethan Fuerst

08/10/2023, 10:21 PM
gotcha, makes sense. I keep my profiles file in
~/.dbt/
because I work on multiple dbt projects in different directories that connect to different databases
I’m hoping to run dagster locally on top of that project but reference the profiles file in my root directory
r

rex

08/10/2023, 10:33 PM
Got it. I believe the integration already does this. We invoke dbt using a subprocess, and so it looks for the profiles in your root directory if it can’t find one in dbt project. The part that you’re stuck on is the scaffold command. For now, you could just add a dummy profiles.yml in your dbt project dir so the scaffold works and creates a sample code location. That way, you can proceed.
👍 1
🙏 1
I can make a change upstream is make the profiles.yml optional when scaffolding a project. Thanks for the feedback
e

Ethan Fuerst

08/10/2023, 10:38 PM
great, thank you!
77 Views