hey all, getting an error when setting up dagster locally for a dbt project. running

dagster-dbt project scaffold --project-name local_dagster

from the setup guide and getting a

FileNotFoundError

because my

profiles.yml

is not in the same directory as my

dbt_project.yml

. as dbt recommends, I have it in a directory in my top level directory. is there a way I can specify a

profiles.yml

location on initialization of a new dagster project?

I can change this behavior so that it doesn’t error on a missing

profiles.yml

. The reason that we expect the

profiles.yml

to be in the dbt project directory is because you need a durable reference to the

profiles.yml

when you use Dagster + dbt on an external platform (e.g. K8s, ECS, Docker compose, Dagster Cloud). Otherwise, dbt will be unable to connect to the data warehouse, since it will not have access to your personal machine’s

~/.dbt

, but only your Dagster code and your dbt project. The reason why dbt recommends

~/.dbt/

is because they take care of

profiles.yml

for you in dbt Cloud. But when deploying

dbt Core

in an open source capacity, you will need to manage

profiles.yml

yourself.

gotcha, makes sense. I keep my profiles file in

~/.dbt/

because I work on multiple dbt projects in different directories that connect to different databases

Got it. I believe the integration already does this. We invoke dbt using a subprocess, and so it looks for the profiles in your root directory if it can’t find one in dbt project. The part that you’re stuck on is the scaffold command. For now, you could just add a dummy profiles.yml in your dbt project dir so the scaffold works and creates a sample code location. That way, you can proceed.

great, thank you!