https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
g

Guy McCombe

07/26/2023, 3:10 PM
I'm having some trouble understanding this new v2 dbt integration. I'm trying to use the
@dbt_assets
decorator to be able to specify a target and a selection, but I can't get past this error message - any tips? (error & code in replies)
error:
code:
Copy code
from tennis.constants import dbt_config

manifest_path = Path(dbt_config["project_dir"]) / "target" / "manifest.json"
assert manifest_path.exists()

@dbt_assets(
        manifest=manifest_path,
        select="tag:atp",
)
def atp_assets(context: OpExecutionContext, dbt_atp: DbtCliResource):
    yield from dbt_atp.cli(["build"], context=context).stream()
Copy code
atp_cli_resource = DbtCliResource(
    project_dir=dbt_config["project_dir"],
    target="atp",
)
Some bonus logs, incl. the dbt command run
r

rex

07/26/2023, 3:15 PM
You can see the full CLI logs if you switch to the compute log view
e.g. this toggle in the UI
g

Guy McCombe

07/26/2023, 3:19 PM
ah thanks very much facepalm
looks like it's an issue with the relative path to my duckdb db since dagster is running from within the dbt project instead of the root directory keanu thanks
All daggy success now, thanks again for the pointer 😁
🚀 1
j

Julius

07/26/2023, 3:40 PM
I got the same error and here my logs
image.png
g

Guy McCombe

07/26/2023, 3:41 PM
Looks like you might need to configure your dbt profiles path. Where are they stored at the moment?
j

Julius

07/26/2023, 3:42 PM
Copy code
DBT_PROJECT_PATH = file_relative_path(__file__, "dbt")
DBT_PROFILES = file_relative_path(__file__, "dbt/config")
here
I work fine on my local, but I got this error when using Cloud version
g

Guy McCombe

07/26/2023, 3:44 PM
it's pretty hacky but you should be able to append the --profiles-dir argument to the cli command likes this:
Copy code
@dbt_assets(...)
def atp_assets(context: OpExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build", "--profiles-dir", DBT_PROFILES], context=context).stream()
❤️ 1
or, you can ensure that when your cloud deployment is getting built, you instead copy your profiles to /root/.dbt/
j

Julius

07/26/2023, 3:45 PM
ok, I will try that!
g

Guy McCombe

07/26/2023, 3:45 PM
best of luck 😁
r

rex

07/26/2023, 4:04 PM
There are a couple of solutions for this problem: (1) Add
--profiles-dir
as config on the `DbtCliResource`:
Copy code
atp_cli_resource = DbtCliResource(
    project_dir=dbt_config["project_dir"],
    global_config_flags=["--profiles-dir", DBT_PROFILES],
    target="atp",
)
(2) Currently your
profiles.yml
is in
dbt/config
. If you put your it in the root of your dbt project, in
dbt/
then this should work without needing to do (1)
j

Julius

07/26/2023, 4:11 PM
Copy code
DBT_PROJECT_PATH = file_relative_path(__file__, "dbt")
DBT_PROFILES = file_relative_path(__file__, "dbt/config")
resources = {
    "dbt": DbtCliResource(
        project_dir=DBT_PROJECT_PATH,
        profiles_dir=DBT_PROFILES,
        global_config_flags=["--no-use-colors"],
        profile="gnm_dwh",
        target="dev",
    ),
}
I'm already do this, so it different from global_config_flags and global_config_flags. Its working on my local but I got that error when using Cloud version
@Guy McCombe thanks again! Its working!
g

Guy McCombe

07/28/2023, 5:43 PM
fyi, @rex, I get
Usage: dbt [OPTIONS] COMMAND [ARGS]...
Try 'dbt -h' for help. Error: No such option: --profiles-dir
when specifying the profiles dir in
global_config_flags
r

rex

07/28/2023, 9:14 PM
Ah sorry, my bad.
--profiles-dir
is not global config. • I have an open PR in dbt core that is along this same vein: https://github.com/dbt-labs/dbt-core/pull/7920. Should probably try to get that in and make
--profile-dir
global config as well. • If you don’t want to specific
--profiles-dir
for every single command that you invoke, you could probably just set
DBT_PROFILE_DIR
as an env var to replicate global config: https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles#2-use-the-dbt_profiles_dir-environ[…]able-to-change-the-default-location