https://dagster.io/ logo
Join Slack
Powered by
I'm having some trouble understanding this new v2 ...
# integration-dbt
g
I'm having some trouble understanding this new v2 dbt integration. I'm trying to use the 
@dbt_assets
decorator to be able to specify a target and a selection, but I can't get past this error message - any tips? (error & code in replies)
error:
code:
Copy code
from tennis.constants import dbt_config

manifest_path = Path(dbt_config["project_dir"]) / "target" / "manifest.json"
assert manifest_path.exists()

@dbt_assets(
        manifest=manifest_path,
        select="tag:atp",
)
def atp_assets(context: OpExecutionContext, dbt_atp: DbtCliResource):
    yield from dbt_atp.cli(["build"], context=context).stream()
Copy code
atp_cli_resource = DbtCliResource(
    project_dir=dbt_config["project_dir"],
    target="atp",
)
Some bonus logs, incl. the dbt command run
r
You can see the full CLI logs if you switch to the compute log view
e.g. this toggle in the UI
g
ah thanks very much facepalm
looks like it's an issue with the relative path to my duckdb db since dagster is running from within the dbt project instead of the root directory keanu thanks
All daggy success now, thanks again for the pointer 😁
🚀 1
j
I got the same error and here my logs
image.png
g
Looks like you might need to configure your dbt profiles path. Where are they stored at the moment?
j
Copy code
DBT_PROJECT_PATH = file_relative_path(__file__, "dbt")
DBT_PROFILES = file_relative_path(__file__, "dbt/config")
here
I work fine on my local, but I got this error when using Cloud version
g
it's pretty hacky but you should be able to append the --profiles-dir argument to the cli command likes this:
Copy code
@dbt_assets(...)
def atp_assets(context: OpExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build", "--profiles-dir", DBT_PROFILES], context=context).stream()
❤️ 1
or, you can ensure that when your cloud deployment is getting built, you instead copy your profiles to /root/.dbt/
j
ok, I will try that!
g
best of luck 😁
r
There are a couple of solutions for this problem: (1) Add 
--profiles-dir
as config on the `DbtCliResource`:
Copy code
atp_cli_resource = DbtCliResource(
    project_dir=dbt_config["project_dir"],
    global_config_flags=["--profiles-dir", DBT_PROFILES],
    target="atp",
)
(2) Currently your 
profiles.yml
is in 
dbt/config
. If you put your it in the root of your dbt project, in 
dbt/
then this should work without needing to do (1)
j
Copy code
DBT_PROJECT_PATH = file_relative_path(__file__, "dbt")
DBT_PROFILES = file_relative_path(__file__, "dbt/config")
resources = {
    "dbt": DbtCliResource(
        project_dir=DBT_PROJECT_PATH,
        profiles_dir=DBT_PROFILES,
        global_config_flags=["--no-use-colors"],
        profile="gnm_dwh",
        target="dev",
    ),
}
I'm already do this, so it different from global_config_flags and global_config_flags. Its working on my local but I got that error when using Cloud version
@Guy McCombe thanks again! Its working!
g
fyi, @rex, I get 
Usage: dbt [OPTIONS] COMMAND [ARGS]...
Try 'dbt -h' for help. Error: No such option: --profiles-dir
when specifying the profiles dir in 
global_config_flags
r
Ah sorry, my bad. 
--profiles-dir
is not global config. • I have an open PR in dbt core that is along this same vein: https://github.com/dbt-labs/dbt-core/pull/7920. Should probably try to get that in and make 
--profile-dir
global config as well. • If you don’t want to specific 
--profiles-dir
for every single command that you invoke, you could probably just set 
DBT_PROFILE_DIR
as an env var to replicate global config: https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles#2-use-the-dbt_profiles_dir-environ[…]able-to-change-the-default-location
Open in Slack
PreviousNext
Powered by