Hi there, I am running a dbt pipeline using Dagste...
# integration-dbt
m
Hi there, I am running a dbt pipeline using Dagster. I have a
profiles.yml
file with several targets, ie. prod, dev, stag, with local being specified as the default. And then in Dagster I have a configured resource so at run time I can specify a
TARGET
and a
PROFILE
to use from
profiles.yml
. The Dagster run only works when using
local
. If I try specify any of the other targets, then I get the following error about partial parsing. Does anyone have any insight into this? I cannot find much help online so far.
Unable to do partial parsing because config vars, config profile, or config target have changed
I have also tried adding the following to the
profiles.yml
file:
Copy code
config:
  partial_parse: false
But, it still doesn't work and can't seem to connect to any of the other targets?
This is how I've specified the configurable resource so that at run time we can specify dev, stag or prod
target
to connect to, but it only works if I specify `local`:
Copy code
from dagster import with_resources, configured, Field
from dagster_dbt import dbt_cli_resource, load_assets_from_dbt_project

from ..config import DBT_PROJECT_DIR, DBT_PROFILES_DIR

@configured(
        dbt_cli_resource, 
        config_schema={
            "profile": Field(str, description="Which profile to load in profiles.yml. Overrides setting in dbt_project.yml. ie. which database."),
            "target": Field(str, description="Which target to load for the given profile, ie. which schema.")
            }
        )
def custom_dbt_cli_resource(config):
    return {
        "profile": config["profile"],
        "target": config["target"],
        "project-dir": DBT_PROJECT_DIR,
        "profiles-dir": DBT_PROFILES_DIR
    }

dbt_assets = with_resources(
    load_assets_from_dbt_project(
        project_dir=DBT_PROJECT_DIR,
        node_info_to_group_fn=lambda _: "predict_product_dbt_assets",
        use_build_command=True),
    {
        "dbt": custom_dbt_cli_resource
    },
)
p
Hi Megan, I’m not sure about why you cannot run it in different environments, but I can tell you about the Warning message that you’re receiving from dbt because I’ve been having it too. This is because you’re changing the configuration on-the-fly, and there is nothing wrong about it (from what I know). In my case, I’m using partitions and the partitions are sending the date I want to run my models, that’s a variable in my
profiles.yml
that I’m overwriting. In your case, I assume it’s because you’re overwriting the profile and target variables.
m
Thanks, yes, I want to connect to a different target each time, namely
dev
,
stag
or
prod
.
p
Just as a quick test, did you try to run dbt directly sending the different targets ?
m
Mmm, yes that also doesn't work but a different error. It says:
Copy code
dbt build --target dev
09:09:22  Running with dbt=1.5.0
09:09:22  Encountered an error:
Runtime Error
  Credentials in profile "predict_product_health", target "dev" invalid: ['dbname'] is not of type 'string'
But, it works with the target set to local. I've tried enclosing the variables in quotes and still giving the same error, like this:
Copy code
dev:
      type: postgres
      threads: 4
      host: xxxx
      port: 5432
      user: postgres
      pass: xxxx
      dbname: "predict_product_health"
      schema: dev
The target for local looks like this and this works:
Copy code
local:
      type: postgres
      threads: 4
      host: localhost
      port: 5433
      user: postgres
      pass: password
      dbname: predict_product_health
      schema: dev
🤔 1
p
maybe a dbt Slack question… 🤔
m
Ok, thanks for you help!
👍 1
208 Views