Hi there I am running a dbt pipeline using Dagster I have a dagster #integration-dbt

Hi there, I am running a dbt pipeline using Dagste...

Megan Rogers

06/14/2023, 8:51 AM

Hi there, I am running a dbt pipeline using Dagster. I have a

profiles.yml

file with several targets, ie. prod, dev, stag, with local being specified as the default. And then in Dagster I have a configured resource so at run time I can specify a

TARGET

and a

PROFILE

to use from

profiles.yml

. The Dagster run only works when using

local

. If I try specify any of the other targets, then I get the following error about partial parsing. Does anyone have any insight into this? I cannot find much help online so far.

Unable to do partial parsing because config vars, config profile, or config target have changed

I have also tried adding the following to the

profiles.yml

file:

Copy code

config:
  partial_parse: false

But, it still doesn't work and can't seem to connect to any of the other targets?

Megan Rogers

06/14/2023, 9:03 AM

This is how I've specified the configurable resource so that at run time we can specify dev, stag or prod

target

to connect to, but it only works if I specify `local`:

Copy code

from dagster import with_resources, configured, Field
from dagster_dbt import dbt_cli_resource, load_assets_from_dbt_project

from ..config import DBT_PROJECT_DIR, DBT_PROFILES_DIR

@configured(
        dbt_cli_resource, 
        config_schema={
            "profile": Field(str, description="Which profile to load in profiles.yml. Overrides setting in dbt_project.yml. ie. which database."),
            "target": Field(str, description="Which target to load for the given profile, ie. which schema.")
            }
        )
def custom_dbt_cli_resource(config):
    return {
        "profile": config["profile"],
        "target": config["target"],
        "project-dir": DBT_PROJECT_DIR,
        "profiles-dir": DBT_PROFILES_DIR
    }

dbt_assets = with_resources(
    load_assets_from_dbt_project(
        project_dir=DBT_PROJECT_DIR,
        node_info_to_group_fn=lambda _: "predict_product_dbt_assets",
        use_build_command=True),
    {
        "dbt": custom_dbt_cli_resource
    },
)

Patricia Ayuso

06/14/2023, 9:03 AM

Hi Megan, I’m not sure about why you cannot run it in different environments, but I can tell you about the Warning message that you’re receiving from dbt because I’ve been having it too. This is because you’re changing the configuration on-the-fly, and there is nothing wrong about it (from what I know). In my case, I’m using partitions and the partitions are sending the date I want to run my models, that’s a variable in my

profiles.yml

that I’m overwriting. In your case, I assume it’s because you’re overwriting the profile and target variables.

Megan Rogers

06/14/2023, 9:04 AM

Thanks, yes, I want to connect to a different target each time, namely

dev

stag

prod

Patricia Ayuso

06/14/2023, 9:05 AM

Just as a quick test, did you try to run dbt directly sending the different targets ?

Megan Rogers

06/14/2023, 9:14 AM

Mmm, yes that also doesn't work but a different error. It says:

Copy code

dbt build --target dev
09:09:22  Running with dbt=1.5.0
09:09:22  Encountered an error:
Runtime Error
  Credentials in profile "predict_product_health", target "dev" invalid: ['dbname'] is not of type 'string'

But, it works with the target set to local. I've tried enclosing the variables in quotes and still giving the same error, like this:

Copy code

dev:
      type: postgres
      threads: 4
      host: xxxx
      port: 5432
      user: postgres
      pass: xxxx
      dbname: "predict_product_health"
      schema: dev

Megan Rogers

06/14/2023, 9:19 AM

The target for local looks like this and this works:

Copy code

local:
      type: postgres
      threads: 4
      host: localhost
      port: 5433
      user: postgres
      pass: password
      dbname: predict_product_health
      schema: dev

🤔 1

Patricia Ayuso

06/14/2023, 9:20 AM

maybe a dbt Slack question… 🤔

Megan Rogers

06/14/2023, 9:26 AM

Ok, thanks for you help!

👍 1

250 Views

Open in Slack

Previous Next