Hello I have recently switched to using Dagster Cloud from S dagster #ask-community

Hello, I have recently switched to using Dagster ...

Don

05/23/2023, 1:25 PM

Hello, I have recently switched to using Dagster Cloud from Standard. Previously I was using the dbt_cli_resource to run dbt. I had configured it to find the profiles.yml and the dbt source on the local file system like so:

Copy code

"dbt": dbt_cli_resource.configured(
        {
            "project_dir": '../relative/path/dbt',
            "profiles_dir": '../relative/path'
        }
    )

Now that I'm running on cloud I'm seeing the following error:

Copy code

Encountered an error:
Runtime Error
fatal: Invalid --project-dir flag. Not a dbt project. Missing dbt_project.yml file

It seems like the dbt folder is not being packaged with my dagster source and deployed to cloud where it can run. Is there a way to deploy the dbt with the dagster code, or do I need to run dbt with dbt-cloud? Thanks, in advance

owen

05/23/2023, 6:02 PM

hi @Don! this is definitely supported, but you'll want to use absolute paths rather than relative paths, as we don't make any guarantees on which working directory a command will be invoked from. The general pattern we recommend is using file_relative_path, which will turn your relative path into an absolute one. see: https://docs.dagster.io/integrations/dbt/using-dbt-with-dagster/part-three#step-1-create-upstream-dagster-assets

Don

05/24/2023, 3:54 PM

Thanks, I'll give that a try

Don

05/24/2023, 5:21 PM

It looks like my example was wrong and I was already using file_relative_path

Copy code

DBT_PROJECT_DIR = file_relative_path(__file__, "../../dbt")
DBT_PROFILES_DIR = file_relative_path(__file__, "../..")

Don

05/24/2023, 5:21 PM

however, my dbt project lives at the same level as my dagster source. so it's like this project -> dbt -> my_dagster -> config job_config.py (this is where the file_relative_path is being used) repository.py dagster_cloud.yaml

Don

05/24/2023, 5:22 PM

my dagster_cloud.yaml looks like this

Copy code

locations:
  - location_name: my_dagster
    code_source:
      package_name: my_dagster

Don

05/24/2023, 5:22 PM

I am guessing that the dbt code is not getting packaged with my dagster code

Don

05/24/2023, 5:22 PM

is there a way to include the whole project in my cloud package or should I move the dbt code into my_dagster?

Don

05/24/2023, 5:27 PM

@owen Thanks for your help, I updated with more clarification

daniel

05/24/2023, 5:38 PM

Hey Don - does this discussion answer the question about how to include the dbt project? https://github.com/dagster-io/dagster/discussions/13767

Don

05/24/2023, 5:41 PM

@daniel, thanks. I will give this a try. So basically, the dbt should be inside the dagster folder, but it can either be done in the repo or during CI/CD

2 Views

Open in Slack

Previous Next