Hi I have a dagster dbt project I was using the following ve dagster #integration-dbt

Hi, I have a dagster/dbt project. I was using the...

Ian Venter

05/03/2023, 12:08 PM

Hi, I have a dagster/dbt project. I was using the following versions of python packages using python version 3.10. Dagster starting up fine with no warnings. dbt-core==1.4.5 dbt-postgres==1.4.5 dagster==1.2.2 dagit==1.2.2 dagster-dbt==0.18.2 dagster_postgres==0.18.2 I've since upgraded my project to python version 3.11 and also upgraded the packages as follows: dbt-core==1.5.0 dbt-postgres==1.5.0 dagster==1.3.2 dagit==1.3.2 dagster-dbt==0.19.2 dagster_postgres==0.19.2 Since upgrading I now get warnings when I run 'dagster dev'. These below warnings are filling up the logs but things still seem to work. Is this an issue? 2023-05-03 215906 +1000 - dagster.builtin - WARNING - The selection criterion 'analyses' does not match any nodes 2023-05-03 215906 +1000 - dagster.builtin - WARNING - The selection criterion 'dbt_packages' does not match any nodes 2023-05-03 215906 +1000 - dagster.builtin - WARNING - The selection criterion 'dbt_project.yml' does not match any nodes 2023-05-03 215906 +1000 - dagster.builtin - WARNING - The selection criterion 'logs' does not match any nodes 2023-05-03 215906 +1000 - dagster.builtin - WARNING - The selection criterion 'macros' does not match any nodes 2023-05-03 215906 +1000 - dagster.builtin - WARNING - The selection criterion 'models' does not match any nodes 2023-05-03 215906 +1000 - dagster.builtin - WARNING - The selection criterion 'packages.yml' does not match any nodes 2023-05-03 215906 +1000 - dagster.builtin - WARNING - The selection criterion 'profiles.yml' does not match any nodes 2023-05-03 215906 +1000 - dagster.builtin - WARNING - The selection criterion 'README.md' does not match any nodes 2023-05-03 215906 +1000 - dagster.builtin - WARNING - The selection criterion 'seeds' does not match any nodes 2023-05-03 215906 +1000 - dagster.builtin - WARNING - The selection criterion 'snapshots' does not match any nodes 2023-05-03 215906 +1000 - dagster.builtin - WARNING - The selection criterion 'target' does not match any nodes 2023-05-03 215906 +1000 - dagster.builtin - WARNING - The selection criterion 'tests' does not match any nodes 2023-05-03 215906 +1000 - dagster.builtin - WARNING - No nodes selected! I load my dbt assets as follows:

Copy code

dbt_assets = load_assets_from_dbt_project(
    project_dir=DBT_PROJECT_PATH, profiles_dir=DBT_PROFILES,use_build_command=True,
)

rex

05/03/2023, 8:43 PM

These look like warnings coming from

dbt-core

. And although Python 3.11 is supported dbt-core, it still is a work in progress. Do you still run into these errors if you’re on

dagster-dbt==0.19.2

, but

dbt-core==1.4.5

Ian Venter

05/03/2023, 10:50 PM

Hi @rex, I've downgraded to dbt-core==1.4.5 and kept dagster-dbt==0.19.2 and the warnings go away. Does this mean I should refrain from using dbt-core==1.5.0?

👍🏽 1

rex

05/04/2023, 2:01 PM

Hey Ian — I’m not able to replicate your problem, and we’re using dbt 1.5.0 on our internal dbt project. I would suggest for you to use 1.4.5 for now, unless you can produce an example for us to replicate the problem.

Ian Venter

05/05/2023, 9:50 PM

Hi @owen , this is my original problem. After spotting Vinnie’s post I thought his issue may have been related to mine.

Ian Venter

05/06/2023, 11:27 AM

Hi @owen & @rex, My issue still persists although I think i may have narrowed it down a bit. Firstly to get my dagit environment to startup using dbt-core==1.5.0, i've had to make a change to the way I load the dbt assets. I was loading my dbt assets as follows:

Copy code

dbt_assets = load_assets_from_dbt_project(
    project_dir=DBT_PROJECT_PATH, profiles_dir=DBT_PROFILES,use_build_command=True,
)

I've had to change this to use the manifest as follows:

Copy code

dbt_assets = load_assets_from_dbt_manifest(
    manifest_json=json.load(open(os.path.join(DBT_PROJECT_PATH,"target","manifest.json"), encoding="utf-8")),
)

This change makes Dagit startup ok using dbt-core==1.5.0, and I can even materialize individual models via Dagit. However when I perform a 'Materialize All' via Dagit, it does not materialize my models and gives me a message that no ops were fired. When I downgrade to dbt-core==1.4.6 all works fine and I can perform a materialize all. I've isolated the command that dagster issues for dbt to run when I perform a materialize all, i'm showing it below for reference. For brevity i've just put a current directory '.' in place of the full directory location for the project-dir and profiles-dir

Copy code

dbt --no-use-colors --log-format json build --project-dir . --profiles-dir . --select *

When I run this command in DBT (taking dagster out of the equation) using dbt-core==1.4.6 it performs a materialization of all my assets Running the same command using dbt-core==1.5.0 presents the original warnings I called out when i first logged this issue

Copy code

{"data": {"spec_raw": "analyses"}, "info": {"category": "", "code": "M030", "extra": {}, "invocation_id": "79953600-1526-48ec-abad-b9962c2ad666", "level": "warn", "msg": "The selection criterion 'analyses' does not match any nodes", "name": "NoNodesForSelectionCriteria", "pid": 20916, "thread": "MainThread", "ts": "2023-05-06T11:38:23.209650Z"}}
{"data": {"spec_raw": "dbt_packages"}, "info": {"category": "", "code": "M030", "extra": {}, "invocation_id": "79953600-1526-48ec-abad-b9962c2ad666", "level": "warn", "msg": "The selection criterion 'dbt_packages' does not match any nodes", "name": "NoNodesForSelectionCriteria", "pid": 20916, "thread": "MainThread", "ts": "2023-05-06T11:38:23.210675Z"}}
{"data": {"spec_raw": "dbt_project.yml"}, "info": {"category": "", "code": "M030", "extra": {}, "invocation_id": "79953600-1526-48ec-abad-b9962c2ad666", "level": "warn", "msg": "The selection criterion 'dbt_project.yml' does not match any nodes", "name": "NoNodesForSelectionCriteria", "pid": 20916, "thread": "MainThread", "ts": "2023-05-06T11:38:23.212676Z"}}
{"data": {"spec_raw": "logs"}, "info": {"category": "", "code": "M030", "extra": {}, "invocation_id": "79953600-1526-48ec-abad-b9962c2ad666", "level": "warn", "msg": "The selection criterion 'logs' does not match any nodes", "name": "NoNodesForSelectionCriteria", "pid": 20916, "thread": "MainThread", "ts": "2023-05-06T11:38:23.213098Z"}}
{"data": {"spec_raw": "macros"}, "info": {"category": "", "code": "M030", "extra": {}, "invocation_id": "79953600-1526-48ec-abad-b9962c2ad666", "level": "warn", "msg": "The selection criterion 'macros' does not match any nodes", "name": "NoNodesForSelectionCriteria", "pid": 20916, "thread": "MainThread", "ts": "2023-05-06T11:38:23.214344Z"}}
{"data": {"spec_raw": "models"}, "info": {"category": "", "code": "M030", "extra": {}, "invocation_id": "79953600-1526-48ec-abad-b9962c2ad666", "level": "warn", "msg": "The selection criterion 'models' does not match any nodes", "name": "NoNodesForSelectionCriteria", "pid": 20916, "thread": "MainThread", "ts": "2023-05-06T11:38:23.215341Z"}}
{"data": {"spec_raw": "packages.yml"}, "info": {"category": "", "code": "M030", "extra": {}, "invocation_id": "79953600-1526-48ec-abad-b9962c2ad666", "level": "warn", "msg": "The selection criterion 'packages.yml' does not match any nodes", "name": "NoNodesForSelectionCriteria", "pid": 20916, "thread": "MainThread", "ts": "2023-05-06T11:38:23.216340Z"}}
{"data": {"spec_raw": "profiles.yml"}, "info": {"category": "", "code": "M030", "extra": {}, "invocation_id": "79953600-1526-48ec-abad-b9962c2ad666", "level": "warn", "msg": "The selection criterion 'profiles.yml' does not match any nodes", "name": "NoNodesForSelectionCriteria", "pid": 20916, "thread": "MainThread", "ts": "2023-05-06T11:38:23.217342Z"}}
{"data": {"spec_raw": "README.md"}, "info": {"category": "", "code": "M030", "extra": {}, "invocation_id": "79953600-1526-48ec-abad-b9962c2ad666", "level": "warn", "msg": "The selection criterion 'README.md' does not match any nodes", "name": "NoNodesForSelectionCriteria", "pid": 20916, "thread": "MainThread", "ts": "2023-05-06T11:38:23.217342Z"}}
{"data": {"spec_raw": "seeds"}, "info": {"category": "", "code": "M030", "extra": {}, "invocation_id": "79953600-1526-48ec-abad-b9962c2ad666", "level": "warn", "msg": "The selection criterion 'seeds' does not match any nodes", "name": "NoNodesForSelectionCriteria", "pid": 20916, "thread": "MainThread", "ts": "2023-05-06T11:38:23.218342Z"}}
{"data": {"spec_raw": "snapshots"}, "info": {"category": "", "code": "M030", "extra": {}, "invocation_id": "79953600-1526-48ec-abad-b9962c2ad666", "level": "warn", "msg": "The selection criterion 'snapshots' does not match any nodes", "name": "NoNodesForSelectionCriteria", "pid": 20916, "thread": "MainThread", "ts": "2023-05-06T11:38:23.219344Z"}}
{"data": {"spec_raw": "target"}, "info": {"category": "", "code": "M030", "extra": {}, "invocation_id": "79953600-1526-48ec-abad-b9962c2ad666", "level": "warn", "msg": "The selection criterion 'target' does not match any nodes", "name": "NoNodesForSelectionCriteria", "pid": 20916, "thread": "MainThread", "ts": "2023-05-06T11:38:23.220341Z"}}
{"data": {"spec_raw": "tests"}, "info": {"category": "", "code": "M030", "extra": {}, "invocation_id": "79953600-1526-48ec-abad-b9962c2ad666", "level": "warn", "msg": "The selection criterion 'tests' does not match any nodes", "name": "NoNodesForSelectionCriteria", "pid": 20916, "thread": "MainThread", "ts": "2023-05-06T11:38:23.221341Z"}}
{"data": {"msg": ""}, "info": {"category": "", "code": "Z017", "extra": {}, "invocation_id": "79953600-1526-48ec-abad-b9962c2ad666", "level": "info", "msg": "", "name": "Formatting", "pid": 20916, "thread": "MainThread", "ts": "2023-05-06T11:38:23.223344Z"}}

If i remove the '--select *' from the command and run it in DBT it works fine under dbt-core==1.5.0 and does what it's suppose to do and materializes all the models in my dbt project.

Copy code

dbt --no-use-colors --log-format json build --project-dir . --profiles-dir .

So in summary it seems like an issue on my environment with DBT and this is the closest i've come to isolating the issue. Is there a way I can load my dbt assets from dagster without it including the '--select *' in the command?

rex

05/08/2023, 1:26 AM

Looks like your environment might be interpreting

as an extended glob. This is why in the logs, you are seeing your file/directory names, like

tests

snapshots

README.md

, etc. You should look in disable that. We are using valid dbt syntax to select all models, so this is a configuration issue in your environment.

Ian Venter

05/08/2023, 3:08 AM

Thanks @rex, i've raised an issue with DBT. It seems odd to me that my project works fine with dbt-core==1.4.5 & 1.4.6 but that this issue surfaces with 1.5.0. https://github.com/dbt-labs/dbt-core/issues/7541

Ian Venter

05/08/2023, 11:52 PM

Hi @rex & @owen, The guys over at DBT have responded to the bug i raised with them about the 'dbt build --select *' issue. Please see their comments on the ticket. Things too note: 1. They are able to recreate the issue on dbt-core versions 1.4.6 as well as 1.5.0 2. They do not and never have supported 'dbt build --select ' and advise using 'dbt build --select fqn: source:*' I can successfully run 'dbt build --select fqn:*' on my environment that uses dbt-core==1.5.0 So my question becomes, is this a Dagster issue?

rex

05/09/2023, 5:15 PM

Looking at the docs, I think we assumed that

is equivalent to

fqn:*

! https://docs.getdbt.com/reference/node-selection/methods#the-file-or-fqn-method. We can change this default to be more explicit.

owen

05/09/2023, 7:26 PM

I looked into this a bit deeper, and I think it's incorrect that

was never supported, at least the way we were using it. It's true that, if you run

dbt build --select *

in the shell, that you'll get the observed behavior, but this is simply because the shell will automatically expand any

arguments. We're not invoking dbt via the shell (we're invoking it through a subprocess), so it's unclear why the star would be getting expanded. This is more equivalent to running

dbt build --select '*'

, which I can confirm does work as expected

owen

05/09/2023, 8:12 PM

@Ian Venter are you on windows? it seems that the

click

library (which

dbt-core

switched to in 1.5.0) will auto-expand glob patterns (source)

owen

05/09/2023, 8:19 PM

ah I see from the issue you opened that you are indeed on windows -- that seems to be the source of what you're seeing. I think this can be fixed if we wrap the select in quotes before passing it into the command, but for an immediate fix you can just call

load_assets_from...

with

select="fqn:*"

to dodge this behavior

Ian Venter

05/10/2023, 12:12 AM

Hi @owen, glad we got to the bottom of the issue. Thanks for your help.

rex

05/10/2023, 3:02 PM

This should resolve https://github.com/dagster-io/dagster/pull/14204

Ian Venter

05/12/2023, 10:52 PM

Thanks @rex

Kalyan Singh

01/15/2024, 8:49 PM

I am encountering errors during the materialization of dbt models in the Dagster UI. The specific error message is: "Error loading configuration: dbt_cloud.yml credentials file for dbt Cloud not found. Download your credentials file from dbt Cloud to

C:\Users\hp\.dbt

4-01-15 17:55:27 +0530 - dagster - ERROR - __ASSET_JOB - 51aa285e-44bd-4c24-8543-d4ce63ec7f11 - 28836 - demo_dbt_dbt_assets - STEP_FAILURE - Execution of step "demo_dbt_dbt_assets" failed. dagster_dbt.errors.DagsterDbtCliRuntimeError: The dbt CLI process failed with exit code 1. Check the stdout in the Dagster compute logs for the full information about the error." The versions of Dagster and dbt being used are as follows: dagster-airbyte==0.22.0 dagster-dbt==0.22.0 dagster-graphql==1.6.0 dagster-pipes==1.6.0 dagster-webserver==1.6.0 dbt-core==1.4.9 dbt-extractor==0.4.1 dbt-semantic-interfaces==0.4.2 dbt-sqlserver==1.4.3 I find myself in a challenging situation; could you please provide assistance?

549 Views

Open in Slack

Previous Next