https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
b

Benjamin Sims

08/28/2023, 8:24 PM
Hi, we’re struggling with implementing dbt-dagster for the first time. We’re using a setup based on the tutorials / example projects, something like:
Copy code
if os.getenv("DAGSTER_DBT_PARSE_PROJECT_ON_LOAD"):
    dbt_parse_invocation = dbt.cli(["parse"], manifest={}).wait()
    dbt_manifest_path = dbt_parse_invocation.target_path.joinpath("manifest.json")
else:
    dbt_manifest_path = dbt_project_dir.joinpath("target", "manifest.json")@dbt_assets(manifest=dbt_manifest_path)
@dbt_assets(manifest=dbt_manifest_path)
def jaffle_shop_dbt_assets(context: OpExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build"], context=context).stream()
Basically, each run fails with:
No such file or directory: ./17954b5/manifest.json
My understanding is that DAGSTER_DBT_PARSE_PROJECT_ON_LOAD should cause the manifest to be generated. However, no manifest.json is generated and I believe that
dbt parse
doesn’t do this (only
dbt build
. Any ideas on what is missing here please?
r

rex

08/28/2023, 8:30 PM
what dbt version are you running?
dbt parse
should be producing a manifest starting in
dbt>=1.5
. If you’re running
dbt~=1.4
, then you’ll need to add
--write-manifest
Copy code
if os.getenv("DAGSTER_DBT_PARSE_PROJECT_ON_LOAD"):
    dbt_parse_invocation = dbt.cli(["parse", "--write-manifest"], manifest={}).wait()
    dbt_manifest_path = dbt_parse_invocation.target_path.joinpath("manifest.json")
else:
    dbt_manifest_path = dbt_project_dir.joinpath("target", "manifest.json")


@dbt_assets(manifest=dbt_manifest_path)
def jaffle_shop_dbt_assets(context: OpExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build"], context=context).stream()
🎉 1
ty spinny 1
b

Benjamin Sims

08/29/2023, 12:08 PM
Thanks! It turns out we’re on dbt-core=1.1.0 due to wanting to use dbt-mysql. Enabling
--write-manifest
works in that it writes the manifest out to the
target
directory, but Dagster is looking for it in a directory generated per-run (eg
target/54f0243/
). It feels like this is config-related, or is it just the ancient version?
r

rex

08/29/2023, 1:39 PM
We're following guidance from https://docs.getdbt.com/docs/dbt-versions/core, and we're basically treating dbt-core<1.3 as EOL. We're using
DBT_TARGET_PATH
to override the target path so that each dbt invocation has a directory for its artifacts: https://docs.getdbt.com/reference/project-configs/target-path. Looks like this env var is only supported starting in 1.2.
b

Benjamin Sims

08/29/2023, 1:40 PM
Ah that makes sense, thank you
n

Nicolas May

09/15/2023, 2:02 AM
@rex You're saving everyone's lives over here! Thanks for the
--write-manifest
flag... I looked and looked for this functionality but just couldn't find it in dbt docs without knowing exactly what to look for 😅
🫡 1
plus1 1
9 Views