Hi, we’re struggling with implementing dbt-dagster for the first time. We’re using a setup based on the tutorials / example projects, something like:

if os.getenv("DAGSTER_DBT_PARSE_PROJECT_ON_LOAD"):
    dbt_parse_invocation = dbt.cli(["parse"], manifest={}).wait()
    dbt_manifest_path = dbt_parse_invocation.target_path.joinpath("manifest.json")
else:
    dbt_manifest_path = dbt_project_dir.joinpath("target", "manifest.json")@dbt_assets(manifest=dbt_manifest_path)
@dbt_assets(manifest=dbt_manifest_path)
def jaffle_shop_dbt_assets(context: OpExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build"], context=context).stream()

Basically, each run fails with:

No such file or directory: ./17954b5/manifest.json

My understanding is that DAGSTER_DBT_PARSE_PROJECT_ON_LOAD should cause the manifest to be generated. However, no manifest.json is generated and I believe that

dbt parse

doesn’t do this (only

dbt build

. Any ideas on what is missing here please?

what dbt version are you running?

dbt parse

should be producing a manifest starting in

dbt>=1.5

. If you’re running

dbt~=1.4

, then you’ll need to add

--write-manifest

if os.getenv("DAGSTER_DBT_PARSE_PROJECT_ON_LOAD"):
    dbt_parse_invocation = dbt.cli(["parse", "--write-manifest"], manifest={}).wait()
    dbt_manifest_path = dbt_parse_invocation.target_path.joinpath("manifest.json")
else:
    dbt_manifest_path = dbt_project_dir.joinpath("target", "manifest.json")


@dbt_assets(manifest=dbt_manifest_path)
def jaffle_shop_dbt_assets(context: OpExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build"], context=context).stream()

Thanks! It turns out we’re on dbt-core=1.1.0 due to wanting to use dbt-mysql. Enabling

--write-manifest

works in that it writes the manifest out to the

target

directory, but Dagster is looking for it in a directory generated per-run (eg

target/54f0243/

). It feels like this is config-related, or is it just the ancient version?

We're following guidance from https://docs.getdbt.com/docs/dbt-versions/core, and we're basically treating dbt-core<1.3 as EOL. We're using

DBT_TARGET_PATH

to override the target path so that each dbt invocation has a directory for its artifacts: https://docs.getdbt.com/reference/project-configs/target-path. Looks like this env var is only supported starting in 1.2.

Ah that makes sense, thank you

@rex You're saving everyone's lives over here! Thanks for the

--write-manifest

flag... I looked and looked for this functionality but just couldn't find it in dbt docs without knowing exactly what to look for 😅