https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
j

Jason Gofford

02/09/2024, 4:02 PM
If I use a
DbtCliResource
to compile a dbt project the artefacts including the manifest end up in a subfolder, e.g.,
8ec9053/manifest.json
. Is the subfoldering a Dagster thing, or a new dbt thing?
r

rex

02/09/2024, 4:05 PM
It's a Dagster thing. We put the artifacts in subfolders so that if you're running multiple commands, the artifacts don't clobber each other
j

Jason Gofford

02/09/2024, 4:06 PM
ah, good to know.
r

rex

02/09/2024, 4:06 PM
If you don't care about that, you can just hard code the target directory in which the artifacts are created when calling `DbtCliResource.cli(...)`: https://docs.dagster.io/_apidocs/libraries/dagster-dbt#dagster_dbt.DbtCliResource.cli
• target_path (Optional[Path]) – An explicit path to a target folder to use to store and retrieve dbt artifacts when running a dbt CLI command. If not provided, a unique target path will be generated.
j

Jason Gofford

02/09/2024, 4:08 PM
cool, that answers my next question. thanks rex!
r

rex

02/09/2024, 4:08 PM
yup!
Do you mind sharing your use case?
j

Jason Gofford

02/09/2024, 4:13 PM
It's similar to the dagster_dbt example. In a local dev environment using local k8s (minikube) I want dagster to trigger an accurate manifest to be generated when the code deployment is refreshed, hence the parse.
r

rex

02/09/2024, 4:16 PM
Ah got it. FWIW, we default to creating the manifest in a static directory now, when running
dagster-dbt project scaffold
. This was after some feedback from the community: https://github.com/dagster-io/dagster/pull/18501/files The relevant tidbit:
Copy code
if os.getenv("DAGSTER_DBT_PARSE_PROJECT_ON_LOAD"):
    dbt_manifest_path = (
        dbt.cli(
            ["--quiet", "parse"],
            target_path=Path("target"),
        )
        .wait()
        .target_path.joinpath("manifest.json")
    )
👍 1
j

Jason Gofford

02/09/2024, 4:26 PM
The real issue I've got (and how I spotted the foldering) is that I'm trying to integrate this into an existing dbt project with a typer CLI that allows the user to
dbt clone
into a dev database from a production manifest that's backed up on GCS. The CLI currently sits in the same project as the dagster assets and the
@dbt_asset()
decorator was forcing a compile on every command because the target path was changing.