https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
r

Robert Lawson

08/23/2023, 1:37 PM
Weirdly, if I use examples straight out of the docs at https://docs.dagster.io/_apidocs/libraries/dagster-dbt I get this error
Copy code
Param "command" is not a str. Got ['build', '--vars', '{"key": "value"}'] which is type <class 'list'>.
Copy code
dbt_build_invocation = dbt.cli(["build", "--vars", json.dumps(dbt_vars)], context=context)
    yield from dbt_build_invocation.stream()
r

rex

08/23/2023, 2:24 PM
Can you confirm that you're uisng
DbtCliResource
?
r

Robert Lawson

08/23/2023, 2:27 PM
Well, it's set up like this
Copy code
from dagster_dbt import (
    dbt_cli_resource,
    load_assets_from_dbt_project,
    dbt_assets,
    DbtCliResource,
    DagsterDbtTranslator

)
.........

def my_dbt_assets(context: OpExecutionContext, dbt: DbtCliResource):
    dbt_vars = {"key": "value"}

    <http://context.log.info|context.log.info>("Running my_dbt_assets")
    # dbt_build_invocation = dbt.cli(" ".join(["build", "--vars", json.dumps(dbt_vars)]), context=context)
    # yield from dbt_build_invocation.stream()
    run_command = " ".join(["run", "--vars", json.dumps(dbt_vars)])
    <http://context.log.info|context.log.info>(f"Running dbt command NO CONTEXT: {run_command}")
    # yield from dbt.cli(run_command, context=context).stream()

    yield from dbt.cli(run_command).stream()
    yield from dbt.cli("test").stream()
Following the definition for it takes me to
~/.local/lib/python3.10/site-packages/dagster_dbt/cli/resources.py
that being said I have made progress by concatenating the strings with
" ".join([the_commands])
and removing the
context
arg
other weirdness - dbt expacts yaml, but we're using
json.dumps
r

rex

08/23/2023, 2:31 PM
you're using
dbt_cli_resource
, not
DbtCliResource
.
dbt_cli_resource
is deprecated and is the old resource.
r

Robert Lawson

08/23/2023, 2:32 PM
acch! You are correct. Thanks.
still had this lingering
Copy code
# "dbt": dbt_cli_resource.configured(
                #     {
                #         "project_dir": DBT_PROJECT_PATH,
                #         "profiles_dir": DBT_PROFILES,
                #     }
                # ),
Thanks!
r

rex

08/23/2023, 2:46 PM
We can definitely make the experience better here in the future: I've raised https://github.com/dagster-io/dagster/issues/16035 so that we raise an error on this next time
r

Robert Lawson

08/23/2023, 2:47 PM
well to be fair it's on me for not checking the resource definition
r

rex

08/23/2023, 2:49 PM
haha fair, but the framework should have warned you, since you annotated it as
: DbtCliResource
🙂
r

Robert Lawson

08/23/2023, 2:49 PM
Well in that case it's on me for not submitting a PR
😂 1
but thank you
Mostly I'm glad to see it's easier to use variables when calling dbt run so this is nice
❤️ 1
r

rex

08/23/2023, 2:51 PM
love to hear that
r

Robert Lawson

08/23/2023, 3:10 PM
Would using this cause changes to dbt models not to be reflected if I reload the module? I.e. I update the code (I have it mounted inside a container) so
Copy code
docker compose exec dagit cat /opt/dagster/dagster_home/modules/batch_puller/batch_puller/models/batch_product_data/stg_batch_data.sql
... this shows up to date sql!
But then in the dagit ui I go to deployment and reload the module, and when I check the definition for stg_batch_data it's still out of date... maybe a result of having previously used
load_assets_from_dbt_project
?
r

rex

08/23/2023, 3:14 PM
There's a lot of problems with
load_assets_from_dbt_project
such as: • Random file contention • Incurs a ton of latency when compiling the manifest for large projects • Incurs repeated latency since it must be done for each run/code location/etc So we don't really recommend using it. But we do give you an escape hatch if you want to use this behavior for local development: https://dagster.slack.com/archives/C04CW71AGBW/p1690395671541379?thread_ts=1690388053.337689&amp;cid=C04CW71AGBW
r

Robert Lawson

08/23/2023, 3:15 PM
thanks again - I am not using load_assets_from_dbt_project but I also am trying to figure out how to get dagster to see that the dbt models have changed (or even been removed entirely)
r

rex

08/23/2023, 3:18 PM
got it: if you're not using
load_assets_from_dbt_project
, then you should recompile your dbt manifest using your CLI. e.g.
Copy code
dbt parse
Dagster uses this manifest to understand the state of your dbt project (which is why you pass it into
@dbt_assets
)
6 Views