https://dagster.io/ logo
#integration-dbt
Title
# integration-dbt
m

Manish Khatri

08/04/2023, 10:12 PM
Hi, question on migrating from
load_assets_from_dbt_project
to the new flexible way of building dbt assets from a dbt project using
@dbt_assets
and
DbtCliResource
like in this tutorial. Currently we use the following:
Copy code
load_assets_from_dbt_project(
    project_dir=DBT_PROJECT_DIR,
    profiles_dir=DBT_PROFILES_DIR,
    node_info_to_group_fn=get_dbt_group_name, -- prefixes the asset group name with "dbt_"
)
I’ve looked at the
1.4.14
docs and it’s not clear to me where we can specify an alternative
profiles.yml
path in the new way:
Copy code
@dbt_assets(manifest=dbt_manifest_path)
def jaffle_shop_dbt_assets(context: OpExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["run"], context=context).stream()
    yield from dbt.cli(["test"], context=context).stream()
With regards to the
node_info_to_group_fn
, is that now replaced with the kwarg of
dagster_dbt_translator
in
@dbt_assets(….)
? Where the value is a DagsterDbtTranslator object specifically like this example?
1
s

Sean Lopp

08/04/2023, 10:17 PM
Here is an example for group_fn https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/hooli_data_eng/assets/dbt_assets.py#L42 For profiles-dir, I believe the default is to look within the main dbt project dir (same as the dbt cli default), but if you want to pass in that arg to every invocation, here is an example of how you could write a lightweight extension of the dbtCliResource to do so: https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/hooli_data_eng/resources/dbt.py The project shows how to reference that resource and configure it as well
m

Manish Khatri

08/04/2023, 10:45 PM
Thanks for the links Sean. Looking at the 2nd link, it seems like I could also achieve this with the following…?
Copy code
@dbt_assets(manifest=dbt_manifest_path)
def jaffle_shop_dbt_assets(context: OpExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["run" ,"--profiles-dir /my/custom/dir"], context=context).stream()
    yield from dbt.cli(["test", "--profiles-dir /my/custom/dir"], context=context).stream()
I would expect that to output a valid CLI command that can be run. I would like to avoid extending the main dagster cli class and try to keep things as “out the box” and vanilla as possible
s

Sean Lopp

08/04/2023, 11:15 PM
Believe so yes, you’ll just need to remember to add that to every one
r

rex

08/04/2023, 11:33 PM
You can follow this: https://dagster.slack.com/archives/C04CW71AGBW/p1690578876840239?thread_ts=1690384210.355069&cid=C04CW71AGBW if you put your profiles.yml in the root of your dbt project dir, then you don’t need to configure it as a CLI flag for every command.
m

Manish Khatri

08/07/2023, 6:47 PM
Thanks! I’ve managed to do both blob highfive. The profiles dir i’m currently passing as an arg, but will move away from that and set the env variable. With regards to the
CustomDagsterDbtTranslator
example, that helped a lot. I should add that the example here would ideally need the
@classmethod
decorator so it matches the exact same function body as
DagsterDbtTranslator
otherwise mypy will complain when doing static type checking.
r

rex

08/07/2023, 6:47 PM
This will be first class in this week’s release once https://github.com/dagster-io/dagster/pull/15722 is landed. Thanks for the feedback.
❤️ 2
6 Views