In dagster dbt the following lines skip adding tests as dags dagster #ask-community

In dagster-dbt, the following lines skip adding te...

Emir Karamehmetoglu

03/21/2023, 4:31 PM

In dagster-dbt, the following lines skip adding tests as dagster assets. https://github.com/dagster-io/dagster/blob/2c20460adb2c600b0c8ab94367229ef4d0a34873/python_modules/libraries/dagster-dbt/dagster_dbt/asset_defs.py#L200 would be great to make that optional, so I can see my tests as assets with proper linking. Has anyone managed to put dbt tests as assets into dagster?

Emir Karamehmetoglu

03/21/2023, 4:36 PM

I hope it’s allowed to tag you @owen as you had corresponded regarding dbt tests before. (https://discuss.dagster.io/t/8026084/does-dagster-have-native-support-for-dbt-tests-for-example-w). The issue I have is different, I think it would be really nice to represent tests as downstream assets as well. Because they are. I often want to react to the output of a test in someway and I am interested in it. So I want my dbt tests as dependent assets on the dbt models they test. Any hints you can give me? I am trying to hack it together in dagster-dbt and rebuild

owen

03/21/2023, 4:44 PM

hi @Emir Karamehmetoglu! at the moment, this would indeed require a fair bit of hacking, as there is no hook for modifying this behavior. I think you'd have to start with updating this line to include tests as an asset resource type: https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/python_modules/libraries/dagster-dbt/dagster_dbt/asset_defs.py?L176:6. The biggest lift would be updating this function: https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/python_modules/libraries/dagster-dbt/dagster_dbt/asset_defs.py?L418 to make sure you get the correct event type for tests (right now, it emits AssetObservations, but you'd need to emit outputs). This definitely isn't supported behavior so you might run into other roadblocks along the way, but that's where I would start approaching this

Emir Karamehmetoglu

03/21/2023, 5:10 PM

Thank you so much! At least I have a place to get started!

Emir Karamehmetoglu

04/01/2023, 3:58 PM

Hi @owen, I got this working in a few ways, but wanted your advice on how to handle the tests as assets when construction via AssetsSelection: Should: • test assets be in the same

AssetsSelection

_dbt_nodes_to_assets

? ◦ and use

build

command, and keep the same

dbt_op

? ◦ use a new

graph

for the NodeDefinition of the AssetsSelection that chooses

dbt_test

dbt_build

dbt_run

based on the asset resource type. • Or be in a second

AssetsSelection

that only contains tests (and obviously their upstreams/downstreams) so that they can be independently triggered? I guess using

dbt test

command. This is a question of "what is ideal?" Maybe one or the other is an easier approach, not sure.

Emir Karamehmetoglu

04/03/2023, 7:18 AM

See Draft PR: https://github.com/dagster-io/dagster/pull/13324

2 Views

Open in Slack

Previous Next