https://dagster.io/ logo
Join Slack
Powered by
What is the general recommendation for running dbt...
# feature-asset-checks
t
What is the general recommendation for running dbt Singular Data Tests in conjunction with asset checks? The asset checks are great in that they run the tests for the individual asset. However, I am then not sure how singular data tests can fit into the picture. In our old MWAA environment, we simply ran everything with tags. So the 
build
call would pick up both the individual asset tests and the singular data tests since they had the same tag. But I need to think differently in Dagster. My current line of thinking as the simplest (but not the greatest) implementation would be to simply have a sensor which kicks off a job once our core asset job has finished, wit this sensor kicking off a job which simply runs all of the “missing” tests - including these singular data tests, and some relational integrity tests.
b
Can they be run as a separate asset check, using the dbt resource and invoking the test, with that test selected?
t
Ah good idea. I’ll give that a try. Thanks for the suggestion @Brendan Jackson
This was a good idea and I got it working. In essence, what I’ve done is: 1. Added a tag to each singular test of 
"dagster_asset_check: model_to_attach_the_test_to"
2. Created a 
build_all_dbt_singular_tests
function which parses the manifest for any tags that begin with 
dagster_asset_check
3. For any tests tagged with 
dagster_asset_check
, parse the model name out from the tag and assign the singular test to that dbt model
There might be a more elegant want to handle this, but this seems to do the trick. The only downside is that you manually have to add in the model tag (required for the uses cases where we have multiple dbt references in a single test)
b
Nice! I've been meaning to try this myself. I'll probably manually assign the checks since I only have a couple right now, but it's what has been holding me back from moving some dbt tests to asset checks
Thanks for giving it a try!
t
I can ping you my (sloppy) code if that will help
b
That's kind of you to offer. I'll give it a try first and get back to you if I hit a block?
👍 1
t
No worries. FYI, there is an open github issue for the Dagster team to implement singular tests
👍 1
So in due course, I’d expect that this workaround can be removed
b
I'm hoping so!
4 Views
Open in Slack
PreviousNext
Powered by