Does anyone know of any quality example Airbyte &a...
# integration-dbt
e
Does anyone know of any quality example Airbyte & DBT Dagster projects to use as a reference? I've gone through all the docs an I can't find any clean setups that show good design practices when working with multiple jobs; everything I find is centered around one
Like say I have two different pipelines. A uses 3 tables via Airbyte to run 3 DBT models, and B does something similar. Like how do I tag them separately? Do I filter down my airbyte asset into smaller asset sets?
a
I think you’re heading down the right path of thinking - you can filter/subset assets to create sensors and jobs.
t
Definitely feel you on the lack of quality examples. It's hard to find much beyond Hello World use cases.
a
I'm not sure there's a good way to provide an approachable, yet complex enough, example to get you thinking the exact right way for your use case. Maybe others think differently, but my personal journey with dagster definitely started with just learning how to use the tool, and then figuring out the best way to model our pipelines (which I've probably refactored at least 2 times as I've learned more). There isn't necessarily one "right" way - there are many possible ways to build pipelines that do the same thing.
1
Anyways, I use both airbyte and dbt with asset sensors quite a bit in our pipelines, so happy to help with snippets if I can. I don't have my full pipeline in a shareable state, unfortunately...it has a bit too much company-specific data encoded in it