Are there any good examples floating about RE creating an ob dagster #integration-dbt

Are there any good examples floating about RE: cre...

Joel Olazagasti

04/25/2023, 5:12 PM

Are there any good examples floating about RE: creating an observable source asset for a DBT source that generates a

DataVersion

based on a query to the table it's generated from? For example, getting the number of rows.

Tim Castillo

04/25/2023, 5:38 PM

Hi! I can't find one right now, but this is an interesting use case! Mind if I ask a question to help out with this? What would you like to do by having a dbt source as a separate observable source asset?

Joel Olazagasti

04/25/2023, 5:40 PM

Hi Tim, sure thing! We have a number of manually loaded data sources in Snowflake that end up as DBT sources, and our ideal scenario is to have the downstream DBT assets rematerialize whenever those sources are changed.

Tim Castillo

04/25/2023, 5:46 PM

That makes sense and it's an interesting (and logical) push for how to connect observable source assets + dbt models work together. Let me take it back to the team and get their thoughts.

Tim Castillo

04/25/2023, 6:08 PM

Update: It should work. There are a couple ways to figure out that data version. I'd lean away from row count and instead try to track when it was last updated. ex. if you have an

updated_on

column, to calculate the data version off of

max(updated_on)

Joel Olazagasti

04/25/2023, 6:19 PM

Great to hear! I have a follow up question, but I think that will pertain more to the Snowflake team, so I'll take it to the appropriate channel

Joel Olazagasti

04/25/2023, 6:19 PM

Appreciate the help

denes

04/25/2023, 8:00 PM

you can get the timestamp of the last update from the TABLES information schema view:

Copy code

select last_altered 
from information_schema.tables 
where table_name = '<TABLE_NAME>'

🤩 1

3 Views

Open in Slack

Previous Next