And / or is anyone looking into using dynamic tabl...
# integration-snowflake
s
And / or is anyone looking into using dynamic tables? Using Dagster as a way to manage dynamic tables w/ monitoring might be pretty cool.
đź‘€ 1
❤️ 1
g
We are using materialized views which might be a pendant
m
We’re looking at using dynamic tables. I believe dbt has support for them which is probably the best way to get the table relationships in to dagit, but the tricky part so far has been managing the interim when a table is being updated. Queries on that table whilst it is being updated fail so there would need to be some special handling for that in any sensors
đź‘Ť 1
I vaguely recall a GitHub issue which talked about a feature in which table views get special handling. That is, if you have a table, a view on that table, and then an asset taking the view as an input, the view shouldn’t need “updating”. It should inherit the “upstream changed” state, but assume that it has automatically been pseudo-materialised This is almost an extension of that. Table A (upstream asset) -> B (Dynamic table) -> C (downstream consumer) If Dagster does something to change the contents of A, changes to B will happen without any need for a job to be kicked off. Dagster can’t kick off a job materialising C until it knows the update to B has finished, but given it didn’t initiate that task it isn’t something for which it has a blocking process to track
đź‘Ť 1
I’m not really sure what the best way to express this would be It isn’t an observable source asset, in that it is a descendant of something in your DAG and you should encode that You could perhaps have a sensor looking at the table attached to asset B, but then you’re constantly spinning up a warehouse to do metadata queries. Not really financially viable Maybe some sort of event triggered sensor which only polls the dynamic table to log completion of the refresh when a known upstream asset gets changed?
f
A way to express this would be: • when "materializing" a dynamic table (for which the code definition has not changed), the operation that Dagster has to do is essentially "wait until the dynamic table is ready" And by extension • when "materializing" a view (for which the code definition has not changed), Dagster has nothing to do