Hey all -
Wondering if anyone here has made an attempt at "asset versioning". We are wondering how to:
1. see which partitions for a given asset differ in version (perhaps the underlying dbt model sql changed)
2. determine if a downstream asset's partition is "out of date" compared to an upstream asset's same partition
11/11/2022, 11:51 PM
If I understand what you're looking for, we're currently working on first-class support for this: https://github.com/dagster-io/dagster/pull/10006.
You'll be able to define versions on your asset definitions, and Dagit will represent assets as "stale" when their most recent materialization used a different version than the current version on the definition. "Staleness" will propagate down the graph to downstream assets. The non-partitioned version of this functionality should land in a week, and then we'll start tackling partitions.
We might ping you for feedback on this soon if you're up for it.
11/12/2022, 12:34 PM
This is absolute 🔥
Happy to give feedback!
Btw, is there a place where we can view yall's roadmap?
This is very cool, thanks for the demo
Looking at this through the lens of dbt I'm wondering how we would handle versioning for our use case. Ideally we can auto increment the version of the dbt op a model corresponds to any time its SQL change. Just food for thought - looking forward to the implementation for partitioned assets!
11/22/2022, 9:11 PM
That's something we've been thinking about too - our plan is to default the version to a hash of the SQL for the dbt model. Would that work for you?