What is the roadmap/timeline for built-in scheduling policy primitives/abstractions for software-defined asset reconciliation? From https://dagster.io/blog/software-defined-assets#orchestration-managing-change-in-assets by @sandy :

In coming versions of Dagster, you'll be able to provide scheduling policies that directly address those above questions: you'll be able to specify the conditions under which your asset is considered unreconciled and specify how long it's ok for your asset to remain in an unreconciled state.

And in the meantime, are there openly licensed examples of scheduling-policy Python code in the wild that people are currently using as an alternative to traditional sensors and schedules to trigger asset (re)materializations?

cc @sandy

Hey @Donny Flynn - we've been discussing this recently and exploring a few options. If you'd be open to it, I'd be interested in hearing what an ideal way of specifying these policies would be for you? We could talk about it on a call if that would be easier?

not sure if this makes sense / is aligned with others, but for me I imagine this helping with monitoring + extra retries beyond the existing static retry policies. I’d like my traditional sensors to have an extra decorator containing: 1. a function which validates if the sensor-triggered job has done it’s job properly (asset exists = job done properly) 2. some kind of retry policy if that validation fails

oops I meant @Donny Winston , not @Donny Flynn, though Donny Flynn, if you have thoughts, interested in hearing them as well!

@sandy I was thinking there would be a decorator similar to

@asset_sensor

(or perhaps an overloading of that decorator?) that would be called not on an asset materialization event, but on an asset "severance" ("deconciliation"?) event, where Dagster has some built-in severance predicate functions but one can also supply a custom predicate function for the Dagster daemon to evaluate periodically (this could dispatch to the existing sensor interface for specifying periodicity, e.g. every 30 seconds or something else by default), either way supplying the predicate function as a keyword arg to the decorator, and also some keyword arg(s) for "how long it's okay to remain unreconciled", also ideally with some Dagster built-in for specifying this (like, I don't know, "green/yellow/red" durations like 0min/1hr/24hr where the daemon will interruptably schedule a run request starting 0min after severance, will prioritize a rerun after 1hr, and "guarantees" reconciliation within 24hr? I may be overthinking the "how long it's okay" stuff, but I think it'd be nice to have built-ins for "try to rerun but don't make it high priority" versus "okay, now it really needs to get done".

@Donny Winston that makes a lot of sense. Do you have thoughts on what kind of severance predicate functions would be most useful to you?

You had good examples in your blog post. (1) sense when there is no materialization of this asset

The most basic discrepancy in need of reconciliation is when you have a software-defined asset with no materializations.

(2) sense when an upstream asset materialization is newer than this asset's latest materialization

If the most recent materialization of the asset occurred earlier than the most recent materialization of the upstream asset, then it’s likely that the asset's concents are stale.

(3) sense when what this asset's latest definition requires is not provided by its latest materialization

A third discrepancy is when the asset definition’s metadata doesn’t match its materialization’s metadata...[for example] discrepancy between the set of columns in the asset’s definition and the set of columns in the latest materialization

@Donny Winston - got it. We'd like to have something like this in the next couple months. We haven't locked in our roadmap yet, so stay tuned for further updates!