Donny Winston05/20/2022, 9:22 PM
In coming versions of Dagster, you'll be able to provide scheduling policies that directly address those above questions: you'll be able to specify the conditions under which your asset is considered unreconciled and specify how long it's ok for your asset to remain in an unreconciled state.And in the meantime, are there openly licensed examples of scheduling-policy Python code in the wild that people are currently using as an alternative to traditional sensors and schedules to trigger asset (re)materializations?
sean05/25/2022, 1:21 PM
sandy05/25/2022, 3:10 PM
Liezl Puzon05/25/2022, 3:25 PM
sandy05/25/2022, 3:34 PM
Donny Winston05/25/2022, 5:45 PM
(or perhaps an overloading of that decorator?) that would be called not on an asset materialization event, but on an asset "severance" ("deconciliation"?) event, where Dagster has some built-in severance predicate functions but one can also supply a custom predicate function for the Dagster daemon to evaluate periodically (this could dispatch to the existing sensor interface for specifying periodicity, e.g. every 30 seconds or something else by default), either way supplying the predicate function as a keyword arg to the decorator, and also some keyword arg(s) for "how long it's okay to remain unreconciled", also ideally with some Dagster built-in for specifying this (like, I don't know, "green/yellow/red" durations like 0min/1hr/24hr where the daemon will interruptably schedule a run request starting 0min after severance, will prioritize a rerun after 1hr, and "guarantees" reconciliation within 24hr? I may be overthinking the "how long it's okay" stuff, but I think it'd be nice to have built-ins for "try to rerun but don't make it high priority" versus "okay, now it really needs to get done".
sandy05/25/2022, 8:44 PM
Donny Winston05/25/2022, 9:13 PM
The most basic discrepancy in need of reconciliation is when you have a software-defined asset with no materializations.(2) sense when an upstream asset materialization is newer than this asset's latest materialization
If the most recent materialization of the asset occurred earlier than the most recent materialization of the upstream asset, then it’s likely that the asset's concents are stale.(3) sense when what this asset's latest definition requires is not provided by its latest materialization
A third discrepancy is when the asset definition’s metadata doesn’t match its materialization’s metadata...[for example] discrepancy between the set of columns in the asset’s definition and the set of columns in the latest materialization
sandy05/26/2022, 9:54 PM