Stephen Bailey12/31/2022, 1:21 AM
Zachary Bluhm01/03/2023, 3:23 PM
owen01/03/2023, 9:33 PM
Zachary Bluhm01/03/2023, 10:09 PM
Stephen Bailey01/04/2023, 2:10 PM
owen01/04/2023, 8:56 PM
Stephen Bailey01/05/2023, 1:32 AM
Nicolas Parot Alvarez01/20/2023, 1:58 PM
for time-window-partitioned assets, no runs will be kicked off for partitions whose windows end more than one day before the latest window for that asset. In essence, this gives a time bound of ~1 day for the sensor to materialize a partition of an asset once that partition comes into being.While I understand this reduces the troubles of generating countless surprise backfills when one adds a new asset into the reconciliation group, this also prevents the automatic backfill of missed partition that this sensor was allowing before. And this use case is rather important for us. Our infrastructure is not great, so it's possible the reconciliation couldn't run during the weekend, or during the vacations (damn French people) of the guy in charge of this pipeline who will know how to backfill. With this change to 1 day of look back, when we put Dagster back online on the next Monday, the reconciliation sensor will not be able to automatically backfill the weekend's or the week's missed partition. So picking up from your initial suggestions @owen, we would like to be able to customize that 1 day time-delta, so we can set it to 1 week or 1 month. What we love with Dagster, is that we can put our efforts into writing smart sensors that will highly reduce the amount of manual infrastructure issues related backfills we have to do.
owen01/20/2023, 5:02 PM
Nicolas Parot Alvarez01/23/2023, 10:11 AM
. But, being able to fully define the reconciliation at the moment of the asset definition would probably be the best and allow removing the reconciliation sensor that sticks out as a different breed than the other sensors. Something like:
build_asset_reconciliation_sensor(look_back_days: int = 1, ...)
@asset(reconciliation_policy: ReconciliationPolicy = None)
Or maybe allow both, and the sensor overwrites the assets' reconciliation policies.
ReconciliationPolicy(assets: Union[Sequence[AssetKey], Sequence[AssetDefinition], AssetSelection] = this_asset.upstream(depth=1), look_back_days: int = 1, ...)