for the `build asset reconciliation sensor` it would be grea dagster #dagster-feedback

for the `build_asset_reconciliation_sensor` it wou...

Charlie Bini

01/29/2023, 6:56 PM

for the

build_asset_reconciliation_sensor

it would be great to have 2 feature additions: 1. be able to limit the number of partitions it looks back. I think currently, it's the max (24) 2. be able to pass some type of custom evaluation function that determines whether or not a asset or partition should be materialized

Charlie Bini

01/29/2023, 6:57 PM

love the functionality this brings, but I've found myself copy/pasting the

reconcile

function in order to insert my own logic

Charlie Bini

01/29/2023, 6:58 PM

for example, if an asset or partition is not materialized, I'd like it to first run a quick count query before deciding to launch an entire pod to run the job if there are no records to retrieve

Nicolas Parot Alvarez

01/30/2023, 3:01 PM

About point 1, it used to backfill all the partitions, which could create huge backfills. To avoid that, it was recently limited to exactly one day. I'm also suggesting to make this limit customizable. See discussion with @owen here: https://dagster.slack.com/archives/C01U5LFUZJS/p1672449665938019

jamie

01/30/2023, 3:44 PM

thanks for the feedback! I’ve passed it on to the team

Charlie Bini

01/30/2023, 8:37 PM

thanks @jamie! yeah cutting it down to 1 day help, but if I want to put a number of partitioned assets on one sensor, it still overloads it and times out

Charlie Bini

01/30/2023, 8:37 PM

being able to limit the number of "lookback" partitions per asset as low as 1 would help a lot

Nicolas Parot Alvarez

01/31/2023, 10:11 AM

So I think that would be two distinct features: limiting the number of look-back days and/or limiting the number of partitions considered per asset.

4 Views

Open in Slack

Previous Next