https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
c

Charlie Bini

01/29/2023, 6:56 PM
for the
build_asset_reconciliation_sensor
it would be great to have 2 feature additions: 1. be able to limit the number of partitions it looks back. I think currently, it's the max (24) 2. be able to pass some type of custom evaluation function that determines whether or not a asset or partition should be materialized
love the functionality this brings, but I've found myself copy/pasting the
reconcile
function in order to insert my own logic
for example, if an asset or partition is not materialized, I'd like it to first run a quick count query before deciding to launch an entire pod to run the job if there are no records to retrieve
n

Nicolas Parot Alvarez

01/30/2023, 3:01 PM
About point 1, it used to backfill all the partitions, which could create huge backfills. To avoid that, it was recently limited to exactly one day. I'm also suggesting to make this limit customizable. See discussion with @owen here: https://dagster.slack.com/archives/C01U5LFUZJS/p1672449665938019
j

jamie

01/30/2023, 3:44 PM
thanks for the feedback! I’ve passed it on to the team
c

Charlie Bini

01/30/2023, 8:37 PM
thanks @jamie! yeah cutting it down to 1 day help, but if I want to put a number of partitioned assets on one sensor, it still overloads it and times out
being able to limit the number of "lookback" partitions per asset as low as 1 would help a lot
n

Nicolas Parot Alvarez

01/31/2023, 10:11 AM
So I think that would be two distinct features: limiting the number of look-back days and/or limiting the number of partitions considered per asset.
4 Views