Daniel Gafni
12/12/2022, 9:18 AMAllPartitionMapping
during upstream backfills.
He doesn't want to perform a full backfill, instead he wants to work with let's say 1 week of partitions from 2 months ago.
Of course, the AllPartitionMapping
doesn't like the fact that not all upstream partitions are materialized.
Meanwhile my colleague wants it to act like "load all partitions that I materialized previously".
I think it should be possible to do with a different partition mapping which would query Dagster for partitions that have ever tried to be materialized (both successfully and not) and then get the latest partition key from this list.
Any tips on how to implement this?
Edit: I realized the downstream asset should probably just be partitioned toosandy
12/12/2022, 10:20 PMDaniel Gafni
12/12/2022, 11:01 PM