Hey all, qq: what happens when you start a backfill that takes longer than the scheduled interval? Will the backfill be smart enough to reload the newest partition as well? for example, I have partitions 1, 2, 3 available when starting the backfill. The backfill takes so long that a partition 4 becomes available (new hour, day etc). Will the backfill automatically include that new partition? If not, any suggestions for getting around that?

It won’t - the partitions for the backfill are locked in when you create it. Why would you want that though? Wouldn’t you want a separate schedule or sensor handling new partitions as they’re made available that would automatically queue a job for partition 4?

my concern is that partition 4 runs before the backfill completes and if partition 4 contains tables that have lookbacks at previous partitions, then partition 4 could have wrong data. Correctness here would require that a backfill reruns all partitions (including those that are created while the backfill is running)

yup, that would do it pretty easily