Zachary Bluhm
01/11/2023, 4:30 PM@asset(
partitions_def=HourlyPartitionsDefinition(start_date="2023-01-01-00:00", timezone=DEFAULT_TIMEZONE),
ins={
"hourly_depends_on_past": AssetIn(
partition_mapping=TimeWindowPartitionMapping(start_offset=-1, end_offset=-1)
)
}
)
def hourly_depends_on_past(hourly_depends_on_past):
time.sleep(3)
return 1
And backfilled 10 partitions at a time. In theory I would expect this to finish locally with no K8s overhead extremely quickly (maybe < 90 secs), but in reality this took closer to 5.5 minutes. What's interesting is that each individual OP only took ~6.5 secondssandy
01/11/2023, 7:05 PMZachary Bluhm
01/11/2023, 7:17 PMsandy
01/11/2023, 7:28 PMZachary Bluhm
01/11/2023, 11:13 PM