https://dagster.io/ logo
#ask-community
Title
# ask-community
a

Alvin Yeap

03/17/2023, 11:45 AM
Hello 👋 is it possible to have time-based partition with greater granularity than
HourlyPartititionsDefinition
? I’d like to poll an endpoint every 5 minutes for updates. I don’t actually need the partition to be on 5 minute intervals. If I can create a schedule that runs every 5 minutes but writes to the current hour’s partition, that would suffice. However, it looks like the partitions don’t get created until after the end of the interval (e.g. 3pm partition is only created after 4pm).
a

Adam Bloom

03/17/2023, 12:05 PM
Not all of what you’re looking for, but the
end_offset
parameter might be useful? https://docs.dagster.io/_apidocs/partitions#dagster.HourlyPartitionsDefinition
a

Alvin Yeap

03/17/2023, 12:07 PM
Hmm interesting…so I can set an
end_offset=1
and the prior hour’s partition would cover the previous hour up to the end of the current hour? Nice, I’ll give it a try! 🙏
2 Views