Is there an existing partitioning strategy for wha...
# ask-community
j
Is there an existing partitioning strategy for what I'd call rolling window? Ie, monthly partitioned asset, but a rolling window of which partitions to process each time. Say I always want to process the current month and the previous two? Do I just set it as monthly partitioned and in my jobs, always launch backfill?
s
Hey Jordan - here's a github discussion with an example: https://github.com/dagster-io/dagster/discussions/14636
j
Not sure if it's the same, but I think managing through backfill works.
I want data for yesterday refreshed today, but I need to keep refreshing it for 90 days. After 90 days, it's locked in.
Better posed example: If this months and previous two months data can update at any moment, what's the correct dagster way to refresh the asset, hourly?
s
Every hour, do you want to refresh the last 90 days of data?
j
Yes, but still have it partitioned so larger backfills can be run concurrently.
s
You can use the schedule decorator to write a schedule that requests multiple partitions at once: https://github.com/dagster-io/dagster/discussions/14622
👍 1