Does anyone know if it is possible to run multiple...
# announcements
w
Does anyone know if it is possible to run multiple partitions within a single scheduled pipeline run? If so, are there are resources or links showing an example of this?
s
Hey Warren - do you mind giving a little more color on your use case? Is it that you'd like to run the same set of partitions for every scheduled run? Or something like a rolling window?
w
I have data that is partitioned by year and month, and would like to schedule a pipeline run to process all of these partitions on a daily basis. We currently use the backfill command to run all the partitions manually and would like to schedule that to run automatically. Is something like that possible?
s
Got it. If you'll be processing every data partition every day, then you don't need Dagster's partition concept. You can use
Copy code
@schedule(pipeline_name=pipeline_name, cron_schedule=...)
def my_daily_schedule(context):
    return <config that parameterizes the pipeline to operate over all partitions>
w
Thanks, I'll give that a try!