Sarah Gruskin

06/10/2021, 9:49 PM
Hello, I'm trying to setup a pipeline that is partitioned and has a schedule. I ended up making this stackoverflow question. Was wondering if anyone would be able to help.


06/10/2021, 9:59 PM
hey sarah - as a side note, it’s a little bit more ergonomic to define a schedule using one of our schedule based decorators:, since you’re defining a partition set based on dates


06/10/2021, 10:09 PM
^ agree with what rex said, but what's going on with your specific code has a pretty simple fix. your get_date_partitions function is returning a list with a single
that contains a list of all the dates. It should probably be written something like this instead:
def get_date_partitions():
    return [
        Partition(datetime.strftime(TODAY - timedelta(days=x), DATE_FORMAT))
        for x in range(get_number_of_days().days)
This will return a list of partitions, representing a single day each. However, the schedule documentation that rex linked is really the easiest way to implement your desired behavior, so I recommend checking that out.

Sarah Gruskin

06/10/2021, 10:10 PM
That did the trick! Thanks so much! SO SIMPLE!
😛artydagster: 1