Hi team we just upgraded to 0 12 3 from 0 11 x We noticed th dagster #ask-community

Hi team, we just upgraded to 0.12.3 from 0.11.x. W...

paul.q

08/02/2021, 1:26 AM

Hi team, we just upgraded to 0.12.3 from 0.11.x. We noticed that our partition-based daily schedule kicked off pipeline runs over the the weekend even though the partition selector we use generated partitions that have already been run, as evidenced by the partition and run_keys shown below - this wasn't happening before. Although we have

cron_schedule="0 3 * * *"

in our schedule, the weekend runs were previously inhibited, I presume due to the run_key having already been used? We can (and should), of course, change the cron_schedule to add MON-FRI, but did we just fluke the desired behaviour before 0.12.x?

sandy

08/02/2021, 3:24 PM

@daniel - is this familiar to you?

daniel

08/02/2021, 3:25 PM

Is it possible to see the partition selector? I'm a little rusty on how exactly run keys work within schedules

daniel

08/02/2021, 3:28 PM

Ok, I just checked - I think setting a run key within a schedule only disambiguates runs within a single tick (so if your schedule did two runs each day, run keys would let you be sure that you only ever got those two runs on that day no matter what). So yeah, the mystery here is why these were ever skipping the weekend..

daniel

08/02/2021, 3:29 PM

it's possible we used to be checking for duplicate partitions more explicitly within the scheduler before, that seems more likely than run keys to me. At the risk of passing the buck again @prha may have context here too

3 Views

Open in Slack

Previous Next