Juan Freire

04/21/2023, 3:29 PM
Hi. Is it possible to ensure a limit of one queued run for each job, prioritizing the most recent, or any other way to discard runs queued for too long? I have a problem right now with a job that executes every few minutes, which sometimes can surpass the scheduled period, and it ends up generating a snowball effect of runs accumulating in queue due to dagster always prioritizing the oldest partition pending to run, while I would prefer if it just skipped right to the last scheduled and leave a gap to be treated later.


04/21/2023, 8:59 PM
hi @Juan Freire! one (admittedly somewhat jank) solution would be to set the priority of your runs to be a function of the current timestamp (to give newer runs higher priority), but there's currently no great solution for automatically removing older runs in the queue

Juan Freire

04/22/2023, 1:12 PM
Thank you so much! Would you recomend opening an issue on this? (if there isn´t one already)