When using the QueuedRunCoordinator is there any way to cancel pending jobs (as in those in the queue but not yet in a 'Preparing' or later state), without canceling all currently running jobs? If not, is this something we might be able to achieve with a custom RunCoordinator?

Hi Zachary. Would it work to navigate to the

Queued

tab on the runs page and bulk terminate from there?

Thanks for the quick response! So it's actually the tasks queued within a run that we're hoping to cancel.

Oh, I see. Yeah, canceling a run is handled by the run coordinator and the run launcher. The run coordinator handles cancelation at the run level (i.e. if it is queued at the run level) and the run launcher can terminate an in-progress run, including issuing commands to terminate all of the in-progress step execution. Usually the run coordinate delegates to the run launcher for in-progress runs.

I see, so the run launcher sounds like where I'd want to start looking around

That’s right… which run launcher are you using?

currently the DockerRunLauncher

Yeah, I think you’d probably need the granularity to launch/cancel execution of individual steps. For example, the

CeleryK8sRunLauncher

works in concert with the

celery_k8s_job_executor

to launch each step as a separate k8s job, so I think it could be modified so that steps that had not yet been picked off the celery queue could be canceled without disrupting in-progress steps.

Hi Zachary- this is not possible out of the box today, but it’s a request we’ve gotten from multiple users. It’s on our radar but we don’t have concrete plans yet. If you were to build this, it would take a custom executor. The current executors forward interrupts to each executing step- instead you’d want to just stop launching new steps and wait for those that are running.