Is it possible yo cancel parallel steps in some fo...
# ask-community
Is it possible yo cancel parallel steps in some form based on a failure or logic in another. Or how can I signal to cancel the entire graph rather than only downstream ops. dag: step1 -> ... long_running_par1 -> val1(async) long_running_par2 -> val2(async) long_running_par3 -> val3(asyn) aggregate_outs(par1,par2,par3) So aggregate moves on before validation (speed reasons maybe). Now if val2 fails I would lile to stop par3 and par1 to cancel i case they are still running. Note that aggregate does not wait for validations to complete so in might finish before val complete but in case it hasn't I want to stop wasting resources immediately. Any hints how I could achieve this?
hi @Samuel Stütz! there's no built-in way to do this at the moment, but this is definitely a reasonable request. One thought that came to mind (although I haven't tried it myself) is to create a failure_hook that fires off something like
. Then you can apply this hook to every step on your job using the hooks parameter on the job decorator (i.e.
there might be some subtitles there where this could end up sending a bunch of cancelation requests at once, so you might need to add some extra checks around that.
In theory this should work if the cancel_run function works. I found no reference of it the docs.