Is it possible yo cancel parallel steps in some fo...
# ask-community
s
Is it possible yo cancel parallel steps in some form based on a failure or logic in another. Or how can I signal to cancel the entire graph rather than only downstream ops. dag: step1 -> ... long_running_par1 -> val1(async) long_running_par2 -> val2(async) long_running_par3 -> val3(asyn) aggregate_outs(par1,par2,par3) So aggregate moves on before validation (speed reasons maybe). Now if val2 fails I would lile to stop par3 and par1 to cancel i case they are still running. Note that aggregate does not wait for validations to complete so in might finish before val complete but in case it hasn't I want to stop wasting resources immediately. Any hints how I could achieve this?
o
hi @Samuel Stütz! there's no built-in way to do this at the moment, but this is definitely a reasonable request. One thought that came to mind (although I haven't tried it myself) is to create a failure_hook that fires off something like
context.instance.run_coordinator.cancel_run(context.run_id)
. Then you can apply this hook to every step on your job using the hooks parameter on the job decorator (i.e.
@job(hooks={my_cancel_hook})
)
there might be some subtitles there where this could end up sending a bunch of cancelation requests at once, so you might need to add some extra checks around that.
s
In theory this should work if the cancel_run function works. I found no reference of it the docs.