hei, is this a bug or am I doing something wrong? I started a larger backfill (over 9000 runs). They started fine but took too long so I decided to cancel remaining runs. Cancelling using UI (backfill -> terminate) didn't do anything. I then used the code found in this channel (send report_run_canceling through the instance). This worked but now all remaining runs are stuck at cancelled. then I look at daemon pod logs I see the message that max runs number has been reached. I have set up max 300 concurrent runs but it seems that now because of large number of runs in status cancelling none of them are actually proceeding with cancelling. Is this an expected behavior? And what is the recommended way to cancel a large number of runs (once again the UI dialog just got stuck)?
using report_run_canceled instead of report_run_canceling cleaned it up