https://dagster.io/ logo
Title
c

Chris Chan

10/25/2021, 5:29 PM
is there a way to programmatically terminate runs? I’m trying to figure out how to build a sensor that will terminate runs if they are taking longer than expected
p

prha

10/25/2021, 5:39 PM
There are two parts to this… If you have a run coordinator configured on your instance, you can call
cancel_run
on that run coordinator (if it supports that functionality)
that makes sure to signal to any in-progress computation to stop execution of the run
The second is to mark the run as canceled, which you can do with
instance.report_run_canceled
.
c

Chris Chan

10/25/2021, 5:42 PM
cool, thanks - I’m using the
QueuedRunCoordinator
- I presume I can access that through
context.instance
?
p

prha

10/25/2021, 5:42 PM
You can do the latter without the former, but it might mean that there might be lingering computation chugging away even though Dagster has marked the run as canceled.
yes, that’s right. It still depends on your run launcher implementation, whether or not you can cancel, but you should be able to check by calling
instance.run_coordinator.can_cancel_run(run_id)
c

Chris Chan

10/25/2021, 5:44 PM
great, thank you!
p

prha

10/25/2021, 5:47 PM
I imagine it’d look something like this (untested):
if instance.run_coordinator.can_cancel_run(run_id):
    instance.run_coordinator.cancel_run(run_id)

# fetch the run to see if its state has changed since calling cancel
run = instance.get_run_by_id(run_id)
if not run.is_finished:
    # just mark run as canceled
    instance.report_run_canceled(run)
m

max

10/25/2021, 5:56 PM
maybe also just worth noting the existence of the graphql api for this as well
p

prha

10/25/2021, 5:58 PM
ah yes, there’s a section in our graphql API docs for this exactly: https://docs.dagster.io/concepts/dagit/graphql#terminate-an-in-progress-run
y

yuhan

06/20/2022, 11:07 PM
@Dagster Bot discussion Programmatically terminate runs
d

Dagster Bot

06/20/2022, 11:07 PM
Question in the thread has been surfaced to GitHub Discussions for future discoverability: https://github.com/dagster-io/dagster/discussions/8509