Hi! im running with a bit of an issue and some jobs that usually take minutes are taking hours. and are piling up due to the scheduler. Is there a way to stop executions that started with cron?

we are working on supporting termination from dagit, from you you’ll have to send a keyboard in interrupt to the process (or kill it for less graceful termination)

Which process should I kill so to make it a little bit as clean as possible? I was using the multiprocessor executor. We killed all the processes that had dagster in it. Also, there were some processes that did a tail on compute logs, we had to remove those manually too.

if you send a

sigint

to the process coordinating the run the multiprocess executor will forward the interrupt and everything should wind down and shut down correctly

I see. How can I identify the coordinator processes?

I believe the engine events in the event log print process id - but not positive

ah true, thanks

they will be invoking

dagster api execute_run

We still had like 40 to clean haha

as opposed to the multiprocess executor which is using the python multiprocessing library

Do you have an active issue for the functionality of terminating a pipeline manually? I would be very interested in following its development.

Thanks, is this the reason why all the runs still appear running in dagit? could you suggest a way to fix the state?

They still appear running in dagit since nothing updated the run DB to put them in to failed state. If they shut down gracefully theyll do that. You can either delete the runs to remove them or update them manually in a python script or shell doing something like

instance = DagsterIntance.get()
run = instance.get_run_by_id(bad_state_run_id)
instance.report_run_failed(run)

Thanks, how can i get all the ids that appear running?

DagsterInstance.get()

will use the

dagster.yaml

in

$DAGSTER_HOME

, as long as you are talking to right

run_storage

you can do what you need You should be able to see the run ids in

dagit

,

I see, this is interesting to implement certain maintanance scripts. Thanks!

ya if you end up with a script you can do

all_runs = instance.get_runs()

and then just poke at the

PipelineRun

s to decide what to do