https://dagster.io/ logo
#dagster-serverless
Title
# dagster-serverless
s

Sterling Paramore

04/14/2023, 4:31 PM
Unable to terminate job. I have a job that got stuck in a “starting” state for the last 33 hours (for a job that is supposed to run every 3 hours). I’ve tried terminating the job, but it persists in the “starting” state. I want to do the “force terminate” but there’s this warning about computational resources. How do I ensure that the resources are cleaned up after doing this? Also, how can I prevent this stuck state in the future or at least get alerted if a job is taking much longer than expected?
And then after I tried force terminating, I got this message. However, now it does look like the job is terminated.
j

Joe

04/14/2023, 4:33 PM
the good news is that in the case that the underlying compute is not cleaned up successfully, we eat that cost and it shouldn't be reflected in your metered usage
it is odd that this run got stuck though...
can you share a link to it?
s

Sterling Paramore

04/14/2023, 5:59 PM
I’ll DM you the link.
Is there a way to put a max run time on a job, and would that even work if it required a “Forced Termination”?
j

Joe

04/14/2023, 9:33 PM
there isn't a way to do that atm there's this gh issue open for tracking https://github.com/dagster-io/dagster/issues/3666 but i think even with that if the run isn't terminating you'd still likely have the same issue
s

Sterling Paramore

06/01/2023, 4:01 PM
This happened again today
d

daniel

06/01/2023, 8:00 PM
We've added job-level timeouts since this was first posted actually
s

Sterling Paramore

06/01/2023, 8:24 PM
ah cool, I saw that the PR mentioned above was still open so I didn’t think it was addressed yet
12 Views