https://dagster.io/ logo
Powered by
Title
a

Arsenii Poriadin

05/02/2023, 2:40 PM
I have a question about run monitoring, specifically about run start timeouts I see in the docs that value 
180
for 
start_timeout_seconds
is the default and doesn't "need to be specified execpt to override" it but can I actually have no timeout for the run to start while still leaving 
run_monitoring
itself 
enabled
? (I need it to have general run timeouts)
@claire @daniel maybe you can give any input on that? couldn't find any supporting documentation elaborating on that config key will try to set it to 
0
or 
-1
and test, maybe this is how I disable this timeout
d

daniel

05/03/2023, 4:29 PM
I don't think we supporting it to 0 or -1 currently although it's a very reasonable ask - in the short term, it's a bit inelegant, but would setting it to a very high value be an option that it will never realistically hit (say, the number of seconds in a year)?
a

Arsenii Poriadin

05/03/2023, 5:09 PM
yes, that's what we did for now
thank you for the reply
:condagster: 1
@daniel while you're still here, maybe you can advise on what I can do if I have a lot of jobs in 
starting
state but none actually start because the k8s pods which daemon created for it do not exist anymore (2nd screenshot) I guess these jobs were somehow requested during the 
helm upgrade
of the whole setup and all these jobs became orphans with no actual pod for them
d

daniel

05/03/2023, 5:57 PM
probably worth a new post for this question - but what I would suggest is to force terminate the runs
a

Arsenii Poriadin

05/03/2023, 5:58 PM
and there is no way to trigger the re-execution of all?
I can do it 1 by 1, and it works by creating a new k8s job for it, but there is a 
1000
of them hehe
d

daniel

05/03/2023, 8:17 PM
Here's a script that you could use: 
from dagster import DagsterInstance, RunsFilter, DagsterRunStatus

instance = DagsterInstance.get() # needs your DAGSTER_HOME to be set
queued_runs = instance.get_runs(filters=RunsFilter(statuses=[DagsterRunStatus.STARTING]))
for run in queued_runs:
    instance.report_run_canceled(run)
you could run that on the dagit pod
a

Arsenii Poriadin

05/04/2023, 7:56 AM
that would help me restarting them, but at least I can clear it out this way, yes thank you
d

daniel

05/04/2023, 3:19 PM
Included a fix that will let you set start_timeout_seconds to 0 here and have it turn off the feature, even if run monitoring is disabled: https://github.com/dagster-io/dagster/pull/14084
a

Arsenii Poriadin

05/04/2023, 5:43 PM
so with this it won't be failing runs if they are in 
STARTING
state for indefinitely long time?
d

daniel

05/04/2023, 5:45 PM
yeah, it would disable the feature
a

Arsenii Poriadin

05/04/2023, 5:45 PM
nice, thanks looking forward to have it released
2 Views
#dagster-supportJoin Slack
Powered by