Is there some maximum time a pod is allowed to be pending be

Question

Is there some maximum time a pod is allowed to be pending before it s treated as failed In my case with 5k dynamic outs eventually some of the pods which were still Status Pending according to kubectl showed up as failed in Dagit stack in thread which caused my job to fail I m trying to figure out if it s a reasonable theory that the agent just got tired of waiting for the pods to schedule even though k8s thought they would eventually get scheduled and decided they must have had an error or if there was some error status or communication error on in the pod I overlooked

Mark Fickett · Answer

```urllib3 exceptions MaxRetryError HTTPSConnectionPool host= 172 20 0 1 port=443 Max retries exceeded with url apis batch v1 namespaces dagster cloud jobs dagster step 2262cd711596e75e5477cb052aa81db2 Caused by NewConnectionError lt urllib3 connection HTTPSConnection object at 0x7f4f7b86d780 gt Failed to establish a new connection Errno 111 Connection refused File usr local lib python3 10 site packages dagster core executor step delegating step delegating executor py line 243 in execute health check result = self step handler check step health File usr local lib python3 10 site packages dagster k8s executor py line 248 in check step health job = self batch api read namespaced job File usr local lib python3 10 site packages kubernetes client api batch v1 api py line 2657 in read namespaced job return self read namespaced job with http info name namespace kwargs noqa E501 File usr local lib python3 10 site packages kubernetes client api batch v1 api py line 2744 in read namespaced job with http info return self api client call api File usr local lib python3 10 site packages kubernetes client api client py line 348 in call api return self call api resource path method File usr local lib python3 10 site packages kubernetes client api client py line 180 in call api response data = self request File usr local lib python3 10 site packages kubernetes client api client py line 373 in request return self rest client GET url File usr local lib python3 10 site packages kubernetes client rest py line 241 in GET return self request GET url File usr local lib python3 10 site packages kubernetes client rest py line 214 in request r = self pool manager request method url File usr local lib python3 10 site packages urllib3 request py line 75 in request return self request encode url File usr local lib python3 10 site packages urllib3 request py line 97 in request encode url return self urlopen method url extra kw File usr local lib python3 10 site packages urllib3 poolmanager py line 336 in urlopen response = conn urlopen method u request uri kw File usr local lib python3 10 site packages urllib3 connectionpool py line 754 in urlopen return self urlopen File usr local lib python3 10 site packages urllib3 connectionpool py line 754 in urlopen return self urlopen File usr local lib python3 10 site packages urllib3 connectionpool py line 754 in urlopen return self urlopen File usr local lib python3 10 site packages urllib3 connectionpool py line 726 in urlopen retries = retries increment File usr local lib python3 10 site packages urllib3 util retry py line 439 in increment raise MaxRetryError pool url error or ResponseError cause ```

daniel · Answer

Hey Mark we don t have a timeout no the k8s executor check up on steps to see if they re still running and it looks like that request failed with a communication error within the k8s cluster But there isn t a timeout at play

alex · Answer

version `1 1 6` includes a change that implements better retries for the specific exception you hit here

Andrea Giardini · Answer

< alex> Can you link to it here

Mark Fickett · Answer

Thanks

alex · Answer