I am getting read timed out during code location d...
# dagster-plus
g
I am getting read timed out during code location deployment upgrade for a 4GB docker image. Can you increase the timeout limit?
d
Hey georg - can you post the exact message you're seeing? (but yes, the timeout can be increased - just need to know where exactly its failing)
g
It is failing sporadically - right now it is green. I will post a message next time I observe the failure
requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1120, in _reconcile new_dagster_servers[to_update_key] = self._start_new_dagster_server( File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1406, in _start_new_dagster_server return self._start_new_server_spinup( File "/dagster-cloud/dagster_cloud/workspace/docker/__init__.py", line 286, in _start_new_server_spinup container, server_endpoint = self._launch_container( File "/dagster-cloud/dagster_cloud/workspace/docker/__init__.py", line 213, in _launch_container container = self._create_container( File "/dagster-cloud/dagster_cloud/workspace/docker/__init__.py", line 116, in _create_container return client.containers.create( File "/usr/local/lib/python3.8/site-packages/docker/models/containers.py", line 906, in create resp = self.client.api.create_container(**create_kwargs) File "/usr/local/lib/python3.8/site-packages/docker/api/container.py", line 431, in create_container return self.create_container_from_config(config, name, platform) File "/usr/local/lib/python3.8/site-packages/docker/api/container.py", line 447, in create_container_from_config res = self._post_json(u, data=config, params=params) File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 296, in _post_json return self._post(url, data=json.dumps(data2), **kwargs) File "/usr/local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner return f(self, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/docker/api/client.py", line 233, in _post return self.post(url, **self._set_request_timeout(kwargs)) File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 635, in post return self.request("POST", url, data=data, json=json, **kwargs) File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 587, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 701, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 578, in send raise ReadTimeout(e, request=request)
@daniel here you go
d
Looks like an issue with your docker client - there’s an env var listed here that appears to control that timeout https://github.com/docker/compose/issues/3927
And some other things to try listed there like giving docker more memory/cpu
g
thanks