I've seen these `Read timed out` errors a couple t...
# dagster-plus
z
I've seen these
Read timed out
errors a couple times recently when emitting events to Dagster via a custom step launcher (error in thread)
Copy code
dagster_cloud_cli.core.errors.GraphQLStorageError: HTTPSConnectionPool(host='empirico.agent.dagster.cloud', port=443): Read timed out. (read timeout=60)
  File "/opt/venv/lib/python3.9/site-packages/dagster/_core/execution/plan/execute_plan.py", line 273, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/opt/venv/lib/python3.9/site-packages/etxlib/etxdagster/infrastructure/databricks/step_launcher.py", line 324, in launch_step
    yield from self.wait_for_completion(
  File "/opt/venv/lib/python3.9/site-packages/etxlib/etxdagster/infrastructure/databricks/step_launcher.py", line 381, in wait_for_completion
    yield from self.step_events_iterator(
  File "/opt/venv/lib/python3.9/site-packages/etxlib/etxdagster/infrastructure/databricks/step_launcher.py", line 243, in step_events_iterator
    step_context.instance.handle_new_event(event)
  File "/opt/venv/lib/python3.9/site-packages/dagster/_core/instance/__init__.py", line 1926, in handle_new_event
    self._event_storage.store_event(event)
  File "/opt/venv/lib/python3.9/site-packages/dagster_cloud/storage/event_logs/storage.py", line 414, in store_event
    self._execute_query(
  File "/opt/venv/lib/python3.9/site-packages/dagster_cloud/storage/event_logs/storage.py", line 292, in _execute_query
    res = self._graphql_client.execute(
  File "/opt/venv/lib/python3.9/site-packages/dagster_cloud_cli/core/graphql_client.py", line 126, in execute
    raise GraphQLStorageError(str(e)) from e
The above exception was caused by the following exception:
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='empirico.agent.dagster.cloud', port=443): Read timed out. (read timeout=60)
  File "/opt/venv/lib/python3.9/site-packages/dagster_cloud_cli/core/graphql_client.py", line 78, in execute
    return self._execute_retry(query, variable_values, headers)
  File "/opt/venv/lib/python3.9/site-packages/dagster_cloud_cli/core/graphql_client.py", line 154, in _execute_retry
    response = <http://self._session.post|self._session.post>(
  File "/opt/venv/lib/python3.9/site-packages/requests/sessions.py", line 635, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "/opt/venv/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/venv/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/opt/venv/lib/python3.9/site-packages/requests/adapters.py", line 578, in send
    raise ReadTimeout(e, request=request)
The above exception occurred during handling of the following exception:
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='empirico.agent.dagster.cloud', port=443): Read timed out. (read timeout=60)
  File "/opt/venv/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/opt/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
  File "/opt/venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/opt/venv/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/opt/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
  File "/opt/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 468, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/opt/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 357, in _raise_timeout
    raise ReadTimeoutError(
The above exception occurred during handling of the following exception:
socket.timeout: The read operation timed out
  File "/opt/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 466, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/opt/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 461, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/local/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.9/http/client.py", line 281, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.9/ssl.py", line 1242, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.9/ssl.py", line 1100, in read
    return self._sslobj.read(len, buffer)
The above exception occurred during handling of the following exception:
TypeError: getresponse() got an unexpected keyword argument 'buffering'
  File "/opt/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 457, in _make_request
    httplib_response = conn.getresponse(buffering=True)
j
We’re investigating if this was related to an infra upgrade on our side. Are they continuing?
Another question- have we had you set the
DAGSTER_CLOUD_IDEMPOTENT_STORE_EVENT
env variable? If not it might improve reliability here- should be set on the agent and the container context for your code location
z
I haven't seen it again since I posted this, although I saw it a few times prior to that. I'll look into getting that env var set