https://dagster.io/ logo
Join Slack
Powered by
Code locations are timing out after upgrading to 1...
# ask-community
c
Code locations are timing out after upgrading to 1.4.0
Copy code
Copy
dagster._core.errors.DagsterUserCodeUnreachableError: User code server request timed out due to taking longer than 60 seconds to complete.

  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 674, in _update_location_data
    upload_location_data=self._get_upload_location_data(
  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 586, in _get_upload_location_data
    external_repository_chunks = list(
  File "/dagster/dagster/_grpc/client.py", line 346, in streaming_external_repository
    for res in self._streaming_query(
  File "/dagster/dagster/_grpc/client.py", line 184, in _streaming_query
    self._raise_grpc_exception(
  File "/dagster/dagster/_grpc/client.py", line 135, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(

The above exception was caused by the following exception:
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-07-21T01:41:36.546066651+00:00", grpc_status:4}"
>

  File "/dagster/dagster/_grpc/client.py", line 180, in _streaming_query
    yield from self._get_streaming_response(
  File "/dagster/dagster/_grpc/client.py", line 169, in _get_streaming_response
    yield from getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 475, in __next__
    return self._next()
  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 881, in _next
    raise self
3/4 of mine typically update pretty quickly before, but now they all die
this error came from another after ~10+ minutes of waiting
Copy code
Copy
Exception: Timed out after waiting 360s for server kippcamden-prod-221af3.dagster-cloud:4000.

Debug information for pod kippcamden-prod-221af3-675fbb9984-jpzt8:

Pod status: Running
Container 'dagster' status: Ready

No logs in pod.

No warning events for pod.

For more information about the failure, run `kubectl describe pod kippcamden-prod-221af3-675fbb9984-jpzt8` or `kubectl describe deployment kippcamden-prod-221af3` in your cluster.

Most recent connection error: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE

Stack Trace:
  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1745, in _wait_for_server_process
    client.ping("")
  File "/dagster/dagster/_grpc/client.py", line 190, in ping
    res = self._query("Ping", api_pb2.PingRequest, echo=echo)
  File "/dagster/dagster/_grpc/client.py", line 157, in _query
    self._raise_grpc_exception(
  File "/dagster/dagster/_grpc/client.py", line 140, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(

The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.232.131.248:4000: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.232.131.248:4000: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-07-21T01:48:09.680657442+00:00"}"
>

Stack Trace:
  File "/dagster/dagster/_grpc/client.py", line 155, in _query
    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
  File "/dagster/dagster/_grpc/client.py", line 130, in _get_response
    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable


  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1343, in _reconcile
    self._wait_for_new_server_ready(
  File "/dagster-cloud/dagster_cloud/workspace/kubernetes/launcher.py", line 365, in _wait_for_new_server_ready
    self._wait_for_dagster_server_process(
  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1723, in _wait_for_dagster_server_process
    self._wait_for_server_process(
  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1758, in _wait_for_server_process
    raise Exception(
j
Looks like they’re up now? Did something change or did they just eventually pass on retry
Ah you reverted to 1.3
👍 1
11 Views
Open in Slack
PreviousNext
Powered by