Hi team it seems that with 1 3 1 the code server request tim dagster #ask-community

Hi team, it seems that with 1.3.1, the code server...

Hebo Yang

05/03/2023, 7:36 PM

Hi team, it seems that with 1.3.1, the code server request timeout problem in Dagit is coming back. Somehow 1.2.6 doesn’t have this problem. Do you know any changes between these versions that might have caused this please?

Hebo Yang

05/03/2023, 7:36 PM

Copy code

/usr/local/lib/python3.7/site-packages/dagster/_core/workspace/context.py:593: UserWarning: Error loading repository location fabricator:dagster._core.errors.DagsterUserCodeUnreachableError: User code server request timed out due to taking longer than 60 seconds to complete.

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/workspace/context.py", line 588, in _load_location
    location = self._create_location_from_origin(origin)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/workspace/context.py", line 508, in _create_location_from_origin
    return origin.create_location()
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/host_representation/origin.py", line 325, in create_location
    return GrpcServerCodeLocation(self)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/host_representation/code_location.py", line 632, in __init__
    self,
  File "/usr/local/lib/python3.7/site-packages/dagster/_api/snapshot_repository.py", line 29, in sync_get_streaming_external_repositories_data_grpc
    repository_name,
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 348, in streaming_external_repository
    defer_snapshots=defer_snapshots,
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 185, in _streaming_query
    e, timeout=timeout, custom_timeout_message=custom_timeout_message
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 138, in _raise_grpc_exception
    ) from e

The above exception was caused by the following exception:
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1683137122.386091274","description":"Error received from peer ipv4:10.4.248.159:3030","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 181, in _streaming_query
    method, request=request_type(**kwargs), timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 169, in _get_streaming_response
    yield from getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 426, in __next__
    return self._next()
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 809, in _next
    raise self

  location_name=location_name, error_string=error.to_string()

daniel

05/03/2023, 8:14 PM

i'm not aware of any specific changes that would cause this. Some options: • Increase the DAGSTER_GRPC_TIMEOUT_SECONDS env var in dagit and the daemon to band-aid over the problem - the default is 60 seconds which is typically plenty • Run py-spy https://github.com/benfred/py-spy on the user code deployment while this is happening to generate more information about what's running slow • Send over a piece of code that reproduces the problem and we'd be happy to take a look • You could try upgrading the user code to the latest version, although it's hard to know if that will fix it without knowing where it's timing out

Hebo Yang

05/03/2023, 9:53 PM

Got it. Thanks Daniel! Let me try these in order:)

129 Views

Open in Slack

Previous Next