Hi guys, we’re having a strange issue with Dagster

0.14.19

on k8 I’ve got 2 Sensors configured. One runs fine but the second one fails with this error:

dagster.core.errors.DagsterUserCodeUnreachableError: Could not reach user code server
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/sensor.py", line 323, in execute_sensor_iteration
    sensor_debug_crash_flags,
  File "/usr/local/lib/python3.7/site-packages/dagster/daemon/sensor.py", line 358, in _evaluate_sensor
    state.instigator_data.cursor if state.instigator_data else None,
  File "/usr/local/lib/python3.7/site-packages/dagster/core/host_representation/repository_location.py", line 795, in get_external_sensor_execution_data
    cursor,
  File "/usr/local/lib/python3.7/site-packages/dagster/api/snapshot_sensor.py", line 61, in sync_get_external_sensor_execution_data_grpc
    cursor=cursor,
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 305, in external_sensor_execution
    sensor_execution_args
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 124, in _streaming_query
    raise DagsterUserCodeUnreachableError("Could not reach user code server") from e
The above exception was caused by the following exception:
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1670572825.385825854","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":81,"grpc_status":4}"
>
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/client.py", line 122, in _streaming_query
    yield from response_stream
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 426, in __next__
    return self._next()
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 826, in _next
    raise self

Besides for that error I can’t find any other error logs from my user code grpc server or from the dagster daemon. Also it doesn’t make sense that the user-code server is unreachable since the second sensor works fine.. Any ideas?

do you know how long each evaluation is taking? Wondering if you're running into something similar to here: https://dagster.slack.com/archives/C01U954MEER/p1669818604570039