just noticing this but I sensor data isn't loading...
# dagster-plus
just noticing this but I sensor data isn't loading, and very few sensor materializations have kicked off the past couple days
Copy code
dagster._core.errors.DagsterUserCodeUnreachableError: The sensor tick timed out due to taking longer than 60 seconds to execute the sensor function. One way to avoid this error is to break up the sensor work into chunks, using cursors to let subsequent sensor calls pick up where the previous call left off.
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 807, in _process_api_request
    api_result = self._handle_api_request(
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 665, in _handle_api_request
    serialized_sensor_data_or_error = client.external_sensor_execution(
  File "/dagster/dagster/_grpc/client.py", line 388, in external_sensor_execution
    chunks = list(
  File "/dagster/dagster/_grpc/client.py", line 184, in _streaming_query
  File "/dagster/dagster/_grpc/client.py", line 135, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(
The above exception was caused by the following exception:
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1681406259.689411933","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":81,"grpc_status":4}"
  File "/dagster/dagster/_grpc/client.py", line 180, in _streaming_query
    yield from self._get_streaming_response(
  File "/dagster/dagster/_grpc/client.py", line 169, in _get_streaming_response
    yield from getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 426, in __next__
    return self._next()
  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 826, in _next
    raise self
I can't get into any sensor detail pages, but this is the error log that shows up on my agent
https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors#evaluation-interval one of the sensors must be taking longer than 60 seconds to evaluate
they do use cursors, this seems more like a bug
like I can't even get to the sensor page
and they were running fine until a few days ago, no changes
gotcha I can try to see if theres any errors in cloud
yeah it's really odd. there's evidence that some sensors are running, but they're still not accessible via the UI
Copy code
Could not load location kipptaf to check for sensors due to the following error: dagster._core.errors.DagsterImportError: Encountered ImportError: `cannot import name 'ResourceParam' from 'dagster'
is one error i see
but i only see that recently, maybe a pr to test upgrading dagster?
yup I resolved that earlier, the code location pod didn't reload properly when I upgraded
We're looking into the sensor timeouts now - I think the ImportError was likely pip/pypi getting confused while the 1.2.7 release was mid-release
thanks all!
this should be resolved now, had to tweak some things after DB index changes
yup, I can view them now