19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
just noticing this but I sensor data isn't loading...
# dagster-plusc
Charlie Bini
04/13/2023, 5:18 PMjust noticing this but I sensor data isn't loading, and very few sensor materializations have kicked off the past couple days
Charlie Bini
04/13/2023, 5:18 PM Copy code
dagster._core.errors.DagsterUserCodeUnreachableError: The sensor tick timed out due to taking longer than 60 seconds to execute the sensor function. One way to avoid this error is to break up the sensor work into chunks, using cursors to let subsequent sensor calls pick up where the previous call left off.
File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 807, in _process_api_request
api_result = self._handle_api_request(
File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 665, in _handle_api_request
serialized_sensor_data_or_error = client.external_sensor_execution(
File "/dagster/dagster/_grpc/client.py", line 388, in external_sensor_execution
chunks = list(
File "/dagster/dagster/_grpc/client.py", line 184, in _streaming_query
self._raise_grpc_exception(
File "/dagster/dagster/_grpc/client.py", line 135, in _raise_grpc_exception
raise DagsterUserCodeUnreachableError(
The above exception was caused by the following exception:
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1681406259.689411933","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":81,"grpc_status":4}"
>
File "/dagster/dagster/_grpc/client.py", line 180, in _streaming_query
yield from self._get_streaming_response(
File "/dagster/dagster/_grpc/client.py", line 169, in _get_streaming_response
yield from getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 426, in __next__
return self._next()
File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 826, in _next
raise selfCharlie Bini
04/13/2023, 5:19 PMI can't get into any sensor detail pages, but this is the error log that shows up on my agent
j
Joe
04/13/2023, 5:45 PMhttps://docs.dagster.io/concepts/partitions-schedules-sensors/sensors#evaluation-interval
one of the sensors must be taking longer than 60 seconds to evaluate
Joe
04/13/2023, 5:46 PMyou could try to refactor them to use cursors https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors#sensor-optimizations-using-cursors
c
Charlie Bini
04/13/2023, 5:46 PMthey do use cursors, this seems more like a bug
Charlie Bini
04/13/2023, 5:47 PMlike I can't even get to the sensor page
Charlie Bini
04/13/2023, 5:47 PMand they were running fine until a few days ago, no changes
j
Joe
04/13/2023, 5:48 PMgotcha I can try to see if theres any errors in cloud
c
Charlie Bini
04/13/2023, 5:51 PMyeah it's really odd. there's evidence that some sensors are running, but they're still not accessible via the UI
j
Joe
04/13/2023, 5:52 PM Copy code
Could not load location kipptaf to check for sensors due to the following error: dagster._core.errors.DagsterImportError: Encountered ImportError: `cannot import name 'ResourceParam' from 'dagster'Joe
04/13/2023, 5:53 PMbut i only see that recently, maybe a pr to test upgrading dagster?
c
Charlie Bini
04/13/2023, 5:53 PMyup I resolved that earlier, the code location pod didn't reload properly when I upgraded
d
daniel
04/13/2023, 5:55 PMWe're looking into the sensor timeouts now - I think the ImportError was likely pip/pypi getting confused while the 1.2.7 release was mid-release
c
Charlie Bini
04/13/2023, 5:59 PMthanks all!
a
alex
04/13/2023, 7:45 PMthis should be resolved now, had to tweak some things after DB index changes
c
Charlie Bini
04/13/2023, 7:50 PMyup, I can view them now
4 Views