hello - recently we've started encountering frequent sensor timeouts; i think it's because we have a number of sensors that all run around the same time - they are all multi-asset sensors that are monitoring exclusive subsets of assets. I've been able to work around a bit by simply toggling them to introduce a bit of stagger, but is there a way to write these sensors in a more performant way?

https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors#sensor-optimizations-using-cursors might help depending on how the sensor works

hmm - we're on serverless, is that a config somewhere?

ah unfortunately not

ok - can confirm that we're using cursors (we just wait until all the assets in the selection have new materialization events and call _`advance_all_cursors`_ - a bump up in resources would be great.

ok I will get that filled, it might not go out till next week thursday? is that ok?

that is fine! thank you!

Hi Leo, due to a configuration issue it looks like the increase may have been recently reverted. I am fixing and the increase should be applied again in a few minutes.

hello @Shalabh Chaturvedi - is there any chance that the increase was reverted again? Starting around noon pacific time yesterday we have been encountering sensor timeouts - and re-deploying the code location does not seem to be helping

Hi Leo - I checked the configuration and it is still running with the increased resources. The max CPU usage is at 100% - do you have some heavy computation in the sensor that regularly pegs the cpu?

we have about 15 multi-asset sensors that run every 5 minutes, they do minimal computation of their own though - a few hit the slack API - but they are generally listening on an Asset Selection and they may trigger other assets or jobs

@Leo Qin this doesn't answer your question about the sensors, but we believe we have implemented a fix for the GraphQLStorageError that you reported - the same asset run that was triggering it before should now be able to complete (let us know if that's different from what you are seeing)

@daniel to add more context to the multi-asset sensor issue @Leo Qin mentioned: the only thing the sensors do is fetch the latest materialization event for a set of assets and evaluate if all assets have a new materialization event by comparing it to the cursor. Is it possible the sensors time out because it takes to long to retrieve the latest materialization event for each asset? Thanks for the help here 🙏

That's very possible - that exact query (getting the most recent materialization event for a given asset) running slower than expected was actually what was causing the other GraphQLStorageError on this thread - we are seeing it take up to 15 seconds for certain assets which was hitting a timeout. We're going to be working on a better indexing scheme for these queries that should help these sensors run faster as well.

hello - is there any update on when these indexing changes are going to happen? Is it soon, or would we get mileage out of trying some workarounds on our end?

Hey Leo. I just flipped on the faster queries. It looks like a bunch of your sensors are still failing though. Are you seeing any differences on your end?

@prha - still seeing a few failures due to timeout, yeah. We have a group of 14 sensors that run every 5 minutes, and it seems like some, but not all of them can run fast enough to succeed within the 60 second timeout

I’m digging into why they’re timing out… can you share a link to one of your multi asset sensors?

PM'd

hi dagster team, we're experiencing sensor timeouts again, would you be able to take a look? Thanks

Hi Henri - do you know if the sensor evaluation is slower for the failing sensor due to slower external queries? I'll look into increasing the resources for your code server, however this will only help if it is actually cpu or memory pegged. Otherwise the sensor may need to refactored to do less work. Another feature that we hope to land in the next month or so is the ability to increase sensor timeouts, which should help.

@Shalabh Chaturvedi most of sensors (~90%) don't make external queries and are multi asset sensors