Hello, I was hoping to find out how to increase/remove the grpc timeout for sensors - we are using a dagster sensor that begins by getting a list of files in a very large directory and I am finding that the grpc deadline is being exceeded before it can yield any runs

Hi Carter, we don't have a great way of tweaking the 60 second timeout currently - we could make it configurable but eventually you'd run into a heartbeat timeout with the daemon if the request takes long enough. We do have some longer term plans to make the daemon more async/parallelizable which will help with this class of problems. In the short term while we figure out our plans there - Is there any way to chunk or limit checkpoint the sensor call so that it can limit the amount of files it needs to look at on each request? Sensors have cursor functionality that can help in cases like this: https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors#sensor-optimizations-using-cursors

Hi Daniel, thanks for the follow-up. I was familiar with the usage of cursors/similar devices, the issue was that the directory I was scanning was large enough that just a listdir call would cause the sensor to timeout so prior to any cursor loading - moving to scandir/lazy evaluation did solve the problem and once the directory files could be scanned everything worked perfectly