Hey. Im writing a sensor that loads 1000s of files from a local filesystem to gcs. Ive setup a job that loads file, saves the file to gcs, deletes the file from local system I wrote a sensor that yields a run request for each file. It keeps timing out before all of the files get triggered. Is there a better way to manage this?

If you’re set on doing this with a sensor, you could use a cursor to chunk the files into a more reasonable number per sensor iteration. But I wonder if you should just be yielding a single run request for the particular batch of files when new files appear, rather than doing it for every single file