The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

I am experiencing some problems with our sensor.
```dagster._core.errors.DagsterUserCodeUnreachableError: dagster._core.errors.DagsterUserCodeUnreachableError: The sensor tick timed out due to taking longer than 60 seconds to execute the sensor function.```
But we have a similar sensor that does the same but on a different dataset that is working fine. When running locally the sensor function completes fast enough.

tried adding logs but those don’t appear.

We should do a better job of exposing those sensor logs back to you in Cloud - I’ll make sure my team is aware of it.

I do see a log line internally that implies the sensor is finding a lot of S3 keys (I’ll DM you the specifics) - this might be an issue where it’s struggling to finish in time the first time it runs? And chunking the work further so it only ever processes _n_ keys and then moves onto the next tick might get this working again.

I'm experiencing the same issue with a sensor I have deployed that reads files from S3

My S3 bucket contains ~120K files and I ran `get_s3_keys` over the bucket (isolated in a notebook), which took ~23 seconds to get all keys. Though I experience a timeout in Dagster Cloud and no runs are requested. We might be able to get around the issue by editing the following <https://github.com/dagster-io/dagster/blob/5238e6faf8cfe8f0b8208f7e4ef1facde888bcf9/python_modules/libraries/dagster-aws/dagster_aws/s3/sensor.py#L7|code> to only extract `N` keys from S3 as you mentioned <@U01EK4V7BM4>

At that point, you’ll probably want to switch from listing the files to something like a bucket notification. And have your sensor pop messages from a queue:

<https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html>

Unless your keys have a matching alphabetical and timestamp sort, it’s not possible with the S3 api alone to limit to only recently created keys when listing.