Hi, I'm hitting GRPC size limits when executing pipelines from a sensor. The sensor collect ~4k records of 500bytes/ea from which it yields a run. Going too much over 4k gives the error. Is there anyway to use IOManager to workaround this issue? eg materialise assets using an IOManager in the sensor and have the first solid in the pipeline pick up from that rather than sending all the data through grpc? cheers

Hi Oliver - is there a reason the sensor needs to collect all the records (as opposed to triggering a pipeline that collects the records in its first solid?) The design thinking with sensors was that they would typically be comparatively lightweight, and just be responsible for deciding whether a run should happen - with the actual heavy lifting happening in the pipelines themselves.

I'm using Kinesis which doesn't have a facility to check for metadata, additionally if kinesis fetches 10mb of data in a single call subsequent calls within the following 5 seconds fail with an exception by design. I will try implement the fetch in both sensor and the pipeline and see what happens