  Conventional image sensors digitize high-resolution images at fast frame
rates, producing a large amount of data that needs to be transmitted off the
sensor for further processing. This is challenging for perception systems
operating on edge devices, because communication is power inefficient and
induces latency. Fueled by innovations in stacked image sensor fabrication,
emerging sensor-processors offer programmability and minimal processing
capabilities directly on the sensor. We exploit these capabilities by
developing an efficient recurrent neural network architecture, PixelRNN, that
encodes spatio-temporal features on the sensor using purely binary operations.
PixelRNN reduces the amount of data to be transmitted off the sensor by a
factor of 64x compared to conventional systems while offering competitive
accuracy for hand gesture recognition and lip reading tasks. We experimentally
validate PixelRNN using a prototype implementation on the SCAMP-5
sensor-processor platform.

Computer Vision and Pattern Recognition