In-sensor analog optoelectronic processing of concurrent event and memory signals for dynamic vision sensing

Abstract

Efficient dynamic vision requires capturing instantaneous changes and temporal context, yet existing image and event sensors rely on power-hungry digital processing. Here, we introduce an in-sensor dual-response architecture that concurrently generates analog event spikes and persistent memory tails. A prototype sensor integrates phosphor pairs with silicon photodiodes and transimpedance amplifiers to achieve microsecond- and millisecond-scale dual kinetics. Measurements during light-emitting diode replay reconstruct event frames that match software frame differences, while the slow channel behaves as a linear reservoir of motion history. A single memory frame fed to a convolutional neural network enables accurate classification of human actions (93.1%) and vehicle trajectories (98.0%), as well as speed estimation with errors of 2.15 km/h. Integration with a compressive optical neural network front end mapping 4900 inputs to 16 per frame yields 93.3% action classification accuracy. By eliminating analog-to-digital conversion and digital accumulation, this approach enables ultralow-latency, ultralow-power neuromorphic vision.