Accurate and High-Throughput Analog-Digital DNN Acceleration using Sub-Network Scheduling

Abstract

Analog Computing-in-Memory (ACiM) devices perform matrix operations directly within memory arrays, offering high throughput for deep neural network inference. At the same time, they are susceptible to various noise sources, which reduces the computational accuracy. This inherent trade-off limits the adoption of ACiM devices as a stand-alone accelerator. In this paper, we propose the first heterogeneous inference-serving framework that coordinates ACiM hardware and conventional digital processors during inference. The system dynamically partitions each neural network into an analog sub-network executed on high-throughput ACiM devices and a digital sub-network handled by precise digital devices. Then, it integrates a transition layer and a custom training strategy to maintain accuracy across varying analog-digital partitions. Also, it adopts a two-level scheduler to adjust the partition ratio at runtime in response to varying query demands. We evaluate our system using five network models to demonstrate the benefits of our system.