Understanding and Reducing Weight-Load Overhead of Systolic Deep Learning Accelerators

Abstract

As an energy-efficient computing engine for deep neural network inference, 2D systolic array architectures have been widely adopted in modern deep learning accelerators. However, despite high compute density and energy-efficient data passing, systolic accelerators suffer a non-Trivial overhead of loading data stationed inside their local register file. This loading overhead becomes a critical issue when a frequent reload of stationary data (e.g., weight parameters) is required. This paper proposes a simple yet practical SW-HW co-optimization that reverses the weight-load order and adds a dedicated path for weight-load. On diverse deep learning applications, the proposed method reduces the weight-load overhead and achieves up to 1.8× speedup with 40% energy savings.