GEBA: Gradient-Error-Based Approximation of Activation Functions

Abstract

Computing-in-memory (CIM) macros aiming at accelerating deep learning operations at low power need activation function (AF) units on the same die to reduce their host-dependency. Versatile CIM macros need to include reconfigurable AF units at high precision and high efficiency in hardware usage. To this end, we propose the gradient-error-based approximation (GEBA) of AFs, which approximates various types of AFs in discrete input domains at high precision. GEBA reduces the approximation error by ca. 49.7%, 67.3%, 81.4%, 60.1% (for sigmoid, tanh, GELU, swish in FP32), compared with the uniform input-based approximation using the same memory as GEBA.