경량 딥러닝 모델의 초저정밀도 양자화를 위한 학습 방식의 개선

Abstract

Deep Learning Model Quantization is the most effective technique to make a model much lighter and cost efficient in terms of computation. Above many quantization algorithms, PROFIT[1] is a specialized algorithm for sub 4-bit mobile network quantization. But this method has sudden accuracy degradation in 2-bit width precision. In this paper, we propose a better training method to deal with this problem in 2-bit weight quantization. We adopt AIWQ, a metric for the activation’s instability induced by weight quantization [1] and make threshold value with this metric. Using threshold value, we stop training some quantized layers which have high sensitivity to weight quantization and fine-tune the rest of the quantized layers with different learning rate and scheduler. With this advanced training method, we improved 2-bit weight quantization accuracy of light deep learning models including EfficientNetB0 and MobilenetV2.