LiNLNet: Gauging required nonlinearity in deep neural networks

Abstract

Feedforward deep neural networks (DNNs) commonly involve layer-wise linear operations and subsequent nonlinear operations, which are repeated through all layers. The nonlinear operations by nonlinear activations in each layer remarkably enhance the expressiveness of DNNs, resulting in the great success in a variety of application domains. Although the necessity of layer-wise nonlinear operations is agreed, the optimal nonlinearity for each layer in a given DNN is not clear. In this regard, we propose an easy-to-use method to layer-wise measure the optimal nonlinearity for a given DNN using its replica termed a linear-nonlinear network (LiNLNet). The key to the LiNLNet is the use of linear-nonlinear units (LiNLUs) whose degree of nonlinearity is parameterized by a trainable parameter p. The parameter p is shared among all LiNLUs in a given layer, thus indicating the layer-wise optimal nonlinearity. This method allows layer-level pruning such that the layers that do not require nonlinearity are merged into the subsequent layers, reducing computational complexity. For proofs of concept, we applied the proposed method to a MLP, AlexNet, VGG16, and ResNet18 on CIFAR-10 and ImageNet. The results commonly indicate the last hidden layer as a linear layer that may be merged into the output layer, reducing memory usage by 27% while maintaining the accuracy for LiNL-AlexNet on ImageNet.