Neural Tangent Kernel Analysis of Deep Narrow Neural Networks
- Authors
- Lee, J.; Choi, J.Y.; Ryu, E.K.; No, A.
- Issue Date
- 2022
- Publisher
- ML Research Press
- Citation
- Proceedings of Machine Learning Research, v.162, pp.12282 - 12351
- Journal Title
- Proceedings of Machine Learning Research
- Volume
- 162
- Start Page
- 12282
- End Page
- 12351
- URI
- https://scholarworks.bwise.kr/hongik/handle/2020.sw.hongik/31495
- ISSN
- 2640-3498
- Abstract
- The tremendous recent progress in analyzing the training dynamics of overparameterized neural networks has primarily focused on wide networks and therefore does not sufficiently address the role of depth in deep learning. In this work, we present the first trainability guarantee of infinitely deep but narrow neural networks. We study the infinite-depth limit of a multilayer perceptron (MLP) with a specific initialization and establish a trainability guarantee using the NTK theory. We then extend the analysis to an infinitely deep convolutional neural network (CNN) and perform brief experiments. Copyright © 2022 by the author(s)
- Files in This Item
- There are no files associated with this item.
- Appears in
Collections - College of Engineering > School of Electronic & Electrical Engineering > 1. Journal Articles
![qrcode](https://api.qrserver.com/v1/create-qr-code/?size=55x55&data=https://scholarworks.bwise.kr/hongik/handle/2020.sw.hongik/31495)
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.