Splitting of Composite Neural Networks via Proximal Operator With Information Bottleneck
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Han, Sang-Il | - |
dc.contributor.author | Nakamura, Kensuke | - |
dc.contributor.author | Hong, Byung-Woo | - |
dc.date.accessioned | 2024-03-06T08:30:17Z | - |
dc.date.available | 2024-03-06T08:30:17Z | - |
dc.date.issued | 2024 | - |
dc.identifier.issn | 2169-3536 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/cau/handle/2019.sw.cau/72668 | - |
dc.description.abstract | Deep learning has achieved efficient success in the field of machine learning, made possible by the emergence of efficient optimization methods such as Stochastic Gradient Descent (SGD) and its variants. Simultaneously, the Information Bottleneck theory (IB) has been studied to train neural networks, aiming to enhance the performance of optimization methods. However, previous works have focused on their specific tasks, and the effect of the IB theory on general deep learning tasks is still unclear. In this study, we introduce a new method inspired by the proximal operator, which sequentially updates the neural network parameters based on the defined bottleneck features between the forward and backward networks. Unlike the conventional proximal-based methods, we consider the second-order gradients of the objective function to achieve better updates for the forward networks. In contrast to SGD-based methods, our approach involves accessing the network's black box, and incorporating the bottleneck feature update process into the parameter update process. This way, from the perspective of the IB theory, the data is well compressed up to the bottleneck feature, ensuring that the compressed information maintains sufficient mutual information up to the final output. To demonstrate the performance of the proposed approach, we applied the method to various optimizers with several tasks and analyzed the results by training on both the MNIST dataset and CIFAR-10 dataset. We also conducted several ablation studies by modifying the components of the proposed algorithm to further validate its performance. | - |
dc.format.extent | 11 | - |
dc.language | 영어 | - |
dc.language.iso | ENG | - |
dc.publisher | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC | - |
dc.title | Splitting of Composite Neural Networks via Proximal Operator With Information Bottleneck | - |
dc.type | Article | - |
dc.identifier.doi | 10.1109/ACCESS.2023.3346697 | - |
dc.identifier.bibliographicCitation | IEEE ACCESS, v.12, pp 157 - 167 | - |
dc.description.isOpenAccess | Y | - |
dc.identifier.wosid | 001135226900001 | - |
dc.identifier.scopusid | 2-s2.0-85181571998 | - |
dc.citation.endPage | 167 | - |
dc.citation.startPage | 157 | - |
dc.citation.title | IEEE ACCESS | - |
dc.citation.volume | 12 | - |
dc.type.docType | Article | - |
dc.publisher.location | 미국 | - |
dc.subject.keywordAuthor | Linear programming | - |
dc.subject.keywordAuthor | Deep learning | - |
dc.subject.keywordAuthor | Task analysis | - |
dc.subject.keywordAuthor | Mutual information | - |
dc.subject.keywordAuthor | Training | - |
dc.subject.keywordAuthor | Optimization methods | - |
dc.subject.keywordAuthor | Biological neural networks | - |
dc.subject.keywordAuthor | information bottleneck | - |
dc.subject.keywordAuthor | stochastic gradient descent | - |
dc.subject.keywordAuthor | proximal algorithm | - |
dc.relation.journalResearchArea | Computer Science | - |
dc.relation.journalResearchArea | Engineering | - |
dc.relation.journalResearchArea | Telecommunications | - |
dc.relation.journalWebOfScienceCategory | Computer Science, Information Systems | - |
dc.relation.journalWebOfScienceCategory | Engineering, Electrical & Electronic | - |
dc.relation.journalWebOfScienceCategory | Telecommunications | - |
dc.description.journalRegisteredClass | scie | - |
dc.description.journalRegisteredClass | scopus | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
84, Heukseok-ro, Dongjak-gu, Seoul, Republic of Korea (06974)02-820-6194
COPYRIGHT 2019 Chung-Ang University All Rights Reserved.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.