Two-Argument Activation Functions Learn Soft XOR Operations Like Cortical Neurons
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Kim, Juhyeon | - |
dc.contributor.author | Orhan, Emin | - |
dc.contributor.author | Yoon, Kijung | - |
dc.contributor.author | Pitkow, Xaq | - |
dc.date.accessioned | 2022-07-06T01:51:03Z | - |
dc.date.available | 2022-07-06T01:51:03Z | - |
dc.date.created | 2022-06-29 | - |
dc.date.issued | 2022-05 | - |
dc.identifier.issn | 2169-3536 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/hanyang/handle/2021.sw.hanyang/138366 | - |
dc.description.abstract | Neurons in the brain are complex machines with distinct functional compartments that interact nonlinearly. In contrast, neurons in artificial neural networks abstract away this complexity, typically down to a scalar activation function of a weighted sum of inputs. Here we emulate more biologically realistic neurons by learning canonical activation functions with two input arguments, analogous to basal and apical dendrites. We use a network-in-network architecture where each neuron is modeled as a multilayer perceptron with two inputs and a single output. This inner perceptron is shared by all units in the outer network. Remarkably, the resultant nonlinearities often produce soft XOR functions, consistent with recent experimental observations about interactions between inputs in human cortical neurons. When hyperparameters are optimized, networks with these nonlinearities learn faster and perform better than conventional ReLU nonlinearities with matched parameter counts, and they are more robust to natural and adversarial perturbations. | - |
dc.language | 영어 | - |
dc.language.iso | en | - |
dc.publisher | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC | - |
dc.title | Two-Argument Activation Functions Learn Soft XOR Operations Like Cortical Neurons | - |
dc.type | Article | - |
dc.contributor.affiliatedAuthor | Yoon, Kijung | - |
dc.identifier.doi | 10.1109/ACCESS.2022.3178951 | - |
dc.identifier.scopusid | 2-s2.0-85131719596 | - |
dc.identifier.wosid | 000808041900001 | - |
dc.identifier.bibliographicCitation | IEEE ACCESS, v.10, pp.58071 - 58080 | - |
dc.relation.isPartOf | IEEE ACCESS | - |
dc.citation.title | IEEE ACCESS | - |
dc.citation.volume | 10 | - |
dc.citation.startPage | 58071 | - |
dc.citation.endPage | 58080 | - |
dc.type.rims | ART | - |
dc.type.docType | Article | - |
dc.description.journalClass | 1 | - |
dc.description.isOpenAccess | Y | - |
dc.description.journalRegisteredClass | scie | - |
dc.description.journalRegisteredClass | scopus | - |
dc.relation.journalResearchArea | Computer Science | - |
dc.relation.journalResearchArea | Engineering | - |
dc.relation.journalResearchArea | Telecommunications | - |
dc.relation.journalWebOfScienceCategory | Computer Science, Information Systems | - |
dc.relation.journalWebOfScienceCategory | Engineering, Electrical & Electronic | - |
dc.relation.journalWebOfScienceCategory | Telecommunications | - |
dc.subject.keywordPlus | RESPONSES | - |
dc.subject.keywordAuthor | Neurons | - |
dc.subject.keywordAuthor | Computer architecture | - |
dc.subject.keywordAuthor | Training | - |
dc.subject.keywordAuthor | Task analysis | - |
dc.subject.keywordAuthor | Licenses | - |
dc.subject.keywordAuthor | Government | - |
dc.subject.keywordAuthor | Transformers | - |
dc.subject.keywordAuthor | Biological and artificial neurons | - |
dc.subject.keywordAuthor | activation functions | - |
dc.subject.keywordAuthor | exclusive-or operation | - |
dc.subject.keywordAuthor | adversarial robustness | - |
dc.identifier.url | https://ieeexplore.ieee.org/document/9785635 | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
222, Wangsimni-ro, Seongdong-gu, Seoul, 04763, Korea+82-2-2220-1365
COPYRIGHT © 2021 HANYANG UNIVERSITY.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.