Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Mamieva, Dilnoza | - |
dc.contributor.author | Abdusalomov, Akmalbek Bobomirzaevich | - |
dc.contributor.author | Kutlimuratov, Alpamis | - |
dc.contributor.author | Muminov, Bahodir | - |
dc.contributor.author | Whangbo, Taeg Keun | - |
dc.date.accessioned | 2023-07-23T12:40:29Z | - |
dc.date.available | 2023-07-23T12:40:29Z | - |
dc.date.created | 2023-07-21 | - |
dc.date.issued | 2023-06 | - |
dc.identifier.issn | 1424-8220 | - |
dc.identifier.uri | https://scholarworks.bwise.kr/gachon/handle/2020.sw.gachon/88595 | - |
dc.description.abstract | Methods for detecting emotions that employ many modalities at the same time have been found to be more accurate and resilient than those that rely on a single sense. This is due to the fact that sentiments may be conveyed in a wide range of modalities, each of which offers a different and complementary window into the thoughts and emotions of the speaker. In this way, a more complete picture of a person's emotional state may emerge through the fusion and analysis of data from several modalities. The research suggests a new attention-based approach to multimodal emotion recognition. This technique integrates facial and speech features that have been extracted by independent encoders in order to pick the aspects that are the most informative. It increases the system's accuracy by processing speech and facial features of various sizes and focuses on the most useful bits of input. A more comprehensive representation of facial expressions is extracted by the use of both low- and high-level facial features. These modalities are combined using a fusion network to create a multimodal feature vector which is then fed to a classification layer for emotion recognition. The developed system is evaluated on two datasets, IEMOCAP and CMU-MOSEI, and shows superior performance compared to existing models, achieving a weighted accuracy WA of 74.6% and an F1 score of 66.1% on the IEMOCAP dataset and a WA of 80.7% and F1 score of 73.7% on the CMU-MOSEI dataset. | - |
dc.language | 영어 | - |
dc.language.iso | en | - |
dc.publisher | MDPI | - |
dc.relation.isPartOf | SENSORS | - |
dc.title | Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features | - |
dc.type | Article | - |
dc.type.rims | ART | - |
dc.description.journalClass | 1 | - |
dc.identifier.wosid | 001015551000001 | - |
dc.identifier.doi | 10.3390/s23125475 | - |
dc.identifier.bibliographicCitation | SENSORS, v.23, no.12 | - |
dc.description.isOpenAccess | Y | - |
dc.identifier.scopusid | 2-s2.0-85164018247 | - |
dc.citation.title | SENSORS | - |
dc.citation.volume | 23 | - |
dc.citation.number | 12 | - |
dc.contributor.affiliatedAuthor | Mamieva, Dilnoza | - |
dc.contributor.affiliatedAuthor | Abdusalomov, Akmalbek Bobomirzaevich | - |
dc.contributor.affiliatedAuthor | Kutlimuratov, Alpamis | - |
dc.contributor.affiliatedAuthor | Whangbo, Taeg Keun | - |
dc.type.docType | Article | - |
dc.subject.keywordAuthor | CNN | - |
dc.subject.keywordAuthor | multimodal emotion recognition | - |
dc.subject.keywordAuthor | facial feature | - |
dc.subject.keywordAuthor | speech feature | - |
dc.subject.keywordAuthor | attention mechanism | - |
dc.subject.keywordPlus | SENTIMENT ANALYSIS | - |
dc.relation.journalResearchArea | Chemistry | - |
dc.relation.journalResearchArea | Engineering | - |
dc.relation.journalResearchArea | Instruments & Instrumentation | - |
dc.relation.journalWebOfScienceCategory | Chemistry, Analytical | - |
dc.relation.journalWebOfScienceCategory | Engineering, Electrical & Electronic | - |
dc.relation.journalWebOfScienceCategory | Instruments & Instrumentation | - |
dc.description.journalRegisteredClass | scie | - |
dc.description.journalRegisteredClass | scopus | - |
Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.
1342, Seongnam-daero, Sujeong-gu, Seongnam-si, Gyeonggi-do, Republic of Korea(13120)031-750-5114
COPYRIGHT 2020 Gachon University All Rights Reserved.
Certain data included herein are derived from the © Web of Science of Clarivate Analytics. All rights reserved.
You may not copy or re-distribute this material in whole or in part without the prior written consent of Clarivate Analytics.