Facial electromyogram-based emotion recognition for virtual reality applications using machine learning classifiers trained on posed expressions

Abstract

Recognition of human emotions holds great potential for various daily-life applications. With the increasing interest in virtual reality (VR) technologies, numerous studies have proposed new approaches to integrating emotion recognition into VR environments. However, despite recent advancements, camera-based emotion-recognition technology faces critical limitations due to the physical obstruction caused by head-mounted displays (HMDs). Facial electromyography (fEMG) offers a promising alternative for human emotion-recognition in VR environments, as electrodes can be readily embedded in the padding of commercial HMD devices. However, conventional fEMG-based emotion recognition approaches, although not yet developed for VR applications, require lengthy and tedious calibration sessions. These sessions typically involve collecting fEMG data during the presentation of audio-visual stimuli for eliciting specific emotions. We trained a machine learning classifier using fEMG data acquired while users intentionally made posed facial expressions. This approach simplifies the traditionally time-consuming calibration process, making it less burdensome for users. The proposed method was validated using 20 participants who made posed facial expressions for calibration and then watched emotion-evoking video clips for validation. The results demonstrated the effectiveness of our method in classifying high- and low-valence states, achieving a macro F1-score of 88.20%. This underscores the practicality and efficiency of the proposed method. To the best of our knowledge, this is the first study to successfully build an fEMG-based emotion-recognition model using posed facial expressions. This approach paves the way for developing user-friendly interface technologies in VR-immersive environments.