Bionic AR: Standalone Augmented Reality System Controlled by Multimodal Biosignals

Abstract

Augmented reality (AR) technology can elevate user experiences by facilitating intuitive and immersive communication. Commercial AR devices typically rely on hand-held controllers or cameras to generate control commands; however, neither of these approaches is feasible when hand usage is limited. To address this limitation, we introduce a novel stand-alone AR system“, Bionic AR.” This system integrates a head-mounted display (HMD) with biosignal-based human–computer interface (HCI) technology. By embedding passive dry electrodes into the proposed AR headset in contact areas such as the occipital area, brow ridge, and cheeks, the system can simultaneously record electroencephalography (EEG), electrooculography (EOG), and facial electromyography (fEMG) data. The acquired multimodal biosignals were utilized to implement a steady-state visual evoked potential-based brain–computer interface (BCI), an eye-writing pattern recognition system, a facial expression recognition system, and a silent speech recognition system. In real-time experiments, each HCI module achieved accuracies exceeding 98%, underscoring the feasibility and practical potential of the proposed Bionic AR system.