AR-Based Multi-sensory Human Interaction in Rehabilitation: Enhancing User Experience in Exercise

Abstract

Augmented Reality (AR) is an innovative technology that integrates virtual content with the real world, maximizing user immersion and interaction. Although AR applications are increasingly expanding into the medical and rehabilitation fields, interactions with virtual objects, particularly from a multi-sensory perspective, remain limited compared to real-world object interactions. This leads to reduced user immersion and decreased user performance in rehabilitation exercises. This study presents the development of an AR-glass-based rehabilitation exercise service designed to enhance multi-sensory interaction with virtual objects. The proposed system is based on an upper limb rehabilitation exercise that involves diagonal arm lifts and incorporates visual, auditory, and haptic feedback to provide users with clear guidance and enhanced immersion. In addition, gamification elements were integrated to encourage user voluntary engagement. The system was progressively enhanced through iterative user testing and refinement. Experimental results demonstrated that multi-sensory interaction with virtual objects effectively reduces the sense of heterogeneity between virtual and real-world objects while improving user performance in rehabilitation exercises. This study is expected to be a foundational reference for future research on immersive multi-sensory feedback-based interfaces in medical and rehabilitation applications.