Gesture Recognition for Note Generation in VR Rhythm Game

Abstract

This paper describes a gesture recognition algorithm for note generation in virtual reality (VR) rhythm game. VR rhythm game is a game in which a user has to move according to the rhythm or music in a virtual environment and usually takes the form of hitting objects, called notes, flying to the user with hands or tools. Notes need be generated to match the rhythm in the game. However, manual generation is often a time-consuming and tedious work. Previous research has focused mainly on the development of algorithms that automatically generate notes from music. However, notes made according to music can improve a sense of rhythm, but they do not have spatial characteristics, so it is difficult to induce spatially meaningful movements. In order to improve the drawback, we propose a method for note generation in a VR rhythm game from a dance consisting of hand taps. Dynamic Time Warning (DTW) method was applied to recognize the hand taps and determine the point at which to position the note. The experimental results show that our method accurately recognizes hand tap gestures and their location, and generates notes accordingly. © 2021 IEEE.