HOLOGESTURE: A MULTIMODAL DATASET FOR HAND GESTURE RECOGNITION ROBUST TO HAND TEXTURES ON HEAD-MOUNTED MIXED-REALITY DEVICES

Abstract

While the recent development of high performance mixed-reality (MR) devices is enabling its use in medical and industrial domains, this requires hand gesture recognition to be robust to different textures inflicted by gloves often worn for hygiene and safety purposes. Unfortunately, most existing hand gesture datasets are not captured using recent commercial MR devices, and none addresses the issue of wearing gloves in gesture recognition. We aim to fill these gaps by introducing a new dataset called HoloGesture, which comprises gesture clips acquired with and without latex gloves using Microsoft HoloLens 2. To leverage the multimodal nature of the latest MR device, we go beyond simply stacking RGB and depth frames and provide spatially aligned depth and RGB images. Experimental results show that i) incorporating gloves for training enhances robustness of gesture recognition to different hand textures and ii) spatial alignment of RGB and depth images enhances the recognition accuracy. Our code and dataset can be found at https://github.com/hellojpark/hologesture.