Radar Based Air-Writing Gesture Recognition Using a Novel Multi-Stream CNN Approach

Abstract

Hand gestures being a convenient and natural way of communication, is getting huge attention for human computer interface designs. Amongst these gestures, detecting mid-air writing is one of the most promising application. Existing radar based solutions often performs the mid-air writing recognition by tracking the hand trajectory using multiple mono-static or bi-static radars. This paper presents a Multi-Stream Convolutional Neural Network (MS-CNN) based in-air digits recognition method using a Frequency Modulated Continuous wave (FMCW) radar. With one FMCW radar comprising of two receiving channels, a novel three stream CNN network with Range-time, Doppler-time and Angle-time spectrograms as inputs is constructed and the features are fused together in later stage before making a final recognition. Unlike the traditional CNN, MS-CNN with multiple independent input layers enable the creation of multi-dimensional deep-learning model for FMCW radars. Twelve human volunteers were invited to writing the digit from zero to nine in the air in both home and lab environment. The three stream CNN architecture based air-writing for digits has shown a promising accuracy of 95%. Comparison of proposed MS-CNN system was made with 45 different variants of CNN and preliminary results shows that MS-CNN outperforms the other traditional CNN architectures for air-writing application. The gestures radar data has also been made available to the research community.