Understanding of hand motion in sentence level using HMM

Abstract

This paper proposes a new method that can recognize a sequence of hand motion expressing a sentence in sign language. Recognition procedure is divided into two steps: separation of the sequence of hand motions into the sub-sequences each of which expresses one word and combination of the words in order to construct a sentence having a meaning. In the first step, sequences of hand motion images are segmented by testing the continuity of the hand motions and by the multiscale image segmentation scheme. The trajectory of the hand motions are estimated by the affine transformation. Each sign in the sentence is represented by the extended chereme analysis model and each chereme is represented by the status vector for determining the transition in the HMM. In the second step, each sentence is also represented by the HMM. The Viterbi algorithm and context-dependent HMM are used to find the best state sequence in the HMM. The proposed algorithm has been tested with ten sequences of images, each of which expresses a sentence in Korean sign language. The experimental results have shown that the proposed algorithm can separate the sentence level image sequence into the word level sub-sequences with the success rate of 75% on average and recognize the sentence with the success rate of 80%.