Online whole-word and stroke-based modeling for hand-written letter recognition in in-car environments

A finger-written, camera-based, hand gesture recognition framework for English letters in an in-vehicle environment based on Hidden Markov models is proposed. Due to the nature of the constrained hand-movement situations on the steering column, we are confronted with at least two challenging research issues, namely varying illumination conditions and noisy hand gestures. The first difficulty is alleviated by utilizing the contrast for background-foreground separation and skin model adaptation. We also adopt sub-letter stroke modeling to reduce the noisy frames of the beginning and ending parts of the letter gestures followed by the trajectory re-normalization. Moreover, the geometric relationship between letter pairs is also utilized to distinguish highly confusable letters. Finally, score fusion between whole-letter and sub-stroke models can be used to further improve the performance. When compared with the baseline system with simple features, our experimental results show that an overall relative error reduction of 66.03% can be achieved by integrating the above four new pieces of information.