Multimodal fusion for cued Speech language recognition

In this paper, a novel method for Cued Speech language recognition is proposed. A multimodal processing framework is developed for the efficient fusion of the modalities under consideration. The robust feature extraction from the audio, lip shape and gesture modalities is also discussed. Moreover, the inherent correlation among the signals is modelled using a modified Coupled Hidden Markov Model. The contribution of each modality to the fused decision is modified by assessing its reliability and assigning a weighting factor to improve the accuracy of the recognized words. The method is experimentally evaluated and is shown to outperform methods that rely only on the perceivable information to infer the transmitted messages.