Improved Multi-Modal Recognition Interface for Intelligent HCI Based on Speech and the KSSL Recognition

Desktop PC and wire communications net-based traditional studies on pattern recognition and multimodal interaction have some restrictions (e.g. limitation of motion, conditionality in space and so on) and general problems according to using of the vision technologies for recognition and representation of the hap tic-gesture information. In this paper, we propose and implement multi-modal recognition interface (hereinafter, MMRI) integrating speech using voice-XML based on WWW and the post wearable PC-based gesture, it have purposes that recognizes and represents the Korean Standard Sign Language (hereinafter, KSSL) which is a dialogue system and interactive elements in the Korean deaf communities. The advantages of our approach are as follows: 1) it improves efficiency of the MMRI input module according to the technology of wireless communication, 2) it shows higher recognition performance than uni-modal recognition system (using gesture or speech), 3) it recognizes and represents continuous sign language of users with flexibility in real time and can offer to user a wider range of personalized and differentiated information using the MMRI more effectively. Experimental results, the MMRI deduces an average recognition rate of 96.1% about significant, dynamic and continuous the KSSL and speech of various users