A Pattern Recognition-Based Multi-Modal HCI Using 3 Sensory Channels

This paper describes a PDA-based MMCR (Multi- Modal Command Recognizer for PDA control and handling) using double-touching with a finger by coupling embedded speech and KSSL recognizer, and suggests an improved synchronization method between multi-modalities for simultaneous multi-modality, for a pattern recognition-based neo multi-modal HCI. The MMCR fuses and recognizes 16 word-based command models that are represented by stylus, speech and KSSI (Korean Standard Sign language), and then translates the recognition result into synthetic speech and visual illustration, for multi-modal PDA handling and interaction.