Mouth motion analysis with space-time interest points

Speech recognition and speaker detection technique from audio visual fusion information attract much attention. In the visual side information, namely lip reading area, most of recent studies are based on analyzing shape of mouth, whereas few studies are based on analyzing lip motion. However, analysis associated with mouth motion gives essential cues for obtaining utterance mechanics. Thus, as a tool to analyze mouth motion, we focus attention on space-time interest points (STIP) that have been effectively applied for analyzing gait and for recognizing human action. This study stems from the idea that the STIP must be also useful for mouth motion analysis. The proposed mouth motion analysis system using STIP needs neither contour estimation nor feature tracking. Then, several image processings are proposed in order to appropriately apply STIP to mouth motion. Additionally, to evaluate the detected STIP as a tool for mouth motion analysis, we classified Japanese vowels utterances into three motion types by using detected STIPs.