Visual speech features representation for automatic lip-reading

A fundamental task in the pattern recognition field is to find a suitable representation for a feature. We present a new visual speech feature representation approach that combines hypercolumn model (HCM) with HMM to perform a complete lip-reading system. We use HCM to extract visual speech features from the input image. The extracted features are modeled by Gaussian distributions using HMM. The proposed lip-reading system can work under varying lip positions and sizes. All images were captured in a natural environment without using special lighting or lip markers. Experimental results are shown to compare favourably with the results of two reported systems, SOM and DCT based systems. HCM provides better performance than both of these systems.