Classification of visemes using visual cues

Psycho-acoustic tests have indicated that human vision classifies the visemes (visual representation of phonemes) into different classes. This study shows that visual features extracted from 2-D images of lip motion can be used to design an automatic classifier for visemes. Audio-visual recordings from 18 native speakers of American English for 12 Vowel-Consonant-Vowel (VCV) sounds were obtained using the consonants /b,v,w,õ,d,z/ and the vowels /a,i/. The lip height, lip width, motion of the upper lip and the rate at which lips move while producing the VCV words were visual features used to represent each VCV sound. Features extracted from nine of the speakers were used to define Linear Discriminant Analysis functions to classify the visemes and features extracted from the remaining nine speakers were used in testing the classifiers. When the VCV sounds were divided into five classes consistent with those obtained from psycho-acoustic tests, the percentage of correct classification was 72.1% in training and 69.3% in testing. When each VCV sequence was treated as an independent class, resulting in 12 classes, the percentage of correct recognition was 55.3% in the training set and 43.1% in the testing set.