An image/speech relational database and its application

There are many reasons to investigate the simultaneous analysis of corresponding speech and image information. For example, in the case of video telephone/conferencing there is clearly a strong connection between the sound phonemes of voiced speech data and the mouth shape of the speaker. Also it is known that most verbal communications use cues from both the visual and acoustic modalities to convey messages. During the production of speech, the visible information provided by the external articulatory organs can influence the understanding of the language by interpreting the combined information into meaningful linguistic expressions. Although the belief is that only the hearing impaired make use of the visual stimuli in percepting the speech, reports have shown that normal hearing people use all the available visual information that accompany speech, especially when there is degradation of speech. Therefore the objective of this project is to investigate and quantify the relationship between speech and image data such that the knowledge gained will assist in longer term multimedia and videophone research. To achieve the above, a statistical database of key parameters derived from corresponding speech and image information channels is discussed