Classifying phonological categories in imagined and articulated speech

This paper presents a new dataset combining 3 modalities (EEG, facial, and audio) during imagined and vocalized phonemic and single-word prompts. We pre-process the EEG data, compute features for all 3 modalities, and perform binary classification of phonological categories using a combination of these modalities. For example, a deep-belief network obtains accuracies over 90% on identifying consonants, which is significantly more accurate than two baseline support vector machines. We also classify between the different states (resting, stimuli, active thinking) of the recording, achieving accuracies of 95%. These data may be used to learn multimodal relationships, and to develop silent-speech and brain-computer interfaces.