Deep Belief Networks for Real-Time Extraction of Tongue Contours from Ultrasound During Speech

Ultrasound has become a useful tool for speech scientists studying mechanisms of language sound production. State-of-the-art methods for extracting tongue contours from ultrasound images of the mouth, typically based on active contour snakes, require considerable manual interaction by an expert linguist. In this paper we describe a novel method for fully automatic extraction of tongue contours based on a hierarchy of restricted Boltzmann machines (RBMs), i.e. deep belief networks (DBNs). Usually, DBNs are first trained generatively on sensor data, then discriminatively to predict human-provided labels of the data. In this paper we introduce the translational RBM (tRBM), which allows the DBN to make use of both human labels and raw sensor data at all stages of learning. This method yields performance in contour extraction comparable to human labelers, without any temporal smoothing or human intervention, and runs in real-time.