Detecting Coarticulation in Sign Language using Conditional Random Fields

Coarticulation is one of the important factors that makes automatic sign language recognition a hard problem. Unlike in speech recognition, coarticulation effects in sign languages are over longer durations and simultaneously impact different aspects of the sign such as the hand shape, position, and movement. Due to this effect, the appearance of a sign, especially at the beginning and at the end, can be significantly different under different sentence contexts, which makes the recognition of signs in sentences hard. We advocate a two-step approach, where in the first step one segments the individual signs in a sentence and in the next step one recognizes the signs. In this work, we show how the first step, i.e. sign segmentation, can be performed effectively by using the conditional random fields (CRF) to directly detect the coarticulation points. The CRF approach does not make conditional independence assumptions about the observations and can be trained with fewer samples than hidden Markov models (HMMs). We validate our approach by demonstrating performance with American sign language (ASL) sentence level data and show that the CRF approach is 85% accurate in segmenting signs compared to 60% for the HMM approach at 0.1 false alarm rate