Towards audio-video based handwritten mathematical content recognition in classroom videos

Recognizing handwritten mathematical content in classroom videos poses a range of interesting challenges. In this paper, we focus on improving the character recognition accuracy in such videos using a combination of video and audio based text recognizers. We propose a two step assembly consisting of a video text recognizer (VTR) as the primary character recognizer and an audio text recognizer (ATR) for disambiguating, if needed, the output of the VTR. We propose techniques for (1) detecting ambiguity in the output of the VTR so that a combination with the ATR may be triggered only for ambiguous characters, (2) synchronizing the output of the two recognizers for enabling combination, and (3) combining the options generated by the two recognizers using measurement and rank based methods. We have implemented the system using an open source implementation of a character recognizer and a commercially available phonetic word-spotter. Through experiments conducted using video recorded in a classroom-like environment, we demonstrate the improvement in the character recognition accuracy that can be achieved using our approach.