Building Handwriting Recognizers by Leveraging Skeletons of Both Offline and Online Samples

We present an approach to leveraging both offline and online handwriting samples to build a single recognizer for recognizing both offline and online handwritings. Given a training set of offline handwriting samples and another set of online handwriting samples, a skeleton is derived first from each offline handwriting sample via vectorization. Then both the skeleton samples and online handwriting samples are normalized and rendered by using the same method to generate a combined training set of skeleton images. Finally a handwriting recognizer based on Deep Bidirectional Long Short-Term Memory (DBLSTM) and Hidden Markov Model (HMM) is built from the skeleton images. In recognition, a preprocessing step consistent with that in training is applied to an unknown offline or online handwriting sample to derive a skeleton image, which is recognized by the hybrid DBLSTM-HMM handwriting recognition system accordingly. We have built such a recognizer by using IAM benchmark databases of offline and online English handwritings plus an internal online handwriting corpus, which outperforms the recognizers built from either offline or online handwriting samples only.