Building a Biomedical Tokenizer Using the Token Lattice Design Pattern and the Adapted Viterbi Algorithm

Proper tokenization of biomedical text is a non-trivial problem. Problematic characteristics of current biomedical tokenizers include idiosyncratic tokenizer output and poor tokenizer extensibility and reuse. To address these problematic characteristics, we identified and completed a novel tokenizer design pattern for biomedical tokenizers. We separated a tokenizer into three components: a token lattice and lattice constructor, a best lattice-path chooser and token transducers. Token transducers create tokens from text. These tokens are assembled into a token lattice by the lattice constructor. The best path (tokenization) is selected from the token lattice, tokenizing the text. We applied our design pattern and our token transducer identification guidelines in the creation of a tokenizer for SNOMED CT concept descriptions and compared our tokenizer to three other tokenizer methods. Medpost and our adapted Viterbi tokenizer perform best with a 90.1% and 93.7% accuracy respectively.