Automatic Pathology Annotation on Medical Images: A Statistical Machine Translation Framework

Large number of medical images are produced daily in hospitals and medical institutions, the needs to efficiently process, index, search and retrieve these images are great. In this paper, we propose a pathology based medical image annotation framework using a statistical machine translation approach. After pathology terms and regions of interest (ROIs) are extracted from training text and images respectively, we use machine translation model IBM Model 1 to iteratively learn the alignment between the ROIs and the pathology terms and generate an ROI-to-pathology translation table. In testing phase, we annotate the ROI in the image with the pathology label of the highest probability in the translation table. The overall annotation results and the retrieval performance are promising to doctors and medical professionals.