Learning scale-space representation of nucleus for accurate localization and segmentation of epithelial squamous nuclei in cervical smears

Computer vision systems are being introduced in pre-screening of cervical cytopathology slides to identify samples that require study by cytopathologists. These systems work on the principle of imaging and analysis of cytology features in general and nuclear features in particular. Thus accurate localization and segmentation of the nuclei is crucial for the systems. Though several methods have been conceptualized, developed and employed to achieve the tasks of localization and segmentation of nuclei in cytology images, most fail to localize nuclei with opened up chromatin. This paper presents a machine learning approach based framework for accurate localization and segmentation of nuclei. The approach uses the random forest model to learn complete scale-space representation of the nuclear chromatin distribution in green and color saturation channels. Based on the multi scale features of an unknown image this model can predict an image such that gray level value of a pixel is proportionate to the probability that the pixel belongs to nuclear region. This predicted image then can be used for accurate localization and segmentation of the nuclei by random walks approach. Accuracy of the system has been tested on a publicly available dataset images and was found to be approximately 97%.