Hybrid segmentation of breast cancer cell images using a new fuzzy active contour model and an enhanced watershed method

Segmentation is the main sensitive problem in the automatic image analysis of histopathology specimens. In the stained breast image tissue, cancer cells present a large variety in their characteristics that bring various difficulties for traditional segmentation algorithms. In this paper, we propose an automatic segmentation method for breast cancer cell images combining a new fuzzy active contour model and an enhanced watershed method. Firstly, a color geometric active contour model incorporating spatial fuzzy clustering algorithm is proposed to detect the contours of all cell nuclei in the image. It combines the classical level set method, together with a Bayes error functional based on color region information. Moreover, the initial contour and the controlling parameters of the model are estimated from the fuzzy clustering results. Secondly, overlapping and touching cell nuclei are separated using an enhanced watershed algorithm based on concave vertex graph. Touching nuclei are located automatically using a robust high concavity point detector. Then, the watershed algorithm is applied on hybrid distance transform in order to get the most significant inner edges. A vertex graph is constructed from the concave points and the inner edges followed by an optimal path computed to select the separating curves of the touching nuclei. The proposed segmentation method is tested on a large dataset containing several breast cancer cell images with different levels of malignancy. The experimental results show the efficiency of the proposed algorithm when compared to other recent segmentation methods.