CNN based Segmentation of Nuclei in PAP-Smear Images with Selective Pre-processing

Cervical cancer is the second most common cause of death among women worldwide, but it can be treated if detected early. However, due to inter and intra observer variability in manual screening, automating the process is need of the hour. For classifying the cervical cells as normal vs abnormal, segmentation of nuclei as well as cytoplasm is a prerequisite. But the segmentation of nuclei is relatively more reliable and equally efficient for classification to that of cytoplasm. Hence, this paper proposes a new approach for segmentation of nuclei based on selective pre-processing and then passing the image patches to respective deep CNN (trained with/without pre-processed images) for pixel wise 3 class labelling as nucleus, edge or background. We argue and demonstrate that a single pre-processing approach may not suit all images, as there are significant variations in nucleus sizes and chromatin patterns. The selective pre-processing is carried out to effectively address this issue. This also enables the deep CNNs to be better trained in spite of relatively less data, and thus better exploit the capability of CNN of good quality segmentation. The results show that the approach is effective for segmentation of nuclei in PAP-smears with an F-score of 0.90 on Herlev dataset as opposed to the without selective pre-processing F-scores of 0.78 (without pre-processing) and 0.82 (with pre-processing). The results also show the importance of considering 3 classes in CNN instead of 2 (nucleus and background) where the latter achieves an F-score as low as 0.63.