A shape context fully convolutional neural network for segmentation and classification of cervical nuclei in Pap smear images

Pap smear is often employed as a screening test for diagnosing cervical pre-cancerous and cancerous lesions. Accurate identification of dysplastic changes amongst the cervical cells in a Pap smear image is thus essential for rapid diagnosis and prognosis. Manual pathological observations used in clinical practice require exhaustive analysis of thousands of cell nuclei in a whole slide image to visualize the dysplastic nuclear changes which make the process tedious and time-consuming. Automated nuclei segmentation and classification exist but are chal-lenging to overcome issues like nuclear intra-class variability and clustered nuclei separation. To address such challenges, we put forward an application of instance segmentation and classification framework built on an Unet architecture by adding residual blocks, densely connected blocks and a fully convolutional layer as a bottleneck between encoder-decoder blocks for Pap smear images. The number of convolutional layers in the standard Unet has been replaced by densely connected blocks to ensure feature reuse-ability property while the introduction of residual blocks in the same attempts to converge the network more rapidly. The framework provides simultaneous nuclei instance segmentation and also predicts the type of nucleus class as belonging to normal and abnormal classes from the smear images. It works by assigning pixel-wise labels to individual nuclei in a whole slide image which enables identifying multiple nuclei belonging to the same or different class as individual distinct instances. Introduction of a joint loss function in the framework overcomes some trivial cell level issues on clustered nuclei separation. To increase the robustness of the overall framework, the proposed model is preceded with a stacked auto-encoder based shape representation learning model. The proposed model outperforms two state-of-the-art deep learning models Unet and Mask_RCNN with an average Zijdenbos simi-larity index of 97 % related to segmentation along with binary classification accuracy of 98.8 %. Experiments on hospital-based datasets using liquid-based cytology and conventional pap smear methods along with benchmark Herlev datasets proved the superiority of the proposed method than Unet and Mask_RCNN models in terms of the evaluation metrics under consideration.