Dual U-Net for the Segmentation of Overlapping Glioma Nuclei

The morphology and surroundings of cells have been routinely used by pathologists to diagnose the pathological subtypes of gliomas and to assess the malignancy of tumors. Thanks to the advent and development of digital pathology technology, it is possible to automatically analyze whole slides of tissue and focus on the nucleus in order to derive a quantitative assessment that can be used for grading, classification, and diagnosis. During the process of computer-assisted diagnosis, the accurate location and segmentation of nuclei from hematoxylin and eosin (H&E)-stained histopathological images is an important step. In this paper, we proposed a U-Net-based multi-task learning network in which the boundary and region information is utilized to improve the segmentation accuracy of glioma nuclei, especially overlapping ones. To refine the segmentation, a classification model is used to predict the boundary, a regression model is used to predict the distance map, and the final segmentation is obtained by using the fusion layers. The proposed approach was compared with other specially designed boundary-aware methods by using a pathological section dataset that consists of 320 glioma cases from the Huashan Hospital at Fudan University. Both the pixel-level and object-level evaluations showed that the structural modification is effective in segmentation with an F1-score of 0.82, a Hausdorff distance (HD) of 3.95, and an aggregated Jaccard index (AJI) of 0.66 (+0.46%, -3.75%, and +4.09% compared with the unimproved methods, respectively). In addition, comparative experiments on multi-organ nuclei segmentation (MoNuSeg) open dataset proved the advanced nature of the proposed method in the field of nuclei segmentation, especially separating touching objects. The proposed method obtains an AJI of 0.59 and an F1-score of 0.79.