2pClPr: A Two-Phase Clump Profiler for Segmentation of Cancer Cells in Fluorescence Microscopic Images

Cancer cell segmentation is challenging since they grow in tightly packed colonies (clumps), causing adjacent cells to overlap. In this work, we proposed an automated vision-based analysis framework: a two-phase clump profiler (2pClPr) for the segmentation of cancer cells in fluorescence microscopy images. In the first phase, we proposed a deep learning (DL) network, Multiscale Cell-Net, for coarse segmentation. Another framework, multiscale region proposal network (MS-RPN), was simultaneously trained in parallel to Multiscale Cell-Net to generate seeds for each cell. The coarse segmentation map was unable to segment the complex clumps. We proposed a novel metric, the Irregularity factor (I-ftr), to identify those complex clumps. Once identified, we mapped them with the seed points generated by MS-RPN. These seeds served as the initialization points for our proposed level-set framework: weighing repelling force embedded-level-set method (WRFe-LSM) which segments the identified complex clumps in the second phase of segmentation. The final segmentation map was generated with the segmented cells from the two phases. We conducted extensive experiments on our private dataset comprising images from four complex cancer cell lines and obtained an aggregated Jaccard index (AJI) of 76.6%, 72.9%, 75.5%, and 69.7% on HeLa, MDA-MB-468, MDA-MB-231, and T-47D, respectively. Furthermore, to show the generalization ability of 2pClPr, we conducted comparative experiments on a publicly available hematoxylin-eosin (H&E)stained histopathological images dataset (MoNuSeg) and obtained an AJI of 66.2%. A detailed evaluation of segmentation performance on both the datasets shows that 2pClPr is robust and effective.

Automated summary

Cancer cell segmentation is challenging due to the overlapping of tightly packed colonies. This study proposes an automated framework, 2pClPr, for the segmentation of cancer cells in fluorescence microscopy images.