Panoptic Feature Fusion Net: A Novel Instance Segmentation Paradigm for Biomedical and Biological Images

Instance segmentation is an important task for biomedical and biological image analysis. Due to the complicated background components, the high variability of object appearances, numerous overlapping objects, and ambiguous object boundaries, this task still remains challenging. Recently, deep learning based methods have been widely employed to solve these problems and can be categorized into proposal-free and proposal-based methods. However, both proposal-free and proposal-based methods suffer from information loss, as they focus on either global-level semantic or local-level instance features. To tackle this issue, we present a Panoptic Feature Fusion Net (PFFNet) that unifies the semantic and instance features in this work. Specifically, our proposed PFFNet contains a residual attention feature fusion mechanism to incorporate the instance prediction with the semantic features, in order to facilitate the semantic contextual information learning in the instance branch. Then, a mask quality sub-branch is designed to align the confidence score of each object with the quality of the mask prediction. Furthermore, a consistency regularization mechanism is designed between the semantic segmentation tasks in the semantic and instance branches, for the robust learning of both tasks. Extensive experiments demonstrate the effectiveness of our proposed PFFNet, which outperforms several state-of-the-art methods on various biomedical and biological datasets.

Automated summary

Instance segmentation in biomedical and biological image analysis is challenging due to complex backgrounds, variable object appearances, overlapping objects, and ambiguous boundaries. Proposed Panoptic Feature Fusion Net (PFFNet) unifies semantic and instance features to address the issue of information loss in proposal-free and proposal-based methods. PFFNet incorporates a residual attention feature fusion mechanism and mask quality sub-branch to improve semantic contextual information learning and align object confidence scores with mask quality prediction, leading to robust learning in both semantic and instance branches. Extensive experiments show PFFNet outperforms state-of-the-art methods on biomedical and biological datasets.