Transformer guided progressive fusion network for 3D pancreas and pancreatic mass segmentation

Pancreatic masses are diverse in type, often making their clinical management challenging. This study aims to address the task of various types of pancreatic mass segmentation and detection while accurately segmenting the pancreas. Although convolution operation performs well at extracting local details, it experiences difficulty capturing global representations. To alleviate this limitation, we propose a transformer guided progressive fusion network (TGPFN) that utilizes the global representation captured by the transformer to supplement long-range dependencies lost by convolution operations at different resolutions. TGPFN is built on a branch-integrated network structure, where the convolutional neural network and transformer branches first perform separate feature extraction in the encoder, and then the local and global features are progressively fused in the decoder. To effectively integrate the information of the two branches, we design a transformer guidance flow to ensure feature consistency, and present a cross-network attention module to capture the channel dependencies. Extensive experiments with nnUNet (3D) show that TGPFN improves the mass segmentation (Dice: 73.93% vs. 69.40%) and detection accuracy (detection rate: 91.71% vs. 84.97%) on 416 private CTs, and also obtains performance improvements of mass segmentation (Dice: 43.86% vs. 42.07%) and detection (detection rate: 83.33% vs. 71.74%) on 419 public CTs.

Automated summary

This study proposes a Transformer guided progressive fusion network (TGPFN) for segmenting and detecting various types of pancreatic masses, while accurately segmenting the pancreas. TGPFN alleviates the limitations of convolution operations in capturing global representations by utilizing global representations captured by the Transformer. Experimental results show that TGPFN improves mass segmentation and detection accuracy on both private and public CT scans.