A ViT-Based Multiscale Feature Fusion Approach for Remote Sensing Image Segmentation

Semantic segmentation plays an indispensable role in automatic analysis of remote sensing image data. However, the abundant semantic information and irregular shape patterns in remote sensing images are difficult to utilize, making it hard to segment remote sensing images only using convolution and single-scale feature maps. To achieve better segmentation performance, a multiscale feature pyramid decoder (MFPD) is proposed to fuse image features extracted by vision transformer (ViT). The decoder employs a novel 2-D-to-3-D transform method to obtain multiscale feature maps that contain rich context information and fuses the multiscale feature maps by channel concatenation. Furthermore, a dimension attention module (DAM) is designed to further aggregate the context information of the extracted remote sensing image features. This approach yields superior mean intersection over union (mIoU) on the Gaofen2-CZ dataset (60.42%) and GID-5 dataset (68.21%). Experimental results indicate that the comprehensive performance of our approach exceeds the compared segmentation methods based on convolutional neural network (CNN) and ViT.

Automated summary

Semantic segmentation is essential for automatic analysis of remote sensing images. However, the utilization of abundant semantic information and irregular shape patterns in these images is challenging. To address this, a multiscale feature pyramid decoder (MFPD) is proposed to fuse image features extracted by vision transformer (ViT). The decoder employs a 2-D-to-3-D transform method to obtain multiscale feature maps and uses channel concatenation for fusion. A dimension attention module (DAM) is also designed to aggregate context information. Experimental results show that this approach outperforms other methods based on convolutional neural networks (CNN) and ViT in terms of mean intersection over union (mIoU).