STDFusionNet: An Infrared and Visible Image Fusion Network Based on Salient Target Detection

In this article, we propose an infrared and visible image fusion network based on the salient target detection, termed STDFusionNet, which can preserve the thermal targets in infrared images and the texture structures in visible images. First, a salient target mask is dedicated to annotating regions of the infrared image that humans or machines pay more attention to, so as to provide spatial guidance for the integration of different information. Second, we combine this salient target mask to design a specific loss function to guide the extraction and reconstruction of features. Specifically, the feature extraction network can selectively extract salient target features from infrared images and background texture features from visible images, while the feature reconstruction network can effectively fuse these features and reconstruct the desired results. It is worth noting that the salient target mask is only required in the training phase, which enables the proposed STDFusionNet to be an end-to-end model. In other words, our STDFusionNet can fulfill salient target detection and key information fusion in an implicit manner. Extensive qualitative and quantitative experiments demonstrate the superiority of our fusion algorithm over the current state of the arts, where our algorithm is much faster and the fusion results look like high-quality visible images with clear highlighted infrared targets. Moreover, the experimental results on the public datasets reveal that our algorithm can improve the entropy (EN), mutual information (MI), visual information fidelity (VIF), and spatial frequency (SF) metrics with about 1.25%, 22.65%, 4.3%, and 0.89% gains, respectively. Our code is publicly available at https://github.com/jiayi-ma/STDFusionNet.

Automated summary

The proposed STDFusionNet is a fusion network for infrared and visible images that preserves thermal targets and texture structures. By utilizing salient target detection and a specific loss function, it successfully extracts and reconstructs features to achieve high-quality fusion results. Through qualitative and quantitative experiments, the algorithm has shown superiority in terms of speed and quality improvement over existing methods.