期刊
IEEE TRANSACTIONS ON MULTIMEDIA
卷 23, 期 -, 页码 1383-1396出版社
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TMM.2020.2997127
关键词
Attention mechanism; generative adversarial networks; infrared and visible image fusion
资金
- Key Program of High-Resolution Earth Observation System [GFZX0404130307]
- NationalNatural Science Foundation ofChina [41901350]
- Fundamental Research Funds of People's Public Security University of China [2019JKF330]
- Fundamental Research Funds for the Central Universities [JZ2019HGBZ0151]
In this paper, a method named AttentionFGAN is proposed to fuse infrared and visible images by integrating multi-scale attention mechanism into Generative Adversarial Networks (GAN). The generator and discriminator both apply attention mechanism to emphasize the focus on typical regions of source images during fusion. Ablation experiments demonstrate the effectiveness of the method, and extensive qualitative and quantitative experiments on three public datasets show the advantages of AttentionFGAN compared to other state-of-the-art methods.
Infrared and visible image fusion aims to describe the same scene from different aspects by combining complementary information of multi-modality images. The existing Generative adversarial networks (GAN) based infrared and visible image fusion methods cannot perceive the most discriminative regions, and hence fail to highlight the typical parts existing in infrared and visible images. To this end, we integrate multi-scale attention mechanism into both generator and discriminator of GAN to fuse infrared and visible images (AttentionFGAN). The multi-scale attention mechanism aims to not only capture comprehensive spatial information to help generator focus on the foreground target information of infrared image and background detail information of visible image, but also constrain the discriminators focus more on the attention regions rather than the whole input image. The generator of AttentionFGAN consists of two multi-scale attention networks and an image fusion network. Two multi-scale attention networks capture the attention maps of infrared and visible images respectively, so that the fusion network can reconstruct the fused image by paying more attention to the typical regions of source images. Besides, two discriminators are adopted to force the fused result keep more intensity and texture information from infrared and visible image respectively. Moreover, to keep more information of attention region from source images, an attention loss function is designed. Finally, the ablation experiments illustrate the effectiveness of the key parts of our method, and extensive qualitative and quantitative experiments on three public datasets demonstrate the advantages and effectiveness of AttentionFGAN compared with the other state-of-the-art methods.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据