Dynamic-Hierarchical Attention Distillation With Synergetic Instance Selection for Land Cover Classification Using Missing Heterogeneity Images

Optical and SAR modalities can provide the complementary information on the land properties, which usually lead to more robust and better classification performance. However, due to the restriction of imaging condition, not all modalities included into the training data sets could be available in real testing samples. Therefore, it is important to explore how to learn discriminative representations using multimodal data during the training stage, while achieving fine land cover classification using missing modalities at test time. In this article, we propose a novel dynamic-hierarchical attention distillation network (DH-ADNet) with multimodal synergetic instance selection (MSIS) for land cover classification using missing data modalities. First, the MSIS realizes the selection of the most representative multimodal instances to enhance the DH-ADNet's ability of discriminative feature extraction. Then, the DH-ADNet is training on the basis of the curriculum learning strategy and promotes the hallucination stream to learn the privileged information. In particular, a novel dynamic-hierarchical attention distillation module (DH-ADM) is introduced, which adaptively highlights different contributions of multilayer attention distillation by carefully exploring the classification losses of multilayer features over the training iterations. Comprehensive evaluations on two coregistered optical and SAR data sets and report state-of-the-art results in the privileged information scenario.

Automated summary

This article proposes a novel dynamic-hierarchical attention distillation network (DH-ADNet) with multimodal synergetic instance selection (MSIS) for land cover classification using missing data modalities. The method utilizes multimodal data and attention mechanism to achieve better classification performance.