期刊
INFORMATION SCIENCES
卷 649, 期 -, 页码 -出版社
ELSEVIER SCIENCE INC
DOI: 10.1016/j.ins.2023.119655
关键词
Heterogeneous remote sensing imagery; Multimodal deep clustering; Transformer; Contrastive learning; Prototypical learning
The article introduces a new method for clustering multimodal remote sensing data, which achieves state-of-the-art performance on large-scale multimodal datasets by employing Transformer, online clustering mechanism, and self-supervised training strategy, and has good scalability.
Given the increasing diversity of available remote sensing data sources, multimodal fusion land cover classification has emerged as a promising direction in the community of Earth observation. However, modern supervised multimodal deep learning methods heavily rely on extensive amounts of human-annotated training data. To address this issue, we propose a novel unsupervised method for multimodal remote sensing land-cover type clustering: Transformer-based Multimodal Prototypical Contrastive Clustering (TMPCC). It is based on three core designs. First, we design a multimodal Transformer that learns a shared space through adaptive interactions between and within modalities. Second, we introduce an online clustering mechanism based on unified prototype learning that is scalable to large-scale multimodal datasets. Third, we employ a self-supervised training strategy that combines instance contrastive loss and clustering loss to enable efficient and effective model training. Coupling these three designs allows for training an end-to-end online clustering network that achieves state-of-the-art performance in multimodal RS data clustering, e.g., the highest clustering accuracy (92.28%) among existing methods on the hyperspectral-LiDAR Trento dataset. Our method demonstrates promising scalability in terms of modality and sample size, and it can also generalize to out-of-sample data. The code for this method is openly available at https://github .com /AngryCai /TMPCC.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据