4.7 Article

Transformer-based contrastive prototypical clustering for multimodal remote sensing data

期刊

INFORMATION SCIENCES
卷 649, 期 -, 页码 -

出版社

ELSEVIER SCIENCE INC
DOI: 10.1016/j.ins.2023.119655

关键词

Heterogeneous remote sensing imagery; Multimodal deep clustering; Transformer; Contrastive learning; Prototypical learning

向作者/读者索取更多资源

The article introduces a new method for clustering multimodal remote sensing data, which achieves state-of-the-art performance on large-scale multimodal datasets by employing Transformer, online clustering mechanism, and self-supervised training strategy, and has good scalability.
Given the increasing diversity of available remote sensing data sources, multimodal fusion land cover classification has emerged as a promising direction in the community of Earth observation. However, modern supervised multimodal deep learning methods heavily rely on extensive amounts of human-annotated training data. To address this issue, we propose a novel unsupervised method for multimodal remote sensing land-cover type clustering: Transformer-based Multimodal Prototypical Contrastive Clustering (TMPCC). It is based on three core designs. First, we design a multimodal Transformer that learns a shared space through adaptive interactions between and within modalities. Second, we introduce an online clustering mechanism based on unified prototype learning that is scalable to large-scale multimodal datasets. Third, we employ a self-supervised training strategy that combines instance contrastive loss and clustering loss to enable efficient and effective model training. Coupling these three designs allows for training an end-to-end online clustering network that achieves state-of-the-art performance in multimodal RS data clustering, e.g., the highest clustering accuracy (92.28%) among existing methods on the hyperspectral-LiDAR Trento dataset. Our method demonstrates promising scalability in terms of modality and sample size, and it can also generalize to out-of-sample data. The code for this method is openly available at https://github .com /AngryCai /TMPCC.

作者

我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。

评论

主要评分

4.7
评分不足

次要评分

新颖性
-
重要性
-
科学严谨性
-
评价这篇论文

推荐

暂无数据
暂无数据