Change Captioning: A New Paradigm for Multitemporal Remote Sensing Image Analysis

Change detection (CD) is among the most important applications in remote sensing (RS) that allows identifying the changes that occurred in a given geographical area across different times. Even though CD systems have seen a lot of progress in RS, their output is either a binary map highlighting the changing area or a semantic change map that indicates the type of change for each pixel. The change maps are often difficult to interpret by end users, and they omit important information such as relationships and attributes of the changed areas. Motivated by the recent advancement of image captioning in the RS community, in this article, we propose to describe the changes over bitemporal images through change sentence descriptions. The aim of this article is to provide a user-friendly interpretation of the occurred changes. To this end, we propose two change captioning (CC) systems that take bitemporal images as input and generate coherent sentence descriptions of the occurred changes. Convolutional neural networks (CNNs) are used to extract discriminative features from the bitemporal images and recurrent neural networks (RNNs) or support vector machines (SVMs) are exploited to generate coherent change descriptions. Furthermore, in the absence of a CC dataset to test our systems, we propose two new datasets. One is based on very high-resolution RGB images, and the other one is based on multispectral RS images. The obtained experimental results show promising capabilities of the proposed systems to generate coherent change descriptions from the bitemporal images. The datasets are available at the following link: https://disi.unitn.it/similar to melgani/datasets.html.

Automated summary

This article introduces a system for describing changes in bitemporal images through change sentences to provide user-friendly interpretation. The system utilizes convolutional neural networks to extract features from bitemporal images and uses recurrent neural networks or support vector machines to generate coherent change descriptions, achieving promising experimental results.