OPEN-SET CLASSIFICATION IN REMOTE SENSING IMAGERY WITH ENERGY-BASED VISION TRANSFORMER

Most scene classification applications in remote sensing images are addressed from a closed set-setting perspective where both the training and testing sets have the same classes. In some applications, the testing set may encounter images belonging to classes not seen during training. In this case, the classifier will face the negative transfer problem, and assign these images to one of the known classes This raises the attention to develop specific open-set methods with unknown image rejection ability. In this paper, we propose an open-set classification method based on vision transformers. An energy-based model is used to learn the density of the training data by reinterpreting the logits of the token classification head of the transformer. At test time, we reject the images with low log-likelihood scores from classification and classify all other images to their labels. The method is evaluated on Optimal-31 a remote sensing dataset, showing comparable results to the state-of-art methods.

Automated summary

In this paper, an open-set classification method based on vision transformers is proposed. An energy-based model is used to learn the density of the training data, enabling rejection of unknown images and classification of other images. Evaluation on a remote sensing dataset shows comparable results to state-of-the-art methods.