Coordinate CNNs and LSTMs to categorize scene images with multi-views and multi-levels of abstraction

Due to complexities of scene images, scene categorization is a challenging task in the computer vision community. To categorize scene images effectively, in this paper, we propose to coordinate Convolutional Neural Networks (CNNs) and Long Short-Term Memory networks (LSTMs) to perform scene categorization with multi-views and multi-levels of abstraction. Specifically, to utilize the complementary properties of features of different levels of abstraction, we employ CNNs to extract features of multi-levels of abstraction based on its hierarchical structure. Furthermore, in order to deal with variations in scene image contents, we represent each image with multiple views, and in order to take correlation between image views into consideration, we treat image view features from the same image as a sequence and employ Long Short-Term Memory networks (LSTMs) to perform classification. Based on the proposed method, information of multi-views and multi-levels of abstraction can be made full use of in a single framework. We evaluate the proposed method on two challenging scene datasets, MIT indoor scene 67 and SUN 397. Obtained results demonstrate the effectiveness of utilizing CNNs and LSTMs to categorize scene images with multi-views and multi-levels of abstraction. Experiments on comparison to state-of-the-art methods show that the proposed method outperforms all the other methods used for comparison. (C) 2018 Elsevier Ltd. All rights reserved.