MOSAICOS: A Simple and Effective Use of Object-Centric Images for Long-Tailed Object Detection

Many objects do not appear frequently enough in complex scenes (e.g., certain handbags in living rooms) for training an accurate object detector, but are often found frequently by themselves (e.g., in product images). Yet, these object-centric images are not effectively leveraged for improving object detection in scene-centric images. In this paper, we propose Mosaic of Object-centric images as Scene-centric images (MOSAICOS), a simple and novel framework that is surprisingly effective at tackling the challenges of long-tailed object detection. Keys to our approach are three-fold: (i) pseudo scene-centric image construction from object-centric images for mitigating domain differences, (ii) high-quality bounding box imputation using the object-centric images' class labels, and (iii) a multi-stage training procedure. On LVIS object detection (and instance segmentation), MOSAICOS leads to a massive 60% (and 23%) relative improvement in average precision for rare object categories. We also show that our framework can be compatibly used with other existing approaches to achieve even further gains. Our pre-trained models are publicly available at https://github.com/czhang0528/MosaicOS/.

Automated summary

This paper introduces a simple and novel framework called "MOSAICOS", which effectively addresses the challenges of long-tailed object detection. The key to this framework lies in pseudo scene-centric image construction, high-quality bounding box imputation, and multi-stage training. Experimental results show a significant relative improvement in average precision for rare object categories with MOSAICOS.