Connecting Images through Sources: Exploring Low-Data, Heterogeneous Instance Retrieval

Along with a new volume of images containing valuable information about our past, the digitization of historical territorial imagery has brought the challenge of understanding and interconnecting collections with unique or rare representation characteristics, and sparse metadata. Content-based image retrieval offers a promising solution in this context, by building links in the data without relying on human supervision. However, while the latest propositions in deep learning have shown impressive results in applications linked to feature learning, they often rely on the hypothesis that there exists a training dataset matching the use case. Increasing generalization and robustness to variations remains an open challenge, poorly understood in the context of real-world applications. Introducing the alegoria benchmark, containing multi-date vertical and oblique aerial digitized photography mixed with more modern street-level pictures, we formulate the problem of low-data, heterogeneous image retrieval, and propose associated evaluation setups and measures. We propose a review of ideas and methods to tackle this problem, extensively compare state-of-the-art descriptors and propose a new multi-descriptor diffusion method to exploit their comparative strengths. Our experiments highlight the benefits of combining descriptors and the compromise between absolute and cross-domain performance.

Automated summary

This article discusses the challenge of understanding and connecting historical territorial imagery collections through content-based image retrieval, introducing the alegoria benchmark which contains mixed multi-date vertical and oblique aerial digitized photography with modern street-level pictures. The study proposes to address the low-data, heterogeneous image retrieval problem by reviewing ideas and methods, comparing state-of-the-art descriptors, and introducing a new multi-descriptor diffusion method to exploit their strengths. Results highlight the benefits of combining descriptors and the trade-off between absolute and cross-domain performance.