A zero-shot deep metric learning approach to Brain-Computer Interfaces for image retrieval

In this paper we propose a deep learning based approach for image retrieval using EEG. Our approach makes use of a multi-modal deep neural network based on metric learning, where the EEG signal from a user observing an image is mapped together with visual information extracted from the image. The inspiration behind this work is the vision of a system which allows the user to navigate their image catalogue just by thinking about the image they want to see. Thanks to our metric learning approach, the system is scalable in that it can operate with new images that have never been used in training, resulting in a zero-shot image retrieval system. This framework is tested in two different standard EEG image-viewing datasets, where we demonstrate state-of-the-art results in this complex scenario.Crown Copyright (c) 2022 Published by Elsevier B.V. All rights reserved.

Automated summary

This paper proposes a deep learning based approach for image retrieval using EEG, which utilizes a multi-modal deep neural network and metric learning to map EEG signals and visual information. With the scalable metric learning approach, the system achieves zero-shot image retrieval with new images and demonstrates state-of-the-art results on standard EEG image-viewing datasets.