Self-Supervised Sound Promotion Method of Sound Localization from Video

Compared to traditional unimodal methods, multimodal audio-visual correspondence learning has many advantages in the field of video understanding, but it also faces significant challenges. In order to fully utilize the feature information from both modalities, we needs to ensure accurate alignment of the semantic information from each modality, rather than simply concatenating them together. This requires consideration of how to design fusion networks that can better perform this task. Current algorithms heavily rely on the network's output results for sound-object localization while neglecting the potential issue of suppressed feature information due to the internal structure of the network. Thus, we propose a sound promotion method (SPM), a self-supervised framework that aims to increase the contribution of voices to produce better performance of the audiovisual learning. We first cluster the audio separately to generate pseudo-labels and then use the clusters to train the backbone of audio. Finally, we explore the impact of our method to several existing approaches on MUSIC datasets and the results prove that our proposed method is able to produce better performance.

Automated summary

Compared to traditional unimodal methods, multimodal audio-visual correspondence learning has many advantages in video understanding, but it also faces significant challenges. In order to fully utilize the feature information from both modalities, accurate alignment of semantic information is crucial. Current algorithms heavily rely on sound-object localization but neglect the potential issue of suppressed feature information. Thus, we propose a sound promotion method (SPM) to increase the contribution of voices and improve audiovisual learning performance.