Bilateral Cross-Modality Graph Matching Attention for Feature Fusion in Visual Question Answering

Answering semantically complicated questions according to an image is challenging in a visual question answering (VQA) task. Although the image can be well represented by deep learning, the question is always simply embedded and cannot well indicate its meaning. Besides, the visual and textual features have a gap for different modalities, it is difficult to align and utilize the cross-modality information. In this article, we focus on these two problems and propose a graph matching attention (GMA) network. First, it not only builds graph for the image but also constructs graph for the question in terms of both syntactic and embedding information. Next, we explore the intramodality relationships by a dual-stage graph encoder and then present a bilateral cross-modality GMA to infer the relationships between the image and the question. The updated cross-modality features are then sent into the answer prediction module for final answer prediction. Experiments demonstrate that our network achieves the state-of-the-art performance on the GQA dataset and the VQA 2.0 dataset. The ablation studies verify the effectiveness of each module in our GMA network.

Automated summary

This article proposes a graph matching attention (GMA) network to address the challenges of answering semantically complicated questions in visual question answering (VQA) tasks. The network builds graphs for both the image and the question, and utilizes a dual-stage graph encoder and bilateral cross-modality GMA to infer the relationships between them. The updated cross-modality features are then used for final answer prediction. Experimental results show that the network achieves state-of-the-art performance on GQA and VQA 2.0 datasets, and ablation studies verify the effectiveness of each module in the GMA network.