期刊
2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, CVPR 2021
卷 -, 期 -, 页码 4205-4214出版社
IEEE COMPUTER SOC
DOI: 10.1109/CVPR46437.2021.00419
关键词
-
资金
- ANR [ANR-20-CHIA-0018]
- Agence Nationale de la Recherche (ANR) [ANR-20-CHIA-0018] Funding Source: Agence Nationale de la Recherche (ANR)
In this paper, it is argued that uncertainty in vision is a significant factor preventing successful learning of reasoning in vision and language problems. The study introduces a visual oracle that is less prone to exploiting spurious dataset biases, and proposes to transfer reasoning patterns from the oracle to a state-of-the-art Transformer-based VQA model. Experimental results show higher overall accuracy and accuracy on infrequent answers, indicating improved generalization and reduced dependency on dataset biases.
Since its inception, Visual Question Answering (VQA) is notoriously known as a task, where models are prone to exploit biases in datasets to find shortcuts instead of performing high-level reasoning. Classical methods address this by removing biases from training data, or adding branches to models to detect and remove biases. In this paper, we argue that uncertainty in vision is a dominating factor preventing the successful learning of reasoning in vision and language problems. We train a visual oracle and in a large scale study provide experimental evidence that it is much less prone to exploiting spurious dataset biases compared to standard models. We propose to study the attention mechanisms at work in the visual oracle and compare them with a SOTA Transformer-based model. We provide an in-depth analysis and visualizations of reasoning patterns obtained with an online visualization tool which we make publicly available(1). We exploit these insights by transferring reasoning patterns from the oracle to a SOTA Transformer-based VQA model taking standard noisy visual inputs via fine-tuning. In experiments we report higher overall accuracy, as well as accuracy on infrequent answers for each question type, which provides evidence for improved generalization and a decrease of the dependency on dataset biases.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据