BERT-SMAP: Paying attention to Essential Terms in passage ranking beyond BERT

Passage ranking has attracted considerable attention due to its importance in information retrieval (IR) and question answering (QA). Prior works have shown that pre-trained language models (e.g. BERT) can improve ranking performance. However, these simple BERT-based methods tend to focus on passage terms that exactly match the question, which makes them easily fooled by the overlapping but irrelevant (distracting) passages. To solve this problem, we propose a self-matching attention-pooling mechanism (SMAP) to highlight the Essential Terms in the question-passage pairs. Further, we propose a hybrid passage ranking architecture, called BERT-SMAP, which combines SMAP with BERT to more effectively identify distracting passages and downplay their influence. BERT-SMAP uses the representations obtained through SMAP to enhance BERT's classification mechanism as an interaction-focused neural ranker, and as the inputs of a matching function. Experimental results on three evaluation datasets show that our model outperforms the previous best BERTbase-based approaches, and is comparable to the state-of-the-art method that utilizes a much stronger pre-trained language model.

Automated summary

Passage ranking is crucial in information retrieval and question answering, and pre-trained language models like BERT have been shown to improve performance. However, these models can be easily fooled by overlapping but irrelevant passages. To address this issue, a self-matching attention-pooling mechanism (SMAP) and a hybrid passage ranking architecture called BERT-SMAP have been proposed to better identify distracting passages.