期刊
PROCEEDINGS OF THE WORLD WIDE WEB CONFERENCE 2021 (WWW 2021)
卷 -, 期 -, 页码 2466-2475出版社
ASSOC COMPUTING MACHINERY
DOI: 10.1145/3442381.3449988
关键词
Prior Knowledge; Semantic Textual Similarity; Deep Neural Networks; BERT
类别
资金
- National Natural Science Foundation of China [61976102, U19A2065]
- Fundamental Research Funds for the Central Universities
This study proposes a novel approach to enhance BERT's performance on semantic textual matching tasks by injecting prior knowledge directly into BERT's multi-head attention mechanism. Experimental results show that the knowledge-enhanced BERT consistently improves performance, especially when training data is scarce.
We study the problem of incorporating prior knowledge into a deep Transformer-based model, i.e., Bidirectional Encoder Representations from Transformers (BERT), to enhance its performance on semantic textual matching tasks. By probing and analyzing what BERT has already known when solving this task, we obtain better understanding of what task-specific knowledge BERT needs the most and where it is most needed. The analysis further motivates us to take a different approach than most existing works. Instead of using prior knowledge to create a new training task for fine-tuning BERT, we directly inject knowledge into BERT's multi-head attention mechanism. This leads us to a simple yet effective approach that enjoys fast training stage as it saves the model from training on additional data or tasks other than the main task. Extensive experiments demonstrate that the proposed knowledge-enhanced BERT is able to consistently improve semantic textual matching performance over the original BERT model, and the performance benefit is most salient when training data is scarce.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据