Journal
NEUROCOMPUTING
Volume 454, Issue -, Pages 14-24Publisher
ELSEVIER
DOI: 10.1016/j.neucom.2021.04.038
Keywords
Natural language processing; Multi-head attention; Diversity; Routing-by-agreement; Neural machine translation; Sentence encoding
Categories
Ask authors/readers for more resources
This paper introduces two methods, disagreement regularization and routing-by-agreement algorithm, to better exploit the diversity of multi-head attention. Experimental results show that these methods can effectively improve model performance, and combining them can further enhance the model performance.
Multi-head attention is appealing for the ability to jointly attend to information from different represen-tation subspaces at different positions. In this work, we propose two approaches to better exploit such diversity for multi-head attention, which are complementary to each other. First, we introduce a dis-agreement regularization to explicitly encourage the diversity among multiple attention heads. Specifically, we propose three types of disagreement regularization, which respectively encourage the subspace, the attended positions, and the output representation associated with each attention head to be different from other heads. Second, we propose to better capture the diverse information distributed in the extracted partial-representations with the routing-by-agreement algorithm. The routing algorithm iteratively updates the proportion of how much a part (i.e. the distinct information learned from a speci-fic subspace) should be assigned to a whole (i.e. the final output representation), based on the agreement between parts and wholes. Experimental results on the machine translation, sentence encoding and log-ical inference tasks demonstrate the effectiveness and universality of the proposed approaches, which indicate the necessity of better exploiting the diversity for multi-head attention. While the two strategies individually boost performance, combining them together can further improve the model performance. (c) 2021 Elsevier B.V. All rights reserved.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available