Journal
MACHINE LEARNING
Volume 111, Issue 3, Pages 917-935Publisher
SPRINGER
DOI: 10.1007/s10994-021-06070-y
Keywords
Neural machine translation; BERT; Context-aware translation
Categories
Ask authors/readers for more resources
The study investigates utilizing BERT to encode contextual information and finds that concatenating all contextual sequences into a longer one and then encoding it with BERT achieves the best translation results. The approach led to state-of-the-art BLEU scores on various translation tasks.
Context-aware neural machine translation (NMT), which targets at translating sentences with contextual information, has attracted much attention recently. A key problem for context-aware NMT is to effectively encode and aggregate the contextual information. BERT (Devlin et al., in: NAACL, 2019) has been proven to be an effective feature extractor in natural language understanding tasks, but it has not been well studied in context-aware NMT. In this work, we conduct a study about leveraging BERT to encode the contextual information for NMT, and explore three commonly used methods to aggregate the contextual features. We conduct experiments on five translation tasks and find that concatenating all contextual sequences as a longer one and then encoding it by BERT obtains the best translation results. Specifically, we achieved state-of-the-art BLEU scores on several widely investigated tasks, including IWSLT' 14 German -> English, News Commentary v11 English -> German translation and OpenSubtitle English -> Russian translation.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available