4.6 Article

Sentence part-enhanced BERT with respect to downstream tasks

期刊

COMPLEX & INTELLIGENT SYSTEMS
卷 9, 期 1, 页码 463-474

出版社

SPRINGER HEIDELBERG
DOI: 10.1007/s40747-022-00819-1

关键词

Natural language processing; BERT; Sentence representation; Sentence part; Fusion strategy

向作者/读者索取更多资源

In this study, a sentence part-enhanced BERT (SpeBERT) model is proposed, which enhances sentence representations by considering sentence parts with respect to downstream tasks. The sentence parts are encoded based on dependency parsing and downstream tasks, and embeddings are extracted through a pooling operation. Experimental results show that the proposed SpeBERT model outperforms competitor models in sentiment classification and semantic textual similarity tasks.
Bidirectional encoder representations from transformers (BERT) have achieved great success in many natural language processing tasks. However, BERT generally takes the embedding of the first token to represent sentence meaning in the tasks such as sentiment analysis and textual similarity, which does not properly treat different sentence parts. Different sentence parts have different levels of importance for different downstream tasks. For example, main parts (subject, predicate, and object) play crucial roles in textual similarity calculation, while secondary parts (adverbial and complement) are more important than the main parts in sentiment analysis. To this end, we propose a sentence part-enhanced BERT (SpeBERT) model that uses sentence parts with respect to downstream tasks to enhance sentence representations. Specifically, we encode sentence parts based on dependency parsing and downstream tasks, and extract embeddings through a pooling operation. Furthermore, we design several fusion strategies to incorporate different embeddings. We evaluate the proposed SpeBERT model on two downstream tasks, sentiment classification, and semantic textual similarity, with six benchmark datasets. The experimental results show that our model achieves better performance than competitor models.

作者

我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。

评论

主要评分

4.6
评分不足

次要评分

新颖性
-
重要性
-
科学严谨性
-
评价这篇论文

推荐

暂无数据
暂无数据