Text information aggregation with centrality attention

A lot of natural language processing problems need to encode the text sequence as a fix-length vector, which usually involves an aggregation process of combining the representations of all the words, such as pooling or self-attention. However, these widely used aggregation approaches do not take higher-order relationships among the words into consideration. Hence we propose a new way of obtaining aggregation weights, called eigen-centrality self-attention. More specifically, we build a fully-connected graph for all the words in a sentence, then compute the eigen-centrality as the attention score of each word. The explicit modeling of relationships as a graph is able to capture some higher-order dependency among words, which helps us achieve better results in 5 text classification tasks and one SNLI task than baseline models such as pooling, self-attention, and dynamic routing. Besides, in order to compute the dominant eigenvector of the graph, we adopt a power method algorithm to get the eigen-centrality measure. Moreover, we also derive an iterative approach to get the gradient for the power method process to reduce both memory consumption and computation requirement.

Automated summary

In this study, a new self-attention mechanism called eigen-centrality self-attention is proposed to incorporate higher-order relationships among words in text sequence encoding, leading to better results in multiple tasks compared to baseline models. The power method algorithm is adopted to compute the dominant eigenvector of the graph, and an iterative approach is derived to reduce memory consumption and computation requirement during the process.