A review on the attention mechanism of deep learning

Attention has arguably become one of the most important concepts in the deep learning field. It is inspired by the biological systems of humans that tend to focus on the distinctive parts when processing large amounts of information. With the development of deep neural networks, attention mechanism has been widely used in diverse application domains. This paper aims to give an overview of the state-of-theart attention models proposed in recent years. Toward a better general understanding of attention mechanisms, we define a unified model that is suitable for most attention structures. Each step of the attention mechanism implemented in the model is described in detail. Furthermore, we classify existing attention models according to four criteria: the softness of attention, forms of input feature, input representation, and output representation. Besides, we summarize network architectures used in conjunction with the attention mechanism and describe some typical applications of attention mechanism. Finally, we discuss the interpretability that attention brings to deep learning and present its potential future trends. (c) 2021 Elsevier B.V. All rights reserved.

Automated summary

This paper provides an overview of state-of-the-art attention models and defines a unified model suitable for most attention structures. It describes in detail each step of the attention mechanism implemented in the model and classifies existing attention models based on four criteria. Additionally, it summarizes the use of attention mechanisms in network architectures and typical applications.