4.8 Article

Attention Spiking Neural Networks

出版社

IEEE COMPUTER SOC
DOI: 10.1109/TPAMI.2023.3241201

关键词

Attention mechanism; efficient neuromorphic inference; neuromorphic computing; spiking neural network

向作者/读者索取更多资源

This paper studies the application of attention mechanisms in brain-inspired spiking neural networks (SNNs). By optimizing the membrane potentials using a multi-dimensional attention module, the performance and energy efficiency of SNNs are improved. Experimental results demonstrate that SNNs with attention achieve better performance and sparser spiking firing in event-based action recognition and image classification tasks. The effectiveness of attention SNNs is theoretically proven and further analyzed using a proposed spiking response visualization method. This work highlights the potential of SNNs as a general backbone for various applications in the field of SNN research.
Brain-inspired spiking neural networks (SNNs) are becoming a promising energy-efficient alternative to traditional artificial neural networks (ANNs). However, the performance gap between SNNs and ANNs has been a significant hindrance to deploying SNNs ubiquitously. To leverage the full potential of SNNs, in this paper we study the attention mechanisms, which can help human focus on important information. We present our idea of attention in SNNs with a multi-dimensional attention module, which infers attention weights along the temporal, channel, as well as spatial dimension separately or simultaneously. Based on the existing neuroscience theories, we exploit the attention weights to optimize membrane potentials, which in turn regulate the spiking response. Extensive experimental results on event-based action recognition and image classification datasets demonstrate that attention facilitates vanilla SNNs to achieve sparser spiking firing, better performance, and energy efficiency concurrently. In particular, we achieve top-1 accuracy of 75.92% and 77.08% on ImageNet-1 K with single/4-step Res-SNN-104, which are state-of-the-art results in SNNs. Comparedwith counterpart Res-ANN-104, the performance gap becomes -0.95/+0.21 percent and the energy efficiency is 31.8x/7.4x. To analyze the effectiveness of attention SNNs, we theoretically prove that the spiking degradation or the gradient vanishing, which usually holds in general SNNs, can be resolved by introducing the block dynamical isometry theory. We also analyze the efficiency of attention SNNs based on our proposed spiking response visualization method. Our work lights up SNN's potential as a general backbone to support various applications in the field of SNNresearch, with a great balance between effectiveness and energy efficiency.

作者

我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。

评论

主要评分

4.8
评分不足

次要评分

新颖性
-
重要性
-
科学严谨性
-
评价这篇论文

推荐

暂无数据
暂无数据