Journal
2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)
Volume -, Issue -, Pages 12155-12164Publisher
IEEE COMPUTER SOC
DOI: 10.1109/CVPR52688.2022.01185
Keywords
-
Funding
- National Natural Science Foundation of China [61876007]
- Australian Research Council [DP210101859]
- University of Sydney SOAR Prize
Ask authors/readers for more resources
This paper studies the efficiency problem of visual transformers and proposes a patch slimming approach to reduce redundant calculations. Experimental results demonstrate that the proposed method can significantly reduce computational costs without sacrificing performance.
This paper studies the efficiency problem for visual transformers by excavating redundant calculation in given networks. The recent transformer architecture has demonstrated its effectiveness for achieving excellent performance on a series of computer vision tasks. However, similar to that of convolutional neural networks, the huge computational cost of vision transformers is still a severe issue. Considering that the attention mechanism aggregates different patches layer-by-layer, we present a novel patch slimming approach that discards useless patches in a topdown paradigm. We first identify the effective patches in the last layer and then use them to guide the patch selection process of previous layers. For each layer, the impact of a patch on the final output feature is approximated and patches with less impacts will be removed. Experimental results on benchmark datasets demonstrate that the proposed method can significantly reduce the computational costs of vision transformers without affecting their performances. For example, over 45% FLOPs of the ViT-Ti model can be reduced with only 0.2% top-1 accuracy drop on the ImageNet dataset.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available