A Hardware-Friendly Tiled Singular-Value Decomposition-Based Matrix Multiplication for Transformer-Based Models

Summary: The transformer model is highly accurate compared to conventional models in various domains, and the importance of the softmax layer has been increasing. However, accelerating the softmax layer has been challenging due to differences in data access patterns. This study addresses the challenge by decomposing the layer into multiple sub-layers and fusing them with adjacent operations, resulting in significant speed improvements.