UltraGesture: Fine-Grained Gesture Sensing and Recognition

With the rising of AR/VR technology and miniaturization of mobile devices, gesture recognition is becoming increasingly popular in the research area of human-computer interaction. Some pioneer ultrasound-based gesture recognition systems have been proposed. However, they mostly rely on low-resolution Doppler Effect, with the focus on whole hand motion and fail to deal with minor finger motions. This paper is to present UltraGesture, an ultrasonic finger motion perception and recognition system based on Channel Impulse Response (CIR). CIR measurements can provide with 7 mm resolution, which is sufficient for minor finger motion recognition. UltraGesture encapsulates CIR measurements into image, and builds a Convolutional Neural Network model to classify these images into different categories corresponding to distinct gestures. Furthermore, we use a sliding-window based method to improve accuracy and reduce response latency. UltraGesture can run on the already existed commercial speakers and microphones on most mobile devices without any hardware modification. Our results demonstrate that UltraGesture can achieve an average accuracy ofgreater than 99 percent for 12 gestures including finger click and rotation.

Automated summary

With the rise of AR/VR technology and miniaturization of mobile devices, gesture recognition is becoming popular in the research area of human-computer interaction. This paper presents UltraGesture, an ultrasonic finger motion perception and recognition system that can run on mobile devices without any hardware modification. Experimental results demonstrate that UltraGesture achieves high accuracy in recognizing various gestures.