HFR-Video-Based Fingertip Velocimeter for Multifinger Tapping Detection

In this study, we propose a novel concept of a software-based fingertip velocimeter using high-frame-rate (HFR) video processing that can simultaneously estimate when and where an operator taps with his/her finger by detecting the high-frequency component that develops when the fingertip actively contacts something. Our softwarebased fingertip velocimeter can precisely estimate the velocities of multiple fingers through HFR video processing in real time. Digital image correlation (DIC) operating at every frame for sub-pixel-precision velocity estimation is hybridized with convolution neural network (CNN)-based object detection operating at intervals of dozens of frames to robustly update the fingertip ROI regions during the frame-by-frame DIC operation. We developed a real-time multifinger tapping detection system that can execute DIC operation on 720x540 resolution images at 500 frames/s with CNN-based fingertip detection at 30 frames/s. By presenting several experimental results for finger tapping detection, including virtual keyboard interaction with a ten-finger keyboard input, the effectiveness of our fingertip velocimeter as a finger tapping interface was demonstrated, which can simultaneously estimate the tapping positions and moments of multiple fingers when finger tapping is performed ten times or more in a second.

Automated summary

In this study, a software-based fingertip velocimeter using high-frame-rate video processing is proposed to estimate when and where an operator taps with his/her finger. The system combines digital image correlation (DIC) with convolution neural network (CNN)-based object detection to estimate fingertip velocities in real time. Experimental results demonstrate the effectiveness of the fingertip velocimeter as a finger tapping interface for multiple fingers.