Handshape Recognition in an Educational Game for Finger Alphabet Practicing

A vision-based method of handshape recognition was developed and used in a simple educational game designed to practice the finger alphabet. It uses a deep neural network to determine the two-dimensional skeleton of the hand and the genetic algorithm to recognize its shape. The classification was carried out by defining and solving an optimization problem, in which the skeleton corresponding to the unknown shape is subjected to an affine transformation and then adjusted to the previously prepared set of templates. The method was tested using the leave-one-subject-out validation protocol on the author's dataset of Polish Finger Alphabet letters and the publicly available Microsoft Kinect and Leap Motion Dataset. Based on the developed algorithm, a simple educational game was prepared, the purpose of which is to practice hand dexterity in showing complex shapes appearing in the finger alphabet.

Automated summary

This article introduces a vision-based method for handshape recognition, which uses a deep neural network and genetic algorithm to recognize the shape of the hand. A simple educational game is developed based on the algorithm to practice the use of finger alphabet.