Wearable Sensors-Based Hand Gesture Recognition for Human-Robot Collaboration in Construction

The development of robotic machines has shown the potential to promote automation in construction. One of the critical enablers of human-robot collaboration is a user-friendly interface to support their interactions. Compared with other interfaces, hand gesture is an effective communication channel on construction sites. This article proposes a system for recognition of construction workers' hand gestures using wearable sensors on fingers. The system starts with synchronizing, normalizing, and smoothing finger motions. Then, the motion data are extracted through a sliding window and fed into an enhanced fully convolutional neural network (FCN) for the hand gesture recognition. The system was tested through a system validation test and achieved the precision and recall of 85.7% and 93.8%, respectively. A pilot study demonstrated the use of the proposed system to interact with a robotic dump truck. The system was further compared with vision-based recognition methods to quantitatively and qualitatively assess their relative benefits and limitations.

Automated summary

This article proposes a system for recognizing construction workers' hand gestures using wearable sensors on fingers. The system extracts motion data by synchronizing, normalizing, and smoothing finger motions and uses an enhanced fully convolutional neural network (FCN) for gesture recognition. The system achieved a precision and recall of 85.7% and 93.8% respectively in the system validation test. A pilot study demonstrated the feasibility of using the proposed system to interact with a robotic dump truck. Furthermore, the system was compared with vision-based recognition methods to assess their relative advantages and limitations quantitatively and qualitatively.