Gaze-aware hand gesture recognition for intelligent construction

The advances in construction robotics in recent decades has been a powerful driver of construction automation. User-friendly interfaces to support human-robot work collaboration are critical for increasing adoption of construction robots. Among different interfaces, hand gesture is an effective and reliable interaction cue in the noisy construction environment. This paper proposes a novel human gaze-aware hand gesture recognition framework as a human-robot interface for intelligent construction. The proposed framework relies on an eye tracker to visually detect and track robotic machines in the first-person view. Then, the machine-of-interest is determined based on the bounding boxes of machines and gaze points. Finally, a hand gesture recognition architecture is incorporated with the machine information for conveying messages to the machine-of-interest. This approach was tested through a framework validation test and achieved precision and recall of 93.8% and 95.0%, respectively. A pilot study was conducted to demonstrate interaction with a robotic excavator and a dump truck, and the results illustrated that the proposed framework could serve as an effective interface for one-to-many human-robot collaborations in construction.

Automated summary

The advances in construction robotics have driven construction automation, and user-friendly interfaces, such as hand gestures, are crucial for increasing the adoption of construction robots. This paper proposes a novel framework that combines human gaze-aware hand gesture recognition with visual detection and tracking using an eye tracker for intelligent construction. The framework achieved high precision and recall rates in a validation test and demonstrated effective human-robot collaboration in construction through a pilot study with an excavator and a dump truck.