An Underwater Human-Robot Interaction Using a Visual-Textual Model for Autonomous Underwater Vehicles

The marine environment presents a unique set of challenges for human-robot interaction. Communicating with gestures is a common way for interacting between the diver and autonomous underwater vehicles (AUVs). However, underwater gesture recognition is a challenging visual task for AUVs due to light refraction and wavelength color attenuation issues. Current gesture recognition methods classify the whole image directly or locate the hand position first and then classify the hand features. Among these purely visual approaches, textual information is largely ignored. This paper proposes a visual-textual model for underwater hand gesture recognition (VT-UHGR). The VT-UHGR model encodes the underwater diver's image as visual features, the category text as textual features, and generates visual-textual features through multimodal interactions. We guide AUVs to use image-text matching for learning and inference. The proposed method achieves better performance than most existing purely visual methods on the dataset CADDY, demonstrating the effectiveness of using textual patterns for underwater gesture recognition.

Automated summary

The marine environment poses unique challenges for human-robot interaction, especially in underwater gesture recognition. A visual-textual model (VT-UHGR) is proposed in this paper to overcome the difficulties caused by light refraction and color attenuation. By encoding the visual and textual features of underwater divers, the VT-UHGR model generates multimodal interactions to guide AUVs in learning and inference. The results show that incorporating textual patterns significantly improves the performance of underwater gesture recognition.