期刊
出版社
ASME
DOI: 10.1115/1.4054297
关键词
human-robot collaboration; gesture recognition; speech recognition; real-time; multi-modal; multiple threads
资金
- National Science Foundation [CMMI-1646162, CMMI-1954548]
- Intelligent Systems Center at Missouri University of Science and Technology
As artificial intelligence and industrial automation continue to advance, human-robot collaboration with advanced interaction capabilities is becoming increasingly important. This paper presents a real-time, multi-modal human-robot collaboration system using speech and gestures. Dynamic gestures are designed and recognized in real time using a convolutional neural network, enabling effective communication between humans and robots.
As artificial intelligence and industrial automation are developing, human-robot collaboration (HRC) with advanced interaction capabilities has become an increasingly significant area of research. In this paper, we design and develop a real-time, multi-model HRC system using speech and gestures. A set of 16 dynamic gestures is designed for communication from a human to an industrial robot. A data set of dynamic gestures is designed and constructed, and it will be shared with the community. A convolutional neural network is developed to recognize the dynamic gestures in real time using the motion history image and deep learning methods. An improved open-source speech recognizer is used for real-time speech recognition of the human worker. An integration strategy is proposed to integrate the gesture and speech recognition results, and a software interface is designed for system visualization. A multi-threading architecture is constructed for simultaneously operating multiple tasks, including gesture and speech data collection and recognition, data integration, robot control, and software interface operation. The various methods and algorithms are integrated to develop the HRC system, with a platform constructed to demonstrate the system performance. The experimental results validate the feasibility and effectiveness of the proposed algorithms and the HRC system.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据