Skeleton-RGB integrated highly similar human action prediction in human-robot collaborative assembly

Human-robot collaborative assembly (HRCA) combines the flexibility and adaptability of humans with the efficiency and reliability of robots during collaborative assembly operations, which facilitates complex product assembly in the mass personalisation paradigm. The cognitive ability of robots to recognise and predict human actions and make responses accordingly is essential but currently still limited, especially when facing highly similar human actions. To improve the cognitive ability of robots in HRCA, firstly, a two-stage skeleton-RGB integrated model focusing on human-parts interaction is proposed to recognise highly similar human actions. Specifically, it consists of a feature guidance module and a feature fusion module, which can balance the accuracy and efficiency of human action recognition. Secondly, an online prediction approach is developed to predict human actions ahead of schedule, which includes a pre-trained skeleton-RGB integrated model and a preprocessing module. Thirdly, considering the positioning accuracy of the parts to be assembled and the continuous update of human actions, a dynamic response scheme of the robot is designed. Finally, the feasibility and effectiveness of the proposed model and approach are verified by a case study of a worm-gear decelerator assembly. The experimental results demonstrate that the proposed model achieves precise human action recognition with a high accuracy of 93.75% and a lower computational cost. Specifically, only 15 frames from a skeleton stream and 5 frames (less than 16 frames in general) from an RGB video stream are adopted. Moreover, it only takes 1.026 s to achieve online human action prediction based on the proposed prediction method. The dynamic response scheme of the robot is also proven to be feasible. It is expected that the efficiency of human-robot interaction in HRCA can be improved from a closed-loop view of perception, prediction, and response.

Automated summary

Human-robot collaborative assembly combines human flexibility and robot efficiency in mass personalization production. To improve robot's cognitive ability, this research proposes a two-stage skeleton-RGB integrated model for recognizing highly similar human actions, an online prediction approach for predicting human actions ahead of schedule, and a dynamic response scheme for accurate part positioning and continuous update of human actions. Experimental results demonstrate the effectiveness of the proposed model and approach in achieving precise human action recognition and online prediction with lower computational cost.