MASD: A Multimodal Assembly Skill Decoding System for Robot Programming by Demonstration

Programming by demonstration (PBD) transforms the robot programming from the code level to automated interface between robot and human, promoting the flexibility of robotized automation. In this paper, we focus on programming the industrial robot for assembly tasks by parsing the human demonstration into a series of assembly skills and compiling the skill to the robot executables. To achieve this goal, an identification system using multimodal information to recognize the assembly skill, called MASD, is proposed including: 1) an initial learning stage using a hierarchical model to recognize the action by considering the features from action-object effect, gesture, and trajectory and 2) a retrospective thinking stage using a segmentation method to cut the continuous demonstrations into multiple assembly skills optimally. Using MASD, the demonstration of assembly tasks can be explained with high accuracy in real time, driving a hypothesis that a PBD system on the top of MASD can be extended to more realistic assembly tasks beyond pure positional moving and picking. In experiments, the skill identification module is used to recognize the five kinds of assembly skills in demonstrations of both single and multiple assembly skills, and outperforms the comparative action identification methods. Besides integrated with the MASD, the PBD system can generate the program based on the demonstration and successfully enable an ABB industrial robotic arm simulator to assemble a flashlight and a switch, verifying the initial hypothesis.