Semantic segmentation based stereo visual servoing of nonholonomic mobile robot in intelligent manufacturing environment

In the interest of developing an intelligent manufacturing environment with an agile, efficient, and optimally utilized transportation system, mobile robots need to achieve a certain level of autonomy as they play an important role in carrying out transportation tasks. Bearing this in mind, in the paper we propose a novel stereo visual servoing method for nonholonomic mobile robot control based on semantic segmentation. Semantic segmentation provides a rich body of information required for an adequate decision-making process in a clustered, dynamic, and ever-changing manufacturing environment. The innovative idea behind the new visual servoing system is to utilize semantic information of the scene for visual servoing, as well as for other mobile robot tasks, such as obstacle avoidance, scene understanding, and simultaneous localization and mapping. Semantic segmentation is carried out by exploiting fully convolutional neural networks. The new visual servoing algorithm utilizes an intensity-based image registration procedure, which results in the image transformation matrix. The transformation matrix encompasses the relations of images taken at the current and desired pose, and that information is directly used for visual servoing. The developed algorithm is deployed on our own developed wheeled differential drive mobile robot RAICO (Robot with Artificial Intelligence based COgnition). The experimental evaluation is carried out in the 3D simulation environment and in the laboratory model of the real manufacturing environment. The experimental results show that the accuracy of the proposed approach is improved when compared to the state-of-the-art approaches while being robust to the partial occlusions of the scene and illumination changes.

Automated summary

This paper proposes a novel stereo visual servoing method for nonholonomic mobile robot control based on semantic segmentation to provide rich information for decision-making process in a manufacturing environment. The method utilizes fully convolutional neural networks for semantic segmentation and improves the accuracy and robustness of the approach compared to existing techniques through experimental evaluation.