Atomic force microscope tip localization and tracking through deep learning based vision inside an electron microscope

Scanning Electron Microscopy (SEM) is an ideal observation tool for small scales robotics. It has the potential to achieve automated nano-robotic tasks such as nano-handling and nano-assembly. Path following control of nano-robot end effectors using SEM vision feedback is a key for an intuitive programming of elementary robotic tasks sequences. It requires the ability to track end effectors under various SEM scan speeds. SEM suffers however from tricky issues that limits robotic tracking capabilities. This paper focuses on one specific issue related to the compromise between the scan speed and the image quality. This restriction seriously limits the performance of conventional vision tracking algorithms when used with electron images. At high scan speed, the image quality is very noisy making very difficult to differentiate the robot end effector from the background, hence limiting the tracking capabilities. The work related in this paper explores for the first time the potential value of Convolutional Neural Networks (ConvNet) in the context of nano-robotic vision tracking inside SEM. The aim is to localize an end-effector, AFM cantilever in the case of the study, from SEM images for any scan speed configuration and despite of low images quality. For that purpose, a data set of AFM tip images is build up from SEM images for the learning algorithm. Network performances are estimated under different SEM scan speeds. Thanks to the learning algorithm, experimental results show robust AFM tip tracking capabilities inside the SEM under various scan speed conditions.