Detection of Fruit-Bearing Branches and Localization of Litchi Clusters for Vision-Based Harvesting Robots

Litchi clusters in fruit groves are randomly scattered and occur irregularly, so it is difficult to detect and locate the fruit-bearing branches of multiple litchi clusters at one time. This is a highly challenging task related to continuous operation in the natural environment for visual-based harvesting robots to carry out. In this study, a reliable algorithm based on RGB-depth (RGB-D) cameras in the fields was developed to accurately and automatically detect and locate the fruit-bearing branches of multiple litchi clusters simultaneously in large environments. A semantics segmentation method, Deeplabv3, was employed to segment the RGB images into three categories: background, fruit and twig. A pre-processing step is proposed to align the segmented RGB images and remove the twigs that did not bear fruits. Subsequently, the twig binary map image was processed via skeleton extraction and pruning operations, which left behind only the main branches of twigs. A method for non-parametric density-based spatial clustering of application with noise was used to cluster the pixels in the three-dimensional space of the skeleton map of the branches; thus, the fruit-bearing branches belonging to the same litchi clusters were determined. Finally, a three-dimensional straight line was fitted to each cluster via principal component analysis, and the linear information corresponded to the location of the fruit-bearing branches. In the experiments, 452 pairs of RGB-D images under different illumination were collected to test the proposed algorithm. The results show that the detection accuracy of a litchi fruit-bearing branch is 83.33%, positioning accuracy is 17.29 degrees +/- 24.57 degrees, and execution time for the determination of a single litchi fruit-bearing branch is 0.464s. Field experiments show that this method can effectively guide the robot to complete continuous picking tasks.