Multi-class fruit-on-plant detection for apple in SNAP system using Faster R-CNN

Deep learning achieved high success of fruit-on-plant detection such as on apple. Most of studies on apple detection identified all target fruits as one class regardless of fruit condition and other canopy objects. However, some detected fruits were physically occluded by branches or trellis wires that could diminish the effectiveness of fruit picking and even damage the end-effector, especially when high-vigor rootstock apple cultivar was used. A multi-class apple detection method in dense-foliage fruiting-wall trees was thus proposed based on Faster Region-Convolutional Neural Network. It detected apples in different conditions such as non-occluded, leaf-occluded, branch/wire-occluded, and fruit-occluded fruit. A total of 800 images were acquired and then augmented to 12,800 images. Average precision of non-occluded, leaf-occluded, branch/wire-occluded, and fruit-occluded fruit were 0.909, 0.899, 0.858, and 0.848, respectively. Overall, the mean average precision of the four classes was 0.879, and an average of 0.241 s was needed to process an image. The results indicated that all the apples in different classes could be effectively detected, which can help the robot to decide the picking strategy (e.g., picking order and path planning) as well as to avoid the potential damage by the branches and trellis wires.