On-tree apple fruit size estimation using stereo vision with deep learning-based occlusion handling

Apple (Malus domestica) fruit size plays an integral role in orchard management decision -making, particularly during chemical thinning, fruit quality assessment, and yield pre-diction. A machine vision system was developed using stereo cameras synchronised to a custom-built LED strobe to perform on-tree sizing of fruit in images with high measure-ment accuracy. Two deep neural network models (Faster R-CNN and Mask R-CNN) were trained to detect fruit candidates for sizing followed by extrapolation of occluded fruit regions to improve size estimation. The segmented fruit shapes were converted to metric surface areas and diameters using spatial resolutions and depth information from the stereo cameras. Monthly field trials from June to October using the camera system were conducted, measuring fruit diameters ranging from 22 to 82 mm, and compared against ground truth diameters. Diameter estimates had a mean absolute error ranging from 1.1 to 4.2 mm for the five-month trial period, an average error of 4.8% compared to ground truth diameter measurements. Standard deviation errors ranged from 0.7 to 1.9 mm. Using neural network models for intelligent sampling of fruit in images followed by extrapolation of missing regions can be an alternative method of handling fruit occlusion in agricultural imaging and improving sizing accuracy.(c) 2022 The Author(s). Published by Elsevier Ltd on behalf of IAgrE. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/ ).

Automated summary

A machine vision system combined with deep neural network models is developed to accurately measure the size of apple fruit, which can be used for orchard management decision-making, fruit quality assessment, and yield prediction. This technology can handle occluded fruit and improve the accuracy of fruit size measurement.