Shadow detection and removal in apple image segmentation under natural light conditions using an ultrametric contour map

The image segmentation technique is a vital component of vision-based fruit harvesting robots because the accuracy of this technique greatly affects the recognition and identification by robots. However, the uncertainty and ambiguity of natural scenes in orchards make image segmentation a challenging task. Here, we developed a new algorithm to detect and remove the shadows generated under intense illumination and direct sunlight conditions. Group pixels and edge probability maps were fused in our algorithm to generate superpixel blocks with precise boundaries. We applied an affinity matrix to obtain an ultrametric contour map to detect shadows and then used a relighting method to remove the detected shadows. Additionally, the shadow detection and removal and image segmentation procedures were evaluated. Our shadow detection results showed that the root mean square error decreased from 7.9% to 6.4% when an edge probability map was applied. Using the new shadow removal algorithm, the precision, balanced accuracy, specificity, and modified segmentation accuracy were improved by 10, 11, 4.5, and 10.1%, respectively. The average segmentation processing time was 0.59 s, which meets the requirements of real-time applications (