UAV environmental perception and autonomous obstacle avoidance: A deep learning and depth camera combined solution

In agriculture, Unmanned Aerial Vehicles (UAVs) have shown great potential for plant protection. Uncertain obstacles randomly distributed in the unstructured farmland usually pose significant collision risks to flight safety. In order to improve the UAV's intelligence and minimize the obstacle's adverse impacts on operating safety and efficiency, we put forward a comprehensive solution which consists of deep-learning based object detection, image processing, RGB-D information fusion and Task Control System (TCS). Taking full advantages of both deep learning and depth camera, this solution allows the UAV to perceive not only the presence of obstacles, but also their attributes like category, profile and 3D spatial position. Based on the object detection results, collision avoidance strategy generation method and the corresponding calculation approach of optimal collision avoidance flight path are elaborated detailly. A series of experiments are conducted to verify the UAV's environmental perception ability and autonomous obstacle avoidance performance. Results show that the average detection accuracy of CNN model is 75.4% and the mean time cost for processing single image is 53.33 ms. Additionally, we find that the prediction accuracy of obstacle's profile and position depends heavily on the relative distance between the object and the depth camera. When the distance is between 4.5 m and 8.0 m, errors of object's depth data, width and height are -0.53 m, -0.26 m and -0.24 m respectively. Outcomes of simulation flight experiments indicated that the UAV can autonomously determine optimal obstacle avoidance strategy and generate distance-minimized flight path based on the results of RGB-D information fusion. The proposed solution has extensive potential to enhance the UAV's environmental perception and autonomous obstacle avoidance abilities.