Automated joint 3D reconstruction and visual inspection for buildings using computer vision and transfer learning

Regular building inspection and maintenance has been deemed as a necessary preventive measure to maintain the functionality, integrity and aesthetics of buildings. 2D computer vision has demonstrated a good capability in detecting and classifying surface defects, but such 2D image-based methods provided limited support to represent the global information of defects concerning their positions and affecting areas for the entire building. Therefore, this paper presents an automated 2D-3D computer vision approach, with the aid of transfer learning, for joint 3D reconstruction and visual inspection of buildings. A new methodology framework is established to integrate the mechanism of 3D scene reconstruction with the transfer learning model towards automated identification of building surface defects on reconstructed 3D scenes. Defect identification and visualisation are accomplished with a revamped ResNet-50 model and Grad-CAM technique. The overlayed images with defects information are prepared as inputs for 3D reconstruction along with depth maps, which are realised by AD-census cost computation and semi-global matching optimisation algorithms. The proposed new methods are demonstrated via an illustrative example which promptly detects surface defects on the 3D scene and underpins smart facility maintenance management.

Automated summary

Regular building inspection and maintenance are necessary preventive measures to maintain building functionality, integrity, and aesthetics. This paper presents an automated 2D-3D computer vision approach, using transfer learning, for joint 3D reconstruction and visual inspection of buildings. The proposed methods integrate 3D scene reconstruction with transfer learning for automated identification of building surface defects, demonstrated through an illustrative example.