Automated segmentation of RGB-D images into a comprehensive set of building components using deep learning

Building information modeling (BIM) has a semantic scope that encompasses all building systems, e.g. architectural, structural, mechanical, electrical, and plumbing. Automated, comprehensive digital modeling of buildings will require methods for semantic segmentation of images and 3D reconstructions capable of recognizing all building component classes. However, prior building component recognition methods have had limited semantic coverage and are not easily combined or scaled. Here we show that a deep neural network can semantically segment RGB-D (i.e. color and depth) images into 13 building component classes simultaneously despite the use of a small training dataset with only 1490 object instances. For this task, the method achieves an average intersection over union (IoU) of 0.5. The dataset was designed using a common building taxonomy to ensure comprehensive semantic coverage and was collected from a diversity of buildings to ensure infra-class diversity. As a consequence of its semantic scope, it was necessary to perform pre-segmentation and 3D to 2D projection as leverage for dataset annotation. In creating our deep learning pipeline, we found that transfer learning, class balancing, and prevention of overfitting effectively overcame the dataset's borderline adequate class representation. Our results demonstrate how the semantic coverage of a building component recognition method can be scaled to include a larger diversity of building systems. We anticipate our method to be a starting point for broadening the scope of the semantic segmentation methods involved in digital modeling of buildings.