Abstract
Reconstruction of 3D facades is an important problem in systems that reconstruct urban scenes. Facade reconstruction can be challenging due to the typically large featureless surfaces involved. In this work, we investigate the use of combining a commercially available LiDAR with a GoPro camera to serve as inputs for a system that generates accurate 3D facade reconstructions. A key challenge is that 3D point clouds from LiDARs tend to be sparse. We propose to overcome this by the use of semantic information extracted from RGB images, along with a state-of-the-art depth completion method. Our results demonstrate that the proposed approach is capable of producing highly accurate 3D reconstructions of building facades that rival the current state-of-the-art.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Becker, S., Haala, N.: Combined feature extraction for façade reconstruction. In: Proceedings of ISPRS Workshop Laser Scanning, pp. 241–247 (2007)
Becker, S., Haala, N.: Refinement of building facades by integrated processing of lidar and image data. Int. Arch. Photogram. Remote Sensing Spat. Inf. Sci. 36, 7–12 (2007)
Biljecki, F., Stoter, J., Ledoux, H., Zlatanova, S., Çöltekin, A.: Applications of 3D city models: state of the art review. ISPRS Int. J. Geo Inf. 4(4), 2842–2889 (2015). https://doi.org/10.3390/ijgi4042842
Chen, J., Chen, B.: Architectural modeling from sparsely scanned range data. Int. J. Comput. Vis. 78(2), 223–236 (2008)
Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: The European Conference on Computer Vision, September 2018
Fruh, C., Zakhor, A.: 3D model generation for cities using aerial photographs and ground level laser scans. In: The IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. II–II, December 2001. https://doi.org/10.1109/CVPR.2001.990921
Gee, T., James, J., Van Der Mark, W., Strozzi, A.G., Delmas, P., Gimel’farb, G.: Estimating extrinsic parameters between a stereo rig and a multi-layer lidar using plane matching and circle feature extraction. In: The Fifteenth IAPR International Conference on Machine Vision Applications, pp. 21–24, May 2017. https://doi.org/10.23919/MVA.2017.7986763
Hao, W., Wang, Y., Liang, W.: Slice-based building facade reconstruction from 3D point clouds. Int. J. Remote Sensing 39(20), 6587–6606 (2018)
He, K., Gkioxari, G., Dollar, P., Girshick, R.: Mask R-CNN. In: The IEEE International Conference on Computer Vision, October 2017. https://doi.org/10.1109/ICCV.2017.322
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition, June 2016. https://doi.org/10.1109/CVPR.2016.90
Hohmann, B., Krispel, U., Havemann, S., Fellner, D.: Cityfit-high-quality urban reconstructions by fitting shape grammars to images and derived textured point clouds. In: The 3rd ISPRS International Workshop 3D-ARCH, pp. 25–28 (2009)
Li, Y., Zheng, Q., Sharf, A., Cohen-Or, D., Chen, B., Mitra, N.J.: 2D–3D fusion for layer decomposition of urban facades. In: The IEEE International Conference on Computer Vision, pp. 882–889, November 2011. https://doi.org/10.1109/ICCV.2011.6126329
Lin, T., et al.: Microsoft COCO: common objects in context. CoRR (2014). http://arxiv.org/abs/1405.0312
Liu, L., Stamos, I.: Automatic 3D to 2D registration for the photorealistic rendering of urban scenes. In: The IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 137–143 (2005). https://doi.org/10.1109/CVPR.2005.80
Liu, L., Stamos, I.: A systematic approach for 2D-image to 3D-range registration in urban environments. In: The IEEE International Conference on Computer Vision, pp. 1–8, October 2007. https://doi.org/10.1109/ICCV.2007.4409215
Ma, F., Karaman, S.: Sparse-to-dense: depth prediction from sparse depth samples and a single image. In: IEEE International Conference on Robotics and Automation, May 2018
Martinović, A., Mathias, M., Weissenberg, J., Van Gool, L.: A three-layered approach to facade parsing. In: The European Conference on Computer Vision, pp. 416–429 (2012)
Marton, Z.C., Rusu, R.B., Beetz, M.: On fast surface reconstruction methods for large and noisy datasets. In: The IEEE International Conference on Robotics and Automation, May 2009. https://doi.org/10.1109/ROBOT.2009.5152628
Müller, P., Zeng, G., Wonka, P., Van Gool, L.: Image-based procedural modeling of facades. In: ACM SIGGRAPH 2007 Papers, SIGGRAPH 2007, ACM, New York (2007). https://doi.org/10.1145/1275808.1276484
Pu, S., Vosselman, G.: Building facade reconstruction by fusing terrestrial laser points and images. Sensors 9(6), 4525–4542 (2009)
Pu, S., Vosselman, G.: Refining building facade models with images. In: ISPRS Workshop, CMRT09-City Models, Roads and Traffic, vol. 38, pp. 3–4 (2009)
Riemenschneider, H., et al.: Irregular lattices for complex shape grammar facade parsing. In: The IEEE Conference on Computer Vision and Pattern Recognition, pp. 1640–1647, June 2012
Stamos, I., Allen, P.K.: 3-D model construction using range and image data. In: The IEEE Conference on Computer Vision and Pattern Recognition, p. 1531, June 2000. https://doi.org/10.1109/CVPR.2000.855865
Stamos, I., Allen, P.K.: Automatic registration of 2-D with 3-D imagery in urban environments. In: The IEEE International Conference on Computer Vision, vol. 2, pp. 731–736 (2001). https://doi.org/10.1109/ICCV.2001.937699
Torr, P.H., Zisserman, A.: MLESAC: a new robust estimator with application to estimating image geometry. Comput. Vis. Image Underst. 78(1), 138–156 (2000)
Tyleček, R., Šára, R.: Spatial pattern templates for recognition of objects with regular structure. In: GCPR: Pattern Recognition, pp. 364–374 (2013)
Yang, L., Sheng, Y., Wang, B.: 3D reconstruction of building facade with fused data of terrestrial lidar data and optical image. Optik Int. J. Light and Electron Opt. 127(4), 2165–2168 (2016). https://doi.org/10.1016/j.ijleo.2015.11.147
Yu, Q., Helmholz, P., Belton, D.: Semantically enhanced 3D building model reconstruction from terrestrial laser-scanning data. J. Surveying Eng. 143(4) (2017). https://doi.org/10.1061/(ASCE)SU.1943-5428.0000232
Zhang, Y., Funkhouser, T.A.: Deep depth completion of a single RGB-D image. CoRR abs/1803.09326 (2018)
Zhao, H., Shibasaki, R.: Reconstructing a textured cad model of an urban environment using vehicle-borne laser range scanners and line cameras. Mach. Vis. Appl. 14(1), 35–41 (2003)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Xu, H., Chen, CY., Delmas, P.J., Gee, T.E., van der Mark, W. (2019). Multimodal 3D Facade Reconstruction Using 3D LiDAR and Images. In: Lee, C., Su, Z., Sugimoto, A. (eds) Image and Video Technology. PSIVT 2019. Lecture Notes in Computer Science(), vol 11854. Springer, Cham. https://doi.org/10.1007/978-3-030-34879-3_22
Download citation
DOI: https://doi.org/10.1007/978-3-030-34879-3_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-34878-6
Online ISBN: 978-3-030-34879-3
eBook Packages: Computer ScienceComputer Science (R0)