research-article

A survey for image based methods in construction: from images to digital twins

Authors:

Ilias Koulalis,

Nikolaos Dourvas,

Theocharis Triantafyllidis,

Konstantinos Ioannidis,

Stefanos Vrochidis,

Ioannis KompatsiarisAuthors Info & Claims

CBMI '22: Proceedings of the 19th International Conference on Content-based Multimedia Indexing

Pages 103 - 110

https://doi.org/10.1145/3549555.3549594

Published: 07 October 2022 Publication History

Abstract

In the construction domain, Digital twins are mostly used for facilities management of buildings, but their applications are still very limited. The virtualization of buildings and bridges in the last 15 years in the form of Building or Bridge Information Models is clearly identified as the starting point for the DTs. The industry has erected a frame with semantically rich 3D reference models that are now heavily enriched with visual sensor data captured on construction sites. This article provides an overview of the research and current practices of computer vision methods in the construction industry and presents typical examples of their applications for 3D reconstruction, safety management and structural monitoring for quality assurance. It then highlights the dominant achievements presented in the literature and concludes with the challenges and research directions applicable to digital twins that need to be addressed and exploited in the future.

References

[1]

Mohamad Abbas, Bahaa Eddine Mneymneh, and Hiam Khoury. 2018. Assessing on-site construction personnel hazard perception in a Middle Eastern developing country: An interactive graphical approach. Safety Science 103(2018), 183–196.

[2]

Islam Asem Salah Abusohyon, Alberto Crupi, Faezeh Bagheri, and Flavio Tonelli. 2021. How to Set Up the Pillars of Digital Twins Technology in Our Business: Entities, Challenges and Solutions. Processes 9, 8 (2021), 1307.

[3]

Francisco Agueera-Vega, Fernando Carvajal-Ramirez, Patricio Martinez-Carricondo, Julián Sánchez-Hermosilla López, Francisco Javier Mesas-Carrascosa, Alfonso Garcia-Ferrer, and Fernando Juan Pérez-Porras. 2018. Reconstruction of extreme topography from UAV structure from motion photogrammetry. Measurement 121(2018), 127–138.

[4]

Raza Ali, Joon Huang Chuah, Mohamad Sofian Abu Talip, Norrima Mokhtar, and Muhammad Ali Shoaib. 2022. Structural crack detection using deep convolutional neural networks. Automation in Construction 133 (2022), 103989.

[5]

Autodesk. 2022. Autodesk Tandem. (2022). https://intandem.autodesk.com/

[6]

Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. 2006. Surf: Speeded up robust features. In European conference on computer vision. Springer, 404–417.

Digital Library

[7]

Bentley. 2021. An Open Platform for Infrastructure Digital Twins: How Bentley Systems’ iTwin Platform is Enabling a Digital Twin Ecosystem for the World’s Infrastructure. (2021).

[8]

A Broekman, GJ Jordaan, 2020. Digital twinning of asphalt pavement surfacings using Visual Simultaneous Localization and Mapping. In Advances in Materials and Pavement Performance Prediction II. CRC Press, 97–100.

[9]

Zaharah A Bukhsh, Andrej Anžlin, and Irina Stipanović. 2021. BiNet: Bridge Visual Inspection Dataset and Approach for Damage Detection. In International Conference of the European Association on Quality Control of Bridges and Structures. Springer, 1027–1034.

[10]

Young-Jin Cha, Wooram Choi, Gahyun Suh, Sadegh Mahmoudkhani, and Oral Büyüköztürk. 2018. Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Computer-Aided Civil and Infrastructure Engineering 33, 9(2018), 731–747.

Digital Library

[11]

Junyi Chai, Hao Zeng, Anming Li, and Eric WT Ngai. 2021. Deep learning in computer vision: A critical review of emerging techniques and application scenarios. Machine Learning with Applications 6 (2021), 100134.

[12]

Tamás Czimmermann, Gastone Ciuti, Mario Milazzo, Marcello Chiurazzi, Stefano Roccella, Calogero Maria Oddo, and Paolo Dario. 2020. Visual-based defect detection and classification approaches for industrial applications—a survey. Sensors 20, 5 (2020), 1459.

[13]

Juan Manuel Davila Delgado and Lukumon Oyedele. 2021. Digital Twins for the built environment: learning from conceptual and process models in manufacturing. Advanced Engineering Informatics 49 (2021), 101332.

Digital Library

[14]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.

[15]

Tom van Dijk and Guido de Croon. 2019. How do neural networks see depth in single images?. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2183–2191.

[16]

Piotr Dollár, Ron Appel, Serge Belongie, and Pietro Perona. 2014. Fast feature pyramids for object detection. IEEE transactions on pattern analysis and machine intelligence 36, 8(2014), 1532–1545.

Digital Library

[17]

Shan Du, Mohamed Shehata, and Wael Badawy. 2011. Hard hat detection in video sequences based on face features, motion and color information. In 2011 3rd international conference on computer research and development, Vol. 4. ieee, 25–29.

[18]

Qi Fang, Heng Li, Xiaochun Luo, Lieyun Ding, Hanbin Luo, Timothy M Rose, and Wangpeng An. 2018. Detecting non-hardhat-use by a deep learning method from far-field surveillance videos. Automation in Construction 85 (2018), 1–9.

[19]

Xin Feng, Youni Jiang, Xuejiao Yang, Ming Du, and Xin Li. 2019. Computer vision algorithms and hardware implementations: A survey. Integration 69(2019), 309–320.

Digital Library

[20]

Yasutaka Furukawa and Carlos Hernández. 2015. Multi-view stereo: A tutorial. Foundations and Trends® in Computer Graphics and Vision 9, 1-2(2015), 1–148.

[21]

Karim Hammoudi and Fadi Dornaika. 2011. A Featureless Approach to 3D Polyhedral Building Modeling from Aerial Images. Sensors (Basel, Switzerland) 11 (2011), 228 – 259.

[22]

Xian-Feng Han, Hamid Laga, and Mohammed Bennamoun. 2019. Image-based 3D object reconstruction: State-of-the-art and trends in the deep learning era. IEEE transactions on pattern analysis and machine intelligence 43, 5(2019), 1578–1604.

[23]

Tero Kaarlela, Sakari Pieskä, and Tomi Pitkäaho. 2020. Digital twin and virtual reality for safety training. In 2020 11th IEEE International Conference on Cognitive Infocommunications (CogInfoCom). IEEE, 000115–000120.

[24]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature 521, 7553 (2015), 436–444.

[25]

Jiewu Leng, Dewen Wang, Weiming Shen, Xinyu Li, Qiang Liu, and Xin Chen. 2021. Digital twins-based smart manufacturing system design in Industry 4.0: A review. Journal of manufacturing systems 60 (2021), 119–137.

[26]

Kang Li, Xiaoguang Zhao, Jiang Bian, and Min Tan. 2018. Automatic safety helmet wearing detection. arXiv preprint arXiv:1802.00264(2018).

[27]

Mingchao Li, Qiaorong Lu, Shuo Bai, Mengxi Zhang, Huijing Tian, and Liang Qin. 2021. Digital twin-driven virtual sensor approach for safe construction operations of trailing suction hopper dredger. Automation in Construction 132 (2021), 103961.

[28]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 740–755.

[29]

Caixia Liu, Dehui Kong, Shaofan Wang, Zhiyong Wang, Jinghua Li, and Baocai Yin. 2021. Deep3D reconstruction: methods, data, and challenges. Frontiers of Information Technology & Electronic Engineering 22, 5(2021), 652–672.

[30]

Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. 2016. Ssd: Single shot multibox detector. In European conference on computer vision. Springer, 21–37.

[31]

YT Liu, YN Yang, W Chao, XY Xu, and Tao Zhang. 2019. Research on Surface Defect Detection Based on Semantic Segmentation. DEStech Trans. Comput. Sci. Eng(2019).

[32]

Zhansheng Liu, Wenyan Bai, Xiuli Du, Anshan Zhang, Zezhong Xing, and Antong Jiang. 2020. Digital twin-based safety evaluation of prestressed steel structure. Advances in Civil Engineering 2020 (2020).

[33]

David G Lowe. 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision 60, 2 (2004), 91–110.

Digital Library

[34]

Bahaa Eddine Mneymneh, Mohamad Abbas, and Hiam Khoury. 2017. Automated hardhat detection for construction safety applications. Procedia engineering 196(2017), 895–902.

[35]

Nipun D Nath, Amir H Behzadan, and Stephanie G Paal. 2020. Deep learning for site safety: Real-time detection of personal protective equipment. Automation in Construction 112 (2020), 103085.

[36]

Hany Omar, Lamine Mahdjoubi, and Gamal Kheder. 2018. Towards an automated photogrammetry-based approach for monitoring and controlling construction site activities. Computers in Industry 98(2018), 172–182.

Digital Library

[37]

Suman Paneru and Idris Jeelani. 2021. Computer vision applications in construction: Current state, opportunities & challenges. Automation in Construction 132 (2021), 103940.

[38]

Man-Woo Park, Nehad Elsafty, and Zhenhua Zhu. 2015. Hardhat-wearing detection for enhancing on-site safety of construction workers. Journal of Construction Engineering and Management 141, 9(2015), 04015024.

[39]

Guiju Ping and Han Wang. 2019. 3D Reconstruction from A Single Image. In 2019 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM). IEEE, 47–52.

[40]

Prateek Prasanna, Kristin Dana, Nenad Gucunski, and Basily Basily. 2012. Computer-vision based crack detection and analysis. In Sensors and smart structures technologies for civil, mechanical, and aerospace systems 2012, Vol. 8345. SPIE, 1143–1148.

[41]

Varun Kumar Reja, Parth Bhadaniya, Koshy Varghese, and Quang Ha. 2021. Vision-Based Progress Monitoring of Building Structures Using Point-Intensity Approach. In ISARC. Proceedings of the International Symposium on Automation and Robotics in Construction, Vol. 38. IAARC Publications, 349–356.

[42]

Varun Kumar Reja, Koshy Varghese, and Quang Phuc Ha. 2022. Computer vision-based construction progress monitoring. Automation in Construction 138 (2022), 104245.

[43]

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 (2015).

[44]

Seungjun Roh, Zeeshan Aziz, and Feniosky Pena-Mora. 2011. An object-based 3D walk-through model for interior construction progress monitoring. Automation in Construction 20, 1 (2011), 66–75.

[45]

Abu HM Rubaiyat, Tanjin T Toma, Masoumeh Kalantari-Khandani, Syed A Rahman, Lingwei Chen, Yanfang Ye, and Christopher S Pan. 2016. Automatic detection of helmet uses for construction safety. In 2016 IEEE/WIC/ACM International Conference on Web Intelligence Workshops (WIW). IEEE, 135–142.

[46]

Johannes L Schonberger and Jan-Michael Frahm. 2016. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4104–4113.

[47]

Kishor Shrestha, Pramen P Shrestha, Dinesh Bajracharya, and Evangelos A Yfantis. 2015. Hard-hat detection for construction safety visualization. Journal of Construction Engineering 2015, 1 (2015), 1–8.

[48]

Bilal Succar. 2009. Building information modelling framework: A research and delivery foundation for industry stakeholders. Automation in construction 18, 3 (2009), 357–375.

[49]

Richard Szeliski. 2010. Computer vision: algorithms and applications. Springer Science & Business Media.

Digital Library

[50]

Hao Wu and Jinsong Zhao. 2018. An intelligent vision-based approach for helmet identification for work safety. Computers in Industry 100 (2018), 267–277.

[51]

Jixiu Wu, Nian Cai, Wenjie Chen, Huiheng Wang, and Guotian Wang. 2019. Automatic detection of hardhats worn by construction personnel: A deep learning approach and benchmark dataset. Automation in Construction 106 (2019), 102894.

[52]

Zaipeng Xie, Hanxiang Liu, Zewen Li, and Yuechao He. 2018. A convolutional neural network based approach towards real-time hard hat detection. In 2018 IEEE International Conference on Progress in Informatics and Computing (PIC). IEEE, 430–434.

[53]

B Xu and Y Huang. 2005. Automatic inspection of pavement cracking distress. In Applications of Digital Image Processing XXVIII, Vol. 5909. SPIE, 590901.

[54]

Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. 2010. Deconvolutional networks. In 2010 IEEE Computer Society Conference on computer vision and pattern recognition. IEEE, 2528–2535.

[55]

Allen Zhang, Kelvin CP Wang, Baoxian Li, Enhui Yang, Xianxing Dai, Yi Peng, Yue Fei, Yang Liu, Joshua Q Li, and Cheng Chen. 2017. Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network. Computer-Aided Civil and Infrastructure Engineering 32, 10(2017), 805–819.

[56]

Lei Zhang, Fan Yang, Yimin Daniel Zhang, and Ying Julie Zhu. 2016. Road crack detection using deep convolutional neural network. In 2016 IEEE international conference on image processing (ICIP). IEEE, 3708–3712.

[57]

Zhiheng Zhao, Leidi Shen, Chen Yang, Wei Wu, Mengdi Zhang, and George Q Huang. 2021. IoT and digital twin enabled smart tracking for safety management. Computers & Operations Research 128 (2021), 105183.

[58]

Botao Zhong, Haitao Wu, Lieyun Ding, Peter ED Love, Heng Li, Hanbin Luo, and Li Jiao. 2019. Mapping computer vision research in construction: Developments, knowledge gaps and implications for research. Automation in Construction 107 (2019), 102919.

[59]

Zhenhua Zhu, Man-Woo Park, and Nehad Elsafty. 2015. Automated monitoring of hardhats wearing for onsite safety enhancement. (2015).

[60]

Aleš Žnidarič, Maja Kreslin, Andrej Anžlin, and Andarž Krivic. 2020. Detection of Delaminated and Cracked Concrete with Unmanned Aerial Vehicles. Routes/Roads 386(2020).

Cited By

Wei RPei HWu DZeng CAi XDuan H(2024)A Semantically Aware Multi-View 3D Reconstruction Method for Urban ApplicationsApplied Sciences10.3390/app1405221814:5(2218)Online publication date: 6-Mar-2024
https://doi.org/10.3390/app14052218
Fişne AKalay AEken S(2024)Fast and efficient computing for deep learning-based defect detection models in lightweight devicesJournal of Intelligent Manufacturing10.1007/s10845-024-02487-zOnline publication date: 23-Sep-2024
https://doi.org/10.1007/s10845-024-02487-z
Vrachnos PKrestenitis MKoulalis IIoannidis KVrochidis S(2024)A Framework for 3D Modeling of Construction Sites Using Aerial Imagery and Semantic NeRFsMultiMedia Modeling10.1007/978-3-031-53302-0_13(175-187)Online publication date: 29-Jan-2024
https://doi.org/10.1007/978-3-031-53302-0_13
Show More Cited By

Index Terms

A survey for image based methods in construction: from images to digital twins
1. Human-centered computing
  1. Visualization

Recommendations

Intelligent Building Construction Management Based on BIM Digital Twin
In order to improve the construction effect of intelligent buildings, this paper combines the BIM digital twin technology to construct the overall structure of the building construction operation and maintenance system driven by the BIM digital twin. ...
A Requirements Study on Model Repositories for Digital Twins in Construction Engineering
Cooperative Information Systems
Abstract
Building information modeling is becoming the preferred tool-assisted methodology in civil and construction engineering for the design, management, and creation of digital replicas of buildings. However, current tool support for creating and ...
Digital Twins in Seaports: Current and Future Applications
Computational Logistics
Abstract
Information systems in major seaports have evolved to digital hubs where state-of-the-art technologies are used to gather and analyze (near) real-time data from infrastructure and superstructure. Digital twins are seen as key enabler of Industry ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

CBMI '22: Proceedings of the 19th International Conference on Content-based Multimedia Indexing

September 2022

208 pages

ISBN:9781450397209

DOI:10.1145/3549555

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CBMI 2022

CBMI 2022: International Conference on Content-based Multimedia Indexing

September 14 - 16, 2022

Graz, Austria

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
180
Total Downloads

Downloads (Last 12 months)86
Downloads (Last 6 weeks)8

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wei RPei HWu DZeng CAi XDuan H(2024)A Semantically Aware Multi-View 3D Reconstruction Method for Urban ApplicationsApplied Sciences10.3390/app1405221814:5(2218)Online publication date: 6-Mar-2024
https://doi.org/10.3390/app14052218
Fişne AKalay AEken S(2024)Fast and efficient computing for deep learning-based defect detection models in lightweight devicesJournal of Intelligent Manufacturing10.1007/s10845-024-02487-zOnline publication date: 23-Sep-2024
https://doi.org/10.1007/s10845-024-02487-z
Vrachnos PKrestenitis MKoulalis IIoannidis KVrochidis S(2024)A Framework for 3D Modeling of Construction Sites Using Aerial Imagery and Semantic NeRFsMultiMedia Modeling10.1007/978-3-031-53302-0_13(175-187)Online publication date: 29-Jan-2024
https://doi.org/10.1007/978-3-031-53302-0_13
Kaur A(2023)CAD Design for CNC Equipments and Industrial Manufacturing Using CNN2023 International Conference on Data Science and Network Security (ICDSNS)10.1109/ICDSNS58469.2023.10245143(1-7)Online publication date: 28-Jul-2023
https://doi.org/10.1109/ICDSNS58469.2023.10245143
Chacón RCasas JRamonell CPosada HStipanovic IŠkarić S(2023)Requirements and challenges for infusion of SHM systems within Digital Twin platformsStructure and Infrastructure Engineering10.1080/15732479.2023.2225486(1-17)Online publication date: 28-Jun-2023
https://doi.org/10.1080/15732479.2023.2225486
Schönfelder PAziz AFaltin BKönig M(2023)Automating the retrospective generation of As-is BIM models using machine learningAutomation in Construction10.1016/j.autcon.2023.104937152(104937)Online publication date: Aug-2023
https://doi.org/10.1016/j.autcon.2023.104937

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents