An Image-Based Convolutional Neural Network System For Road Defects Detection
An Image-Based Convolutional Neural Network System For Road Defects Detection
Corresponding Author:
Mohamed Anis Benallal
Laboratoire SIMPA, Département d’informatique, Faculté des Mathématiques et Informatique, Université des
Sciences et de la Technologie d’Oran Mohamed Boudiaf (USTO-MB)
BP 1505, El M’naouer, Oran, Algeria
Email: mohamed.benallal@univ-usto.dz
1. INTRODUCTION
Countries around the world are likely to encounter the current issues with infrastructure management
and maintenance [1]. Traditionally, monitoring the road surface condition was done manually, but this is still
the case in our country. There are many ways to build a road. However, the great majority of our roads are
constructed of an asphalt concrete layer, followed by one or more layers of angular rock, known as the base
course (also called tarmac). Despite being a cost-effective solution, as it provides smooth and durable surface,
roads suffer from a set of problems. The most encountered ones are cracks and potholes. Sunlight, rain, snow,
sleet, and freezing weather, in addition to cars and trucks traffic contribute to the slow process of altering the
road surface.
Since roadways infrastructure is one of the highest value assets owned by a country, maintaining it has
a huge prevalence and a direct effect on the economy and the prosperity of people and businesses in a country,
that is, to extend the life of an asphalt surface, it needs to be repaired as soon as a defect appears. That means
that detecting a damage earlier provides a way of sealing and maintaining a correct road surface quality, and its
Int J Artif Intell, Vol. 12, No. 2, June 2023 : 577 – 584
Int J Artif Intell ISSN: 2252-8938 r 579
inary results of an experimental application of the YOLO deep-learning technique on tackling the problem of
detecting cracks and potholes based on road images, that is, more investigation should be provided by applying
YOLO on other regions.
2. METHOD
This section presents the used algorithm and how it can be applied on our study. The details about
the used database are also presented to explain how data are used in the algorithm. In fact, raw images of road
surface had been prepared and sampled through different steps.
The intersection over union (IoU) is then calculated to group together the anchor boxes that detect the
same object (we can see this through the groups of colours in the bottom-right). YOLO uses IoU to provide
a box that surrounds the whole targeting object. Basically, it divides the overlap area between the boxes by
An image-based convolutional neural network system for road defects detection (Mohamed Anis Benallal)
580 r ISSN: 2252-8938
the union of those areas. Then finally, thanks to the non-max suppression algorithm included in the YOLO
algorithm, we go from thousands of anchor boxes to only 5 in this example (one for cracks and four for
potholes in the bottom-left).
(a) (b)
Figure 2. Examples of the studied defects (YOLO algorithm inputs); (a) cracks example and (b) potholes
examples
Int J Artif Intell, Vol. 12, No. 2, June 2023 : 577 – 584
Int J Artif Intell ISSN: 2252-8938 r 581
Working with YOLO implies defining a set of parameters to perform the learning process. In fact, it
is implemented based on the darknet, an open source neural networks. Note that we used the v4 version of the
YOLO algorithm [38]. Actually , this version remains the best choice to balance between accuracy and custom
configuration. Having two classes, we need to set the configuration file according to, that is: line classes to
2, max batches to the minimum of 6,000, line steps to 80% and 90% of max batches and filters to 21. All of
these parameters work in a network sized at 416 by 416. Due to Colab limitations (in its free version), a total of
5,700 iterations were performed. To test the performance of our model, and in addition to the selected images
for training, we also selected 650 images from the used database in which 450 visually contain similar learned
defects and the rest of 200 images which contain no learned defects (to test false positive cases) or no defects
at all. Note that these 650 images have not been used in any step of the learning process. Indeed, we consider
that doing so ensures a real and true test to the generalization capacity of the trained network.
To provide this step, we injected all these testing images to the model and then checked manually all
the resulting images one by one. For every defect (cracks in all types and potholes), we went through counting
the correctness of the detection. Note that we can have many defects in the same image. Figure 3 presents
examples of a visual representation for the obtained results. As we can see in its sub-figures, boxes provided
by the YOLO algorithm make it capable of: i) avoiding any detection where there is no defect (Figure 3(a)), ii)
Detecting multiple defects of the same class (Figure 3(b) for cracks and Figure 3(c) for potholes), and detecting
multiple defects of the different classes (Figure 3(d)).
(a) (b)
(c) (d)
Figure 3. Examples of the algorithm result detections; (a) no detection, (b) cracks detection, (c) potholes
detection, and (d) both crack and pothole detection
Table 1 shows that the testing step, considering the testing dataset as different from the training one
and with a comparable number of images, brings us a general detection rate of 91% with approximately 90%
and 93% for cracks and potholes respectively. Furthermore, we got an insignificant number of false positive
cases for both classes. These results show that, even with taking into consideration multiple types of cracks
in the same class, this gives an acceptable detection rate. However, this could be certainly improved with
creating multiple classes which anyone correspond to a specific type of cracks. For potholes, even if results are
An image-based convolutional neural network system for road defects detection (Mohamed Anis Benallal)
582 r ISSN: 2252-8938
similar to those of cracks, we consider that we should have a larger number of images to be more consistent.
Nevertheless, that lead us to work on other databases, but also create and publish a local one(since local roads
significantly suffer from poor road conditions).
4. CONCLUSION
We presented in the paper an experimental test on detecting the two main classes of road defects
(cracks and potholes). The observed lack of studies involving YOLO as a multiple and dedicated road-damages
detector induce us to answer the question of, to what extent, the YOLO can be an algorithm to consider using
in road surface damages detection and classification. An overall rate of 91% (90% and 93% for cracks and
potholes respectively) was obtained using a Brazilian road images database. As a first attempt, we consider
these results favourable to more investigations. Furthermore, this comforts us in using the YOLO algorithm as
model-based technique to provide a more powerful and versatile system to detect better cracks and potholes,
but also work on other kinds of defects. Therefore, this lead us to focus on two main aspects to improve our
out-coming results: i) capture videos/images, in order to create a local database of Algerian road defects (work
in progress at the time of this paper to provide more images of the studied problem but also to build an effective
local detector), and ii) take into consideration the differences between defect features of the same class. In fact,
our hypothesis about the observed results of cracks detection rate is mainly due to the choice of combining
multiple cracks types, in terms of directions and shape. Overall, results are very motivating to explore the
model optimisation in terms of input data, meta-parametrization, and model training.
REFERENCES
[1] H. Maeda, Y. Sekimoto, T. Seto, T. Kashiyama, and H. Omata, “Road damage detection and classification using deep
neural networks with smartphone images,” Computer-Aided Civil and Infrastructure Engineering, vol. 33, no. 12, pp.
1127–1141, 2018, doi: 10.1111/mice.12387.
[2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,”
Communications of the ACM, vol. 60, no. 6, pp. 84–90, May 2017, doi: 10.1145/3065386.
[3] I. Hachchane, A. Badri, A. Sahel, I. Elmourabit, and Y. Ruichek, “Image and video face retrieval with query image
using convolutional neural network features,” IAES International Journal of Artificial Intelligence (IJ-AI), vol. 11,
no. 1, pp. 102-109, Mar. 2022, doi: 10.11591/ijai.v11.i1.pp102-109.
[4] N. A-Alam, M. Ahsan, M. A. Based, J. Haider, and M. Kowalski, “An intelligent system for automatic fingerprint
identification using feature fusion by Gabor filter and deep learning,” Computers and Electrical Engineering, vol. 95,
p. 107387, Oct. 2021, doi: 10.1016/j.compeleceng.2021.107387.
[5] K. S. Ananda Kumar, A. Y. Prasad, and J. Metan, “A hybrid deep CNN-Cov-19-Res-Net transfer learning architype
for an enhanced brain tumor detection and classification scheme in medical image processing,” Biomedical Signal
Processing and Control, vol. 76, p. 103631, Jul. 2022, doi: 10.1016/j.bspc.2022.103631.
[6] S. A. Shadab, M. A. Ansari, N. Singh, A. Verma, P. Tripathi, et R. Mehrotra, “Detection of cancer from histopathol-
ogy medical image data using ML with CNN ResNet-50 architecture,” in Computational Intelligence in Healthcare
Applications, Elsevier, 2022, pp. 237-254, doi: 10.1016/B978-0-323-99031-8.00007-7.
[7] M. Sha and A. Boukerche, “Performance evaluation of CNN-based pedestrian detectors for autonomous vehicles,”
Ad Hoc Networks, vol. 128, p. 102784, Apr. 2022, doi: 10.1016/j.adhoc.2022.102784.
[8] S. Narmadha and V. Vijayakumar, “Spatio-temporal vehicle traffic flow prediction using multivariate CNN and LSTM
model,” Materials Today: Proceedings, May 2021, doi: 10.1016/j.matpr.2021.04.249.
[9] W. S. Aji, K. H. Bin Ghazali, and S. A. Akbar, “Oil palm unstripped bunch detector using modified faster regional
convolutional neural network,” IAES International Journal of Artificial Intelligence (IJ-AI), vol. 11, no. 1, pp. 189-
200, Mar. 2022, doi: 10.11591/ijai.v11.i1.pp189-200.
[10] M. A. Anwer, S. M. Shareef, and A. M. Ali, “Accident vehicle types classification: a comparative study between
different deep learning models,” Indonesian Journal of Electrical Engineering and Computer Science (IJEECS), vol.
21, no. 3, pp. 1474-1484, Mar. 2021, doi: 10.11591/ijeecs.v21.i3.pp1474-1484.
Int J Artif Intell, Vol. 12, No. 2, June 2023 : 577 – 584
Int J Artif Intell ISSN: 2252-8938 r 583
[11] K. Gayathri and S. Thangavelu, “Novel deep learning model for vehicle and pothole detection,” Indonesian Jour-
nal of Electrical Engineering and Computer Science (IJEECS), vol. 23, no. 3, pp. 1576-1582, Sep. 2021, doi:
10.11591/ijeecs.v23.i3.pp1576-1582.
[12] J. Shi, Y. Chang, C. Xu, F. Khan, G. Chen, and C. Li, “Real-time leak detection using an infrared cam-
era and Faster R-CNN technique,” Computers & Chemical Engineering, vol. 135, p. 106780, Apr. 2020, doi:
10.1016/j.compchemeng.2020.106780.
[13] D. Biswas, H. Su, C. Wang, A. Stevanovic, and W. Wang, “An automatic traffic density estimation using single shot
detection (SSD) and MobileNet-SSD,” Physics and Chemistry of the Earth, Parts A/B/C, vol. 110, p. 176-184, Apr.
2019, doi: 10.1016/j.pce.2018.12.001.
[14] W. Liu et al., “SSD: single shot multibox detector,” in European conference on computer vision (ECCV 2016),
Springer International Publishing, 2016, pp. 21–37. doi: 10.1007/978-3-319-46448-0 2.
[15] J. Azimjonov and A. Özmen, “A real-time vehicle detection and a novel vehicle tracking systems for estimating
and monitoring traffic flow on highways,” Advanced Engineering Informatics, vol. 50, p. 101393, Oct. 2021, doi:
10.1016/j.aei.2021.101393.
[16] Y.-J. Cha, W. Choi, and O. Büyüköztürk, “Deep learning-based crack damage detection using convolutional neu-
ral networks,” Computer-Aided Civil and Infrastructure Engineering, vol. 32, no. 5, pp. 361–378, May 2017, doi:
10.1111/mice.12263.
[17] Q. Fang et al., “A deep learning-based method for detecting non-certified work on construction sites,” Advanced
Engineering Informatics, vol. 35, pp. 56–68, Jan. 2018, doi: 10.1016/j.aei.2018.01.001.
[18] Y. Pi, N. D. Nath, and A. H. Behzadan, “Convolutional neural networks for object detection in aerial imagery
for disaster response and recovery,” Advanced Engineering Informatics, vol. 43, p. 101009, Jan. 2020, doi:
10.1016/j.aei.2019.101009.
[19] W. Dai, A. Mujeeb, M. Erdt, and A. Sourin, “Soldering defect detection in automatic optical inspection,” Advanced
Engineering Informatics, vol. 43, p. 101004, Jan. 2020, doi: 10.1016/j.aei.2019.101004.
[20] H. Bello-Salau, A. M. Aibinu, A. J. Onumanyi, E. N. Onwuka, J. J. Dukiya, and H. Ohize, “New road anomaly
detection and characterization algorithm for autonomous vehicles,” Applied Computing and Informatics, vol. 16, no.
1/2, pp. 223–239, May 2018, doi: 10.1016/j.aci.2018.05.002.
[21] P. M. Harikrishnan and V. P. Gopi, “Vehicle vibration signal processing for road surface monitoring,” IEEE Sensors
Journal, vol. 17, no. 16, pp. 5192–5197, Aug. 2017, doi: 10.1109/JSEN.2017.2719865.
[22] T. Nomura and Y. Shiraishi, “A method for estimating road surface conditions with a smartphone,” International
Journal of Informatics Society, vol. 7, no. 1, pp. 29–36, 2015.
[23] R. Madli, S. Hebbar, P. Pattar, and V. Golla, “Automatic detection and notification of potholes and humps on roads to
aid drivers,” IEEE Sensors Journal, vol. 15, no. 8, pp. 4313–4318, Aug. 2015, doi: 10.1109/JSEN.2015.2417579.
[24] X. Yu and E. Salari, “Pavement pothole detection and severity measurement using laser imaging,” in 2011 IEEE
International Conference on Electro/Information Technology, May 2011, pp. 1–5. doi: 10.1109/EIT.2011.5978573.
[25] H. Wu et al., “Road pothole extraction and safety evaluation by integration of point cloud and images de-
rived from mobile mapping sensors,” Advanced Engineering Informatics, vol. 42, p. 100936, Oct. 2019, doi:
10.1016/j.aei.2019.100936.
[26] M. Gavilán et al., “Adaptive road crack detection system by pavement classification,” Sensors, vol. 11, no. 10, pp.
9628–9657, Oct. 2011, doi: 10.3390/s111009628.
[27] T. S. Nguyen, M. Avila, and S. Begot, “Automatic detection and classification of defect on road pavement using
anisotropy measure,” in 2009 17th European Signal Processing Conference, Aug. 2009, pp. 617–621.
[28] M. H. Yousaf, K. Azhar, F. Murtaza, and F. Hussain, “Visual analysis of asphalt pavement for detec-
tion and localization of potholes,” Advanced Engineering Informatics, vol. 38, pp. 527–537, Oct. 2018, doi:
10.1016/j.aei.2018.09.002.
[29] S. C. Radopoulou and I. Brilakis, “Automated detection of multiple pavement defects,” Journal of Computing in Civil
Engineering, vol. 31, no. 2, p. 4016057, Mar. 2017, doi: 10.1061/(asce)cp.1943-5487.0000623..
[30] D. Kassem and C. Arce-Lopera, “Road-condition monitoring and classification for smart cities,” in Advances in Arti-
ficial Intelligence, Software and Systems Engineering. AHFE 2020. Advances in Intelligent Systems and Computing,
Jul. 2020, pp. 437–441. doi: 10.1007/978-3-030-51328-3 60.
[31] S. Li et al., “Detection of concealed cracks from ground penetrating radar images based on deep learning algorithm,”
Construction and Building Materials, vol. 273, p. 121949, Mar. 2021, doi: 10.1016/j.conbuildmat.2020.121949.
[32] Y. Zhang, J. Huang, and F. Cai, “On bridge surface crack detection based on an improved YOLO v3 algorithm,”
IFAC-PapersOnLine, vol. 53, no. 2, pp. 8205–8210, Jan. 2020, doi: 10.1016/j.ifacol.2020.12.1994.
[33] Y. Du, N. Pan, Z. Xu, F. Deng, Y. Shen, and H. Kang, “Pavement distress detection and classification based on
YOLO network,” International Journal of Pavement Engineering, vol. 22, no. 13, pp. 1659–1672, Nov 2021, doi:
10.1080/10298436.2020.1714047.
[34] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: unified, real-time object detection,” in
Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016, vol.
An image-based convolutional neural network system for road defects detection (Mohamed Anis Benallal)
584 r ISSN: 2252-8938
BIOGRAPHIES OF AUTHORS
Int J Artif Intell, Vol. 12, No. 2, June 2023 : 577 – 584