SAVITRIBAI PHULE PUNE UNIVERSITY

A PROJECT REPORT ON

“Deep Learning Approach To Detect And Classify Road

Surface Damage Using Mobile Sensors.”
SUBMITTED TO THE SAVITRIBAI PHULE PUNE UNIVERSITY, PUNE
IN THE PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE AWARD
OF THE DEGREE

BACHELOR OF ENGINEERING
Computer Engineering
SUBMITTED BY

Mr.Sameer Dnyanoba Arote Exam No: B150104207

Mr.Ganesh Manish Gopale Exam No: B150104247
Mr.Samir Ramdas Aher Exam No: B150104201
Mr.Yogeshwar Sambhaji Ghule Exam No:B150104246

Under The Guidance of

Ms. K.U. Rahane

DEPARTMENT OF COMPUTER ENGINEERING

Amrutvahini College of Engineering, Sangamner.
A/P Ghulewadi, Sangamner, Dis-Ahmednagar,
Maharashtra-422605

2021-22
 AMRUTVAHINI COLLEGE OF ENGINEERING,SANGAMNER
DEPARTMENT OF COMPUTER ENGINEERING

CERTIFICATE

This is certify that the project entitled

“Deep Learning Approach To Detect And Classify Road Surface
Damage Using Mobile Sensors.“
submitted by

Mr.Sameer Dnyanoba Arote B150104207

Mr.Ganesh Manish Gopale B150104247
Mr. Samir Ramdas Aher B150104201
Mr. Yogeshwar Sambhaji Ghule B150104246

are bonafide students of this institute and the work has been carried out by them
under the guidance of Ms.K.U.Rahane and it is approved for the partial fulfillment
of the requirement of Savitribai Phule Pune University, for the award of the degree
of Bachelor of Engineering (Computer Engineering).

Date: / /

( Ms. K.U.Rahane ) (Dr. M. A. Wakchaure)

Project Guide Project Coordinator

(Prof. R. L. Paikrao) (Dr. M. A. Venkatesh )

HOD, Computer Department Principal
i
 SAVITRIBAI PHULE PUNE UNIVERSITY

CERTIFICATE
This is to certify that,
Mr. Sameer Dnyanoba Arote [B150104207]
Mr.Ganesh Manish Gopale [B150104247]
Mr. Samir Ramdas Aher [B150104201]
Mr. Yogeshwar Sambhaji Ghule [B150104246]

of BE in Computer Engineering was examined in the

Project Examination entitled
”Deep Learning Approach To Detect And Classify
Road Surface Damage Using Mobile Sensors”

on / /2022
At
DEPARTMENT OF COMPUTER ENGINEERING
Amrutvahini College of Engineering, Sangamner

Internal Examiner External Examiner

 Acknowledgement

Achievement is Finding out what you have been doing and what you have to do.
The higher is summit, the harder is climb. The goal was fixed and we began with
the determined resolved and put in a ceaseless sustained hard work. Greater the
challenge, greater was our determination and it guided us to overcome all difficulties.
It has been rightly said that we are built on the shoulders of others. For everything we
have achieved, the credit goes to who had really help us to complete this project and
for the timely guidance and infrastructure. Before we proceed any further, We would
like to thank all those who have helped us in all the way through. To start with we
thank our Honorable Principal ,Dr. M. A. Venkatesh , for his encouragement and
support, our respected Head of Department,Prof. R. L. Paikrao,we would also like
to take this opportunity to thanks to project Coordinator Dr. M. A.Wakchaure and
also thankful to our guide Ms. K. U. Rahane, for her guidance, care and support.
At last we must express our sincere heartfelt gratitude to all the staff members of
Computer Engineering Department who helped us directly or indirectly during this
course of work.

Mr.Sameer Dnyanoba Arote

Mr.Ganesh Manish Gopale
Mr.Samir Ramdas Aher
Mr.Yogeshwar Sambhaji Ghule
(B.E. Computer Engg.)

ii
 ABSTRACT

Road network is an important asset of country’s development. Poor road surface

condition create risk of vehicle damage and increase the chances of road accidents,
and eventually affect development and other services which depends heavily on road
conditions. Due to very large network of roads, road condition monitoring is a chal-
lenging problem. Traditional road monitoring is done by human inspection, which
is very time consuming and costly.
Computer Vision is a computer science field where an image can be processed to
extract and classify features based on this technology. we propose a deep learning
approach to classify and identify road surface damage using a low quality images
taken from a mobile camera mounted on a vehicle. This solution completely elimi-
nates human dependency of the road condition monitoring process, and able to work
at high speed and with very low cost.

iii
 Contents
CERTIFICATE i

ACKNOWLEDGEMENT iii

ABSTRACT iv

LIST OF FIGURES viii

LIST OF TABLES ix

ABBREVATIONS 2

1 Introduction 3
1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Motivation of the Project . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Organization of Report . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Literature Survey 8

3 PROBLEM DEFINITION AND SCOPE 11

3.1 Limitation of existing system . . . . . . . . . . . . . . . . . . . . 11
3.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2.1 Goals and Objectives . . . . . . . . . . . . . . . . . . . . . 11
3.3 Mathematical Model . . . . . . . . . . . . . . . . . . . . . . . . . 12

4 Software Requirements Specification 14

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.1.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.1.2 Assumptions and Dependencies . . . . . . . . . . . . . . . 14
4.2 External Interface Requirements . . . . . . . . . . . . . . . . . . . 14
4.2.1 User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . 14
4.2.2 Hardware Interfaces . . . . . . . . . . . . . . . . . . . . . 14
4.2.3 Software Interfaces . . . . . . . . . . . . . . . . . . . . . . 15
4.3 Nonfunctional Requirements . . . . . . . . . . . . . . . . . . . . . 15
4.3.1 Performance Requirement . . . . . . . . . . . . . . . . . . 15
4.3.2 Safety Requirements . . . . . . . . . . . . . . . . . . . . . 15
4.3.3 Software Quality Attributes . . . . . . . . . . . . . . . . . 15

iv
 4.4 Other Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.4.1 Hardware Requirements . . . . . . . . . . . . . . . . . . . 16
4.4.2 Software Requirements . . . . . . . . . . . . . . . . . . . . 16

5 Methodology AND System Design 17

5.1 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.2 Data Flow Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.2.1 Level 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.2.2 Level 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.2.3 Level 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.3 UML Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.3.1 Use Case Diagram . . . . . . . . . . . . . . . . . . . . . . 20
5.3.2 Activity Diagram . . . . . . . . . . . . . . . . . . . . . . . 21
5.3.3 Sequence Diagram . . . . . . . . . . . . . . . . . . . . . . 23
5.3.4 Class Diagram . . . . . . . . . . . . . . . . . . . . . . . . 24

6 Technical Specification 25
6.1 Technology details . . . . . . . . . . . . . . . . . . . . . . . . . . 25
6.1.1 python . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
6.1.2 JavaScript . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
6.1.3 MySQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
6.1.4 REST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

7 PROJECT ESTIMATE, SCHEDULE AND TEAM STRUCTURE 28

7.1 Project Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
7.1.1 COCOMO Model . . . . . . . . . . . . . . . . . . . . . . 28
7.1.2 Equations: . . . . . . . . . . . . . . . . . . . . . . . . . . 30
7.1.3 Organic project: . . . . . . . . . . . . . . . . . . . . . . . 30
7.1.4 Analysis Models: SDLC Model to be applied . . . . . . . . 31
7.2 Project plan and Team Structure . . . . . . . . . . . . . . . . . . . 33
7.3 System Implementation Plan . . . . . . . . . . . . . . . . . . . . . 36
7.3.1 Systematic plan . . . . . . . . . . . . . . . . . . . . . . . 36
7.3.2 Timeline Chart . . . . . . . . . . . . . . . . . . . . . . . . 36

8 Software Implementation 37
8.1 Dataset Description . . . . . . . . . . . . . . . . . . . . . . . . . . 37
8.2 Project modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
8.2.1 CNN Model . . . . . . . . . . . . . . . . . . . . . . . . . . 38
 8.2.2 Data Collection . . . . . . . . . . . . . . . . . . . . . . . 39
8.2.3 Data Distribution . . . . . . . . . . . . . . . . . . . . . . . 39
8.3 Algorithms used . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
8.3.1 YOLO V4 Model . . . . . . . . . . . . . . . . . . . . . . . 40
8.3.2 Steps for Model Creation . . . . . . . . . . . . . . . . . . . 42

9 Software Testing And Validation 43

9.1 Type of Testing done . . . . . . . . . . . . . . . . . . . . . . . . . 43
9.2 Test cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
9.3 Risk Management . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
9.3.1 Risk Identification . . . . . . . . . . . . . . . . . . . . . . 51
9.3.2 Risk Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 52

10 Advantages, Limitations And Applications 53

10.1 Advantages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
10.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
10.3 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

11 Results And Analysis 54

11.1 Login Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
11.2 Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
11.2.1 Data Distribution Platform . . . . . . . . . . . . . . . . . . 59

12 Conclusion And Future Scope 61

12.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
12.2 Future Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

References 61

Annexure A 62

Annexure B 65

vi
 List of Figures
5.1 Block Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.2 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.3 Data Flow Diagram (Level 0) . . . . . . . . . . . . . . . . . . . . . 19
5.4 Data Flow Diagram (Level 1) . . . . . . . . . . . . . . . . . . . . . 19
5.5 Data Flow Diagram (Level 2 ) . . . . . . . . . . . . . . . . . . . . 20
5.6 Use Case Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.7 Activity Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.8 Sequence Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . 23
5.9 Class Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

6.1 Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
6.2 JavaScript . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
6.3 SQL Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
6.4 REST API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

7.1 Software Development Life Cycle . . . . . . . . . . . . . . . . . . 32

7.2 Project Work Breakdown Structure . . . . . . . . . . . . . . . . . 33
7.3 Time Line Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

8.1 Damage Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

8.2 Dataset Line Chart . . . . . . . . . . . . . . . . . . . . . . . . . . 38
8.3 Yolo Model 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
8.4 Yolo Model 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
8.5 Yolo V5 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 41

9.1 Login Successfully . . . . . . . . . . . . . . . . . . . . . . . . . . 47

9.2 Camera And GPS Connection Successful . . . . . . . . . . . . . . 47
9.3 Detection Successfully . . . . . . . . . . . . . . . . . . . . . . . . 48
9.4 Login Failed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
9.5 Survey Page test . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
9.6 Sensor test Failed . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
9.7 Road code missing . . . . . . . . . . . . . . . . . . . . . . . . . . 50
9.8 camera and GPS test Failed . . . . . . . . . . . . . . . . . . . . . 50
9.9 GPS Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

11.1 Login Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

11.2 Survey Page 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
11.3 Survey Page 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

vii
 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

11.4 Survey Page 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

11.5 Survey Page 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
11.6 Detection Page 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
11.7 Detection Page 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
11.8 Detection Page 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
11.9 Detection Page 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
11.10Detection Page 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
11.11Survey or Information . . . . . . . . . . . . . . . . . . . . . . . . . 59
11.12Survey Information . . . . . . . . . . . . . . . . . . . . . . . . . . 60
11.13Survey Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

12.1 Plagiarism Report . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

12.2 Plagiarism Report . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
12.3 Plagiarism Report . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
12.4 Certificate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
12.5 Certificate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
12.6 Certificate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
12.7 Certificate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

AVCOE, Sangamner viii Department of Computer Engineering

 List of Tables
4.1 Hardware List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

7.1 Modes of development . . . . . . . . . . . . . . . . . . . . . . . . 29

7.2 List of Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
7.3 Task Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
7.4 List of Developers . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
7.5 Implementation Plan . . . . . . . . . . . . . . . . . . . . . . . . . 36

9.1 Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

ix
 ABBREVATIONS
• YOLO - you only look once

• CNN - convolutional neural network

• AI - artificial intelligence

• DPM - Deformable Parts Model

• SQL - Structured Query Language

• DFD - data flow diagram

• RDBMS - relational database management system

• LAMP - Linux, Apache, MySQL, Perl/PHP/Python.

• CSPNet - Cross Stage Partial Networks

• PANet - Path Aggregation Network

• JVM - Java virtual machine

1
 Chapter 1

Introduction
1.1 Overview
Maintaining good road quality is therefore essential not only to support an efficient
road network but also to minimize risk of traffic accidents. However, maintaining
roads regularly is a challenging task due to the heavy traffic, weather conditions and
high costs of manpower. Since frequent repairs are needed to prevent road quality
from deteriorating, a reliable and low-latency road condition monitoring system is
highly desirable to identify critical road segments and to optimally allocate limited
maintenance resources.
Road infrastructure is a crucial public asset as it contributes to economic devel-
opment and growth while bringing critical social benefits. It connects communities
and businesses and provides access to education, employment, social, and health ser-
vices. However, road surface wears and deteriorates over time from factors related
to location, age, traffic volume, weather, engineering solutions, and materials being
used to build it.
Pavement distress, one of the pavement condition characteristics, is typically eval-
uated using one of two approaches: manual, semi-automated. The traditional meth-
ods for obtaining pavement condition data include manual and semi-automated sur-
veys. In manual surveys, raters perform a visual inspection of the pavement sur-
face through either walking on or along the pavement surface or by conducting a
windshield survey from a slow-moving vehicle. Visual inspection of the road sur-
face suffers from a subjective judgment of inspectors. It requires a significant hu-
man intervention that is proven to be time consuming, given the extensive length of
road networks. Moreover, inspectors must often be physically present in the travel
lane, exposing themselves to potentially hazardous conditions. In semi-automated
pavement condition evaluations, the road images are collected automatically from
a fast-moving vehicle, but distress identification is postponed to an off-line process
running in workstations at the office. This approach improves safety but still uses
manual distress identification, which is very time-consuming.
Meanwhile, mobile devices such as smartphones have evolved in recent years into
devices containing high resolution digital cameras, sensors, and powerful proces-
sors therefore, examples of road inspection using smartphones are becoming more
common. The advantage of employing smartphones is that they enable efficient and

2
 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

cost-effective inspection of the road surface of large road networks.

Deep learning is an arising field of Machine learning which is still in its research.
Machine learning concentrates on tasks such as recognising the images, converting
speech to text, recognition of speech and visual object, drug discovery, face detec-
tion and recognition, weather forecast etc. Deep learning techniques are used more
in these applications where adaptive learning is done. Deep learning can do cog-
nitive learning such as learning the features, characteristics and attributes with the
help of good algorithms which can learn by itself and deep architecture. The family
of deep learning has been increasing which includes neural networks, various un-
supervised and supervised learning algorithms for recognising feature such as Deep
Convolutional Neural Network and Recurrent Neural Network It has the capacity
to process complex data given as input and results in giving the predicted output
through effective recognition.
Object detection is a phenomenon in computer vision that involves the detection
of various objects in digital images or videos. Some of the objects detected include
people, cars, chairs, stones, buildings, and animals.
Object detection consists of various approaches such as fast R-CNN, Retina-Net,
and Single-Shot MultiBox Detector (SSD). Although these approaches have solved
the challenges of data limitation and modeling in object detection, they are not able
to detect objects in a single algorithm run. YOLO algorithm has gained popularity
because of its superior performance over the mentioned object detection techniques.
YOLO is an abbreviation for the term ‘You Only Look Once’. This is an algo-
rithm that detects and recognizes various objects in a picture (in real-time). Object
detection in YOLO is done as a regression problem and provides the class probabil-
ities of the detected images.
YOLO algorithm employs convolutional neural networks (CNN) to detect objects
in real-time. As the name suggests, the algorithm requires only a single forward
propagation through a neural network to detect objects. This means that prediction
in the entire image is done in a single algorithm run. The CNN is used to predict
various class probabilities and bounding boxes simultaneously. The YOLO algo-
rithm consists of various variants. Some of the common ones include tiny YOLO
and YOLOv3.
YOLO algorithm is important because of the following reasons:
Speed: This algorithm improves the speed of detection because it can predict
objects in real-time.
High accuracy: YOLO is a predictive technique that provides accurate results

AVCOE, Sangamner 3 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

with minimal background errors.

Learning capabilities: The algorithm has excellent learning capabilities that en-
able it to learn the representations of objects and apply them in object detection.
YOLO algorithm works using the following three techniques:

• Residual blocks

• Bounding box regression

• Intersection Over Union (IOU)

First, the image is divided into grid cells. Each grid cell forecasts B bounding
boxes and provides their confidence scores. The cells predict the class probabilities
to establish the class of each object.
Intersection over union ensures that the predicted bounding boxes are equal to the
real boxes of the objects. This phenomenon eliminates unnecessary bounding boxes
that do not meet the characteristics of the objects (like height and width). The final
detection will consist of unique bounding boxes that fit the objects perfectly.
Surveying problem can be solve by automated method. There is a one desktop
application to detect the data and distribute those through API. In desktop first login
through credentials and enter the information about the road of Survey done. after
starting the system , after some interval the machine will start clicking images of
particular road and those taken images passed through machine Learning module.
Machine will detect the specific type of damages and also it’s location as well as
Machines shows the prediction Probability and this survey part is stored in Database.
If any Organization are interested in to get this data they can subscribe the API to
access those Data.So,this is how overall system works.

1.2 Objectives
• To create a model which could be capable to detect and classify the road dam-
ages like cracks, potholes, etc.

• To create an automated system for road surveying and monitoring surface con-
ditions.

• To reduce the computation time.

• To use the data collected by automated system for analysis and visualization
using geotagging.

AVCOE, Sangamner 4 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

1.3 Motivation of the Project

• Due to huge road networks road condition monitoring is a challenging problem,
which require an efficient and automated system which will eliminate human
dependency of the process.

• Challenging problem of road condition monitoring could be automated.

• Data collected by the system could be used for efficient road maintenance, will
improve road quality and decrease risk of road accidents.

1.4 Organization of Report

Report consists of twelve chapters which gives step by step implementation of sys-
tem “Driver Drowsiness and Alcohol Detection for Safe Driving”.

• Chapter 1:
The chapter covers basic introduction about the system. It also states the ob-
jectives and motivation for the system.

• Chapter 2:
The chapter explain literature survey. It includes literature review through tech-
nologies which earlier researchers have stated.

• Chapter 3:
The limitations of system are observed in this chapter. Studying these, problem
statement is given and mathematical model stated.

• Chapter 4:
The chapter contains the requirement analysis for the system such as software
needed and hardware needed for the project.

• Chapter 5:
The chapter emphasis on design of system. The system architecture, DFD dia-
grams and UML diagrams are discussed in the chapter.

• Chapter 6:
The chapter covers the technical specification with tools used and software’s
used in system.

• Chapter 7:
The project estimate and team structure is planned in this chapter.

AVCOE, Sangamner 5 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

• Chapter 8:
The chapter discusses about implementation and methodology about the sys-
tem.

• Chapter 9 :
The chapter discusses Testing strategy, Test Cases and Risk analysis.

• Chapter 10 :
The Chapter gives the advantages and applications of system.

• Chapter 11 :
The chapter shows the implemented system with screenshots and analysis.

• Chapter 12 :
The chapter contains Conclusion and Future scope. Some future improvements
that can be made in the project are explained in chapter.

AVCOE, Sangamner 6 Department of Computer Engineering

 Chapter 2

Literature Survey

• TITLE : PAVEMENT DISTRESS DETECTION AND CLASSIFICATION

BASED ON YOLO NETWORK.
Year of Publication : 2020
Authors : Du, Y., Pan, N., Xu, Z., Deng, F., Shen, Y., and Kang, H.
Paper Review :

Utilize YOLO network for detecting and classifying pavement distresses.

However, the images are collected using an industrial high-resolution camera,
which gives more accuracy.

• TITLE : OBJECT DETECTION USING CONVOLUTIONAL NEURAL

NETWORKS.
Year of Publication : 2018
Authors : Reagan L. Galvez; Argel A. Bandala; Elmer P. Dadios; Ryan Rhay
P. Vicerra; Jose Martin Z. Maningo.
Paper Review :

In this paper, Convolutional Neural Networks (CNN) is used to detectob-

jects in the environment. Two state of the art models are compared for object
de- tection, Single Shot Multi-Box Detector (SSD) with Mobile Net V1 and
a Faster Region-based Convolutional Neural Network (Faster-RCNN) with In-
ceptionV2. Result shows that one model is ideal for real-time application be-
cause of speed and the other can be used for more accurate objectdetection.

7
 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

• TITLE :AUTOMATIC PAVEMENT CRACK DETECTION BASED ON

STRUCTED PREDICTION WITH CONVOLUTIONAL NEURAL NET-
WORK.
Year of Publication : 2018
Authors : Fan, Z., Wu, Y., Lu, J., and Li, W. Paper Review :

Supervised, convolutional neural network (CNN) was trained to recognize

different pavement conditions, and results presented in the paper show that it
can deal well with varying pavement textures.

• TITLE : TITLE : ROAD DAMAGE DETECTION AND CLASSIFICA-

TION USING DEEP NEURAL NEWORK WITH SMARTPHONE IM-
AGES.
Year of Publication : 2018
Authors : Maeda, H., Sekimoto, Y., Seto, T., Kashiyama, T., and Omata,H.
Paper Review :

The detection method developed classified pavement deterioration of Japan

road network into eight categories based on images captured by mobile devices
with a resolution of 300 x 300 pixels.

• TITLE : ROAD CRACK DETECTION USING DEEP CONVOLUTIONAL

NETWORK.
Year of Publication : 2017
Authors : Zhang, L., Yang, F., Zhang, Y. D., and Zhu, Y. J.
Paper Review :

Classify pavement images taken by smart phones around the Temple Univer-
sity campus. However, proposed road damage detection methods focused only
on the determination of the existence of damage.

AVCOE, Sangamner 8 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

• TITLE : DEEP NEURAL NETWORKS FOR OBJECT DETECTION.

Year of Publication : 2016
Authors : Christian Szegedy, Alexander Toshev, Dumitru Erhan.
Paper Review :
Deep Neural Networks (DNNs) have recently shown outstanding performance
on image classification tasks. This system goes one step further and address
the problem of object detection using Deep Neural Network that is not only
classifying but also precisely localizing objects of various classes. They have
presented a simple and yet powerful formulation of object detection as a re-
gression problem to object bounding box masks. They defined a multi-scale
inference procedure which is able to produce high-resolution object detections
at a low cost by a few network applications.

AVCOE, Sangamner 9 Department of Computer Engineering

 Chapter 3

PROBLEM DEFINITION AND SCOPE

3.1 Limitation of existing system
• Road authorities use manual surveying methods to monitor road conditions, but
due to huge road network surveying become challenging.

• Due to human dependency of process surveying is costly and time consuming

process, affecting road surface quality and maintenance.

• Developed countries like USA and UK use automated road surface monitor-
ing which involves Li-DAR(Light Detection and Ranging) scanning, ground
penetration radar (GPR) and combination of multiple sensors costing between
80000to22000.

3.2 Problem Statement

• Road surface condition monitoring is a worldwide problem, it is necessity to
monitor surface conditions of the road because road networks plays an impor-
tant role in transportation. Services like health are affected by road conditions
also it affects economic growth, so it is important to monitor road conditions to
maintain roads.

• Currently Developing nations like India use conventional method of surveying,

which is very slow and not efficient, to make this process fast and efficient and
automated system is required.

• Computer Vision is a field of data science which process video/images to ex-

tract useful information, A CNN model could be trained to classify and de-
tect the road surface damages by processing a video stream input from a cam-
era sensor, using a GPS sensor we could aassociate the model output with a
location/co-ordinate. Data collected by automated system can be use to moni-
tor road surface conditions.
3.2.1 Goals and Objectives
Goal and Objectives:

• To create a model which could be capable to detect and classify the road dam-
ages like cracks, potholes, etc.

10
 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

• To create an automated system for road surveying and monitoring surface con-
ditions.

• To study and analyze existing Object Detection Techniques.

• To implement Yolo v5 model for object detection.

• To perform analysis on the Result generated over a period of Time.

3.3 Mathematical Model

• Let

• S be Closed system defined as, S = Ip, Op, Ss, Su, Fi, A

• To select the input from the system and perform various actions from the set of
actions A so that Su state can be attained

• S=Ip,Op,Ss,Su,Fi,A – Where,

• IP1= Image

• Set of actions=A

• A = F1,F2,F3,F4 – Where

• F1= Image Capture

• F2= Pre-processing

• F3= Segmentation

• F4= Road Surface Detection

• S = Set of users

• Ss= rest state, capturing image, processing image, detection of damage, type of
damage

• Su - success state is successful analysis

• Fi - failure state

• Objects:

• 1) Input1: Ip1 = Camera Input

• 2) Input2 : Ip2= GPS Sensor

AVCOE, Sangamner 11 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

• 1) Output1 : Op1 = Road Surface Detection

• 2) Output2 : Op2 = Road Surface Classification

• 3) Output3 : Op3 =GPS Location With Latitude and Longitude .

• 4) Output4 : Op4 = Probality Value.

AVCOE, Sangamner 12 Department of Computer Engineering

 Chapter 4

Software Requirements Specification

4.1 Introduction
This document will provide a general description of our project, including user re-
quirements, product perspective, and overview of requirements, general constraints.
In addition, it will also provide the specific requirements and functionality needed
for this project – such as interface, functional requirements and performance require-
ments.
4.1.1 Purpose
We will implement project which will be the high speed and low cost road surface
damage detection and classifications while doing road monitoring by the government
authorities.
4.1.2 Assumptions and Dependencies
While deployment this is assumed that the system has a full set of information
related to the road surface damage detection and classification which will give the
best yield to the system and this all information will be shared with the user with the
help of API . user want subscribe this API and user will have access to it.
While testing some assumptions and dependencies are considered. Because it is
said that ’test’ challenges the assumptions, risks and uncertainty inherent the work
of the other disciplines, addressing those concerns by concrete demonstration and
impartial evaluation. .

4.2 External Interface Requirements

App Interface provides timely information to users.
4.2.1 User Interfaces
The user interface or UI for the software should be compatible to be used by win-
dows operating system. Using this UI user can have ac cess to the system. The UI
or user interface can be developed by using Tkinter.
4.2.2 Hardware Interfaces
A hardware interface is needed to run the software. Python, Keras, Numpy compat-
ible hardware is required which is minimal requirement.

13
 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

4.2.3 Software Interfaces

• It uses Tkinter as the front end programming tool. MySQL has been used as
back end application tool. Latest version of python anything higher than 3 can
be used.

4.3 Nonfunctional Requirements

4.3.1 Performance Requirement
• The accuracy of the system depends on the dataset, to get outputs with good
accuracy large dataset with correct labelled data is required.

• The system depends on the hardware capability for speed, a system with good
hardware capability gives more computation power and could generate outputs
in real-time.

• To obtain accurate results we need large dataset for frame comparison.

4.3.2 Safety Requirements

• The System classifies the waste into 6 categories, so its dataset given should
be large in size for comparison. If we don’t have large dataset then there are
chances of inaccurate results.

4.3.3 Software Quality Attributes

• Efficiency: Model Used in the system is said to give best Efficiency Compared
the existing models available in the market. YOLO guaranties more Speed and
Accuracy.

• Reliability: System is reliable enough to sustain in any condition. System

responsibleness is measured in terms of working of project beneath completely
different operating surroundings and different conditions.

• Maintainability: Maintenance of the system is efficient and straight forward.

System is simple to keep up and correcting defects. Its is also able to adopt the
new modifications.

• Extensibility: Proposed system can be extended to next level. It is able to

adopt new modifications.

AVCOE, Sangamner 14 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

4.4 Other Requirements

4.4.1 Hardware Requirements

Table 4.1: Hardware List

Sr. No. Parameter Minimum Requirement Justification

1 CPU 1.2GHz dual-core or higher Required
2 GPU NVIDIA GeForce GTX 940 Required
or higher
3 RAM 8 GB minimum, 16 GB or Required
higher is recommended
4 Camera and GPS 4000 * 3000 pixels, 30fps or Required
higher specifications

4.4.2 Software Requirements

• Operating System

– Windows
– Linux

• IDE

– Visual Studio
– PyCharm.

• Programming Language

– python
– JacvaScript

• Database

– MySql
– Mongo-DB

• Dependencies

– Google Colab
– Jupyter Notebook
– Anaconda

AVCOE, Sangamner 15 Department of Computer Engineering

 Chapter 5

Methodology AND System Design

5.1 System Architecture
• The System is divided in to different modules, main component of the system
is the CNN Model, CNN model uses the dataset to train and testing data for
validation CNN module takes the input from the camera and sensor which give
a stream of frames as input to the CNN Model. Model will process the frame
and filter the results based on the threshold values.

• System requires two basic sensors Camera and GPS; Camera give the video
stream output and GPS sensor give latitude and longitude information.

• API is the web service module of the system which makes the data available for
the different use cases, it is simple server which communicate with the database
for the data.

Figure 5.1: Block Diagram

16
 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 5.2: System Architecture

AVCOE, Sangamner 17 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

5.2 Data Flow Diagram

A data flow diagram (DFD) is a graphical representation of the “flow” of data
through an information system, model in gits process aspects. A DFD is often
used as a preliminary step to create an overview of the system,which can later be
elaborated. DFDs can also be used for the visualization of data processing. Figure
5.4showing the system as a single process with its relationship to external entities. It
represents the entire system as a single bubble with input and output data indicated
by incoming/outgoing arrows.
5.2.1 Level 0

Figure 5.3: Data Flow Diagram (Level 0)

5.2.2 Level 1

Figure 5.4: Data Flow Diagram (Level 1)

AVCOE, Sangamner 18 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

5.2.3 Level 2

Figure 5.5: Data Flow Diagram (Level 2 )

5.3 UML Diagram

5.3.1 Use Case Diagram

Use case diagrams are typically used to model circumstances where your system or
application interfaces with outside entities, which are known as actors. Use case di-
agrams are usually referred to as behavior diagrams used to describe a set of actions
(use cases) that some system or systems (subject) should or can perform in collabo-
ration with one or more external users of the system (actors). Each use case should
provide some observable and valuable result to the actors or other stakeholders of
the system. Below diagram shows the functionality and services provided by our
system. It shows the interaction of user and admin with the system. It represents
number of use cases which are involved in the process. Above use cases are con-
nected to each other using ’include’ relation which shows that they are dependent
on each other. We can observe that it includes use cases like training and testing
the data, read Camera and GPS sensors, process video stream for classification and
detection these functions belong to the admin. Collecting the data, Geo-tagging,
analysis and visualization of data are functions belong to the user.
Figure 5.6 shows the functionality and services provided by our system. It shows
the interaction of user with the system. It represents number of use cases which are
involved in the process. Above use cases are connected to each other using ’include’
relation which shows that they are dependent on each other.

AVCOE, Sangamner 19 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 5.6: Use Case Diagram

5.3.2 Activity Diagram

Activity diagram is another important diagram in UML to describe the dynamic
aspects of the system. Activity diagram is basically a flowchart to represent the flow
from one activity to another activity. The activity can be described as an operation of
the system. The control flow is drawn from one operation to another. This flow can
be sequential, branched, or concurrent. Activity diagrams deal with all type of flow
control by using different elements such as fork, join, etc. The activities are carried
out by capturing the image to detect the road surface damages. then classification of
road surface damages and GPS locations of that damages are used for the analysis
of the road data.

AVCOE, Sangamner 20 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 5.7: Activity Diagram

Figure 5.7 shows software processes as a progression of actions. These actions

can be carried out by people, software components or system. Activity diagrams
are used to describe business processes and use cases as well as to document the
implementation of system processes.

AVCOE, Sangamner 21 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

5.3.3 Sequence Diagram

Sequence diagrams provide a graphical representation of object interactions over
time. These typically show a user or actor, and the objects and components they
interact with in the execution of a use case. One sequence diagram typically rep-
resents a single Use Case ’scenario’ or own of events. Sequence diagrams are an
excellent way of documenting usage scenarios and both capturing required objects
early in analysis and verifying object use later in design. The diagrams show the
own of messages from one object to another, and as such correspond to the methods
and events supported by a class/object. The sequence show the sequence how the
activities are carried out in system. In this it begins with capturing the image and
then classifying the road surface damages and GPS location of that damage for the
monitoring of road surface. This data is also used for the analysis of the road data.

Figure 5.8: Sequence Diagram

AVCOE, Sangamner 22 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

5.3.4 Class Diagram

Class diagram is not only used for visualizing, describing, and documenting differ-
ent aspects of a system but also for constructing executable code of the software
application. In software engineering, a class diagram in the Unified Modeling Lan-
guage (UML) is a type of static structure diagram that describes the structure of a
system by showing the systems classes, their attributes, operations (or methods), and
the relationships among objects. It shows the various classes and their relationship
with each other. There are different classes like GPS, Camera which represents the
sensors, CNN model represents the classification and detection model, Survey tool
is a class which uses the camera, GPS and CNN model together to collect data, API
is a web service to get data from database.

Figure 5.9: Class Diagram

AVCOE, Sangamner 23 Department of Computer Engineering

 Chapter 6

Technical Specification
6.1 Technology details
6.1.1 python
Desktop application for collection of data is developed in python . Python is an
interpreted, object-oriented, high-level programming language, supports modules
and packages, which encourages program modularity and code reuse. User interface
of the application is developed using Tkinter and kivy libraries. For dealing with the
images and image processing OpendCV library is used. The Yolo model loading,
processing is supported by python library Tensorflow.

Figure 6.1: Python

24
 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

6.1.2 JavaScript
The REST-API is implemented using javascript based Node.js and Express frame-
work. Express is a framework for scripting web servers. Communication with
database for fetching data and serving the data based on the API call is managed
by the server supported by Node.js. Javascript based library JQuery is being used to
design the user interface for system. The web application for presentation of data is
built using Angular which communicate with the API is based on Javascript.

Figure 6.2: JavaScript

6.1.3 MySQL
MySQL is relatioinal database, prvide data defination and data manipulation meth-
ods. The project require to store Admin information, Surveyor information, Survey
Data etc, MySQL store data using tables. SQL is data manipulation language for
relational MySQL database, CRUD operations are supported by SQL. Project use
the views and trigger suppored by MySQL for better data manipulation.

AVCOE, Sangamner 25 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 6.3: SQL Database

6.1.4 REST
REST stands for Representational State Transfer. It relies on stateless client server.
REST (Representational State Transfer) is an architectural style, and an approach to
communications that is often used in the development of Web services. The use of
REST is often preferred over the more heavyweight SOAP (Simple Object Access
Protocol) style because REST does not leverage as much bandwidth, which makes
it a better fit for use over the Internet. The SOAP approach requires writing or using
a provided server program (to serve data) and a client program (to request data).

Figure 6.4: REST API

AVCOE, Sangamner 26 Department of Computer Engineering

 Chapter 7

PROJECT ESTIMATE, SCHEDULE AND

TEAM STRUCTURE
7.1 Project Cost
7.1.1 COCOMO Model
Bohem’s [Boehm 8] COCOMO Model is one of the mostly used models commer-
cially. The first version of the model delivered in 1981 and COCOMO II is available
now. COCOMO’81 is delivered from the analysis of 63 software projects in 1981.
Boehm proposed three levels of the model: Basic, intermediate, detailed.

• The basic COCOMO’81 model is a singled-valued,static model that computes

software development effort(and cost)as a function of program size expressed
in estimated lines of code(LOC).

• The intermediate COCOMO’81 model computes software devel- opment ef-

fort as a function of program size and a set of ”cost drivers” that include sub-
jective assessments of product,hardware,personnel,and project attributes.

• The detailed COCOMO’81 model incorporates all characteristicsof the inter-

mediate version with an assessment of the cost drivers impact on each step(analysis,design,etc
the software engineering process.

• COCOMO’81 model depends on the two main equations:

First is development effort(based on MM - man-month/Person- month/staff-
month is one month of effort by one person). In CO- COMO’81, there are
152 hours per Person-Month.According to orga- nization this values may differ
from standard by 10.
MM=aKDSIb
Second is effort and development time(TDEV)
TDEV=cMMd
KDSI means the number of thousand delivered source in- structions and it is
a measure of size. The coefficients a,b,c and d are depend on the mode of
development. There are three modes of development.

27
 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Table 7.1: Modes of development

Development mode Innovation Deadline Dev. Environment

Organic little not light Stable
Semi-detached Medium Medium Medium
Embedded Large Greater Tight Complex hardware

Here are the coefficients related to development modes for intermediate model.

Development mode a b c d
Organic 3.2 1.05 2.5 0.38
Semi-detached 3.0 1.12 2.5 0.35
Embedded 2.8 1.20 2.5 0.32
In intermediate mode development effort equation becomes:
MM=aKDSIbC
C is effort estimation factor, is calculated simply multiplying the values of cost
drivers, So the intermediate model is more accurate than the basic model.

• The steps in producing an estimate using the intermediate model CO-

COMO’81 are :

– Identify the mode(organic,semi-detached,and embedded)of de- velopment

for the new prodect.
– Estimate the size of the product in KDSI to derive a norminat effort pre-
diction.
– Adjust 15 cost drivers to reflect your project.
– Calculate the predicted project effort using first equation and the effort
adjustment factor(C).
– Calculate the project duration using second equation.
The model followed is the construction cost model( COCOMO)for esti-
mating the effort required in completing the project. Like all the estimation
models, the COCOMO model requires sizing in- formation.
This information can be specified in the form of :
1. Object Point
2. Funtion Point(FP)
3. Line of Source Code

AVCOE, Sangamner 28 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

For our project, we use the sizing information in the form of Line of Source
Code.
1. Total Line of Code for our project,KLOC= 6k(approx).
2. Cost of each person per month,Cp = Rs. 7,415 /- (per person month)

7.1.2 Equations:
E= a*(KLOC)b
where,
a = 3.2,b = 1.05, for an organic project.
E = Efforts in person month D = a*(E)b
7.1.3 Organic project:
Project of moderate size and complexity, where teams with teams with mixed ex-
perience levels must meet a mixed rigid and less than rigid requirements(project
modway between embedded and oraganic types).
Number of People:

Equation for calculation of number of people required for comple- tion of project,
using the COCOMO model is: N=E/D
where,
N = Number of people required
E = Efforts in person-month
D = Duration of project in months
Cost of Project:

Equation for calculation of cost of project, using COCOMO model is:

C=D*Cp
where,
C = Cost of project
D = Duration in months
Cp = Cost incurred per person-month
Calculation

Efforts:
E= 3.2*(6)1.05
E = 24.62 person-months
Total of 24.62 person-months are required to complete the project successfully.

AVCOE, Sangamner 29 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Duration of Project:

D= 2.5*(E)0.32
D = 6 months
The approximate duration of project is 6 months.
Number of people required for the project:

N=20.99/6
N=3.5
N=4 people
Therefore 4 people are required to successfully complete the project on schedule.
Cost of Project:

C=4.00*5000 = 20,000 /-
Therefore, the cost of project is 20,000/- (approx)
7.1.4 Analysis Models: SDLC Model to be applied
The Waterfall Model was first Process Model to be introduced. It is also referred
to as a linear sequential life cycle model. It is very simple to understand and use.
In a waterfall model, each phase must be completed fully before the next phase can
begin. This type of model is basically used for the for the project which is small
and there are no uncertain requirements. At the end of each phase, a review takes
place to determine if the project is on the right path and whether or not to continue
or discard the project. In this model the testing starts only after the development is
complete. In waterfall model phases do not overlap.

Applications:

1. This model is simple and easy to understand and use.

2. It is easy to manage due to the rigidity of the model - each phase has specific
deliverables and a review process.

3. In this model phases are processed and completed one at a time. Phases do not
overlap.

4. Waterfall model works well for smaller projects where requirements are very
well understood.

AVCOE, Sangamner 30 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 7.1: Software Development Life Cycle

• Requirements:
All possible requirements of the system to be developed are captured in this
phase and documented in a requirement specification document.

• Specification and Design:

The requirement specifications from first phase are studied in this phase and the
system design is prepared. This system design helps in specifying hardware and
system requirements and helps in defining the overall system architecture.

• Implementation:
With inputs from the system, the system is first developed in small programs
called units, which are integrated in the next phase. Each unit is developed and
tested for its functionality, which is referred to as Unit Testing.

• Testing:
All the units developed in the implementation phase are integrated into a system
after testing of each unit. Post integration the entire system is tested for any
faults and failures.

• Deployment:

AVCOE, Sangamner 31 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Once the functional and non-functional testing is done; the product is deployed
in the customer environment or released into the market.

• Maintainance:
There are some issues which come up in the client environment. To fix those
issues, patches are released. Also to enhance the product some better versions
are released. Maintenance is done to deliver these changes in the customer
environment.
In ”The Waterfall” approach, the whole process of software development is
divided into separate phases. In this Waterfall model, typically, the outcome of
one phase acts as the input for the next phase sequentially.

7.2 Project plan and Team Structure

Figure 7.2: Project Work Breakdown Structure

AVCOE, Sangamner 32 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Table 7.2: List of Tasks

Task No. Task Title

T1 Topic Finalization

T2 Requirement specification

T3 Technology Familiarization

T4 System Set up

T5 Concept Review Study

T6 Study of technologies used in the project

T7 Design of user interface

T8 Designing of Constraints Rules

T9 Creation of Connectivity for Services

T10 Creation of database files and rules

T11 Designing the Architectural layout

T12 Creating module using database and rules

T13 Integration of T11 to T7 Tasks

T14 Testing

T15 Documentation Preparation

T16 Maintenance

AVCOE, Sangamner 33 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Table 7.3: Task Organization

Task No. No. of Days Developer

T1 7 D1,D2,D3,D4

T2 4 D1,D2,D3,D4

T3 4 D1,D2,D3,D4

T4 2 D1,D2,D3,D4

T5 4 D1,D2,D3,D4

T6 7 D1,D2,D3

T7 5 D2,D3

T8 8 D1,D4

T9 5 D1,D2,D3,D4

T10 5 D1,D2,D3,D4

T11 7 D1,D2,D3,D4

T12 5 D1,D2,D3

T13 10 D1,D2,D3,D4

T14 7 D1,D2,D3

T15 6 D1,D2,D3

T16 4 D1,D2,D3,D4

Table 7.4: List of Developers

Developer ID Developer

D1 Mr. Sameer Dnyanoba Arote

D2 Mr. Ganesh Manish Gopale

D3 Mr. Samir Ramdas Aher

D4 Mr. Yogeshwar Sambhaji Ghule

AVCOE, Sangamner 34 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

7.3 System Implementation Plan

7.3.1 Systematic plan

Table 7.5: Implementation Plan

Task No. Task to be Accomplished

T1 Topic Finalization
T2 Requirement specification
T3 Technology Familiarization
T4 System Set up
T5 Concept Review Study
T6 Study of technologies used in the project
T7 Design of user interface
T8 Designing of Constraints Rules
T9 Creation of Connectivity for Services
T10 Creation of database files and rules
T11 Designing the Architectural layout
T12 Creating module using database and rules
T13 Integration of T11 to T7 Tasks
T14 Testing
T15 Documentation Preparation
T16 Maintenance

7.3.2 Timeline Chart

Figure 7.3: Time Line Chart

AVCOE, Sangamner 35 Department of Computer Engineering

 Chapter 8

Software Implementation
8.1 Dataset Description
The RDD2020 dataset contains 26336 road images collected from India, Japan, and
the Czech Republic with more than 31000 instances of road damage. The dataset
contains annotation for four damage categories: Longitudinal Cracks(D00), Trans-
verse Cracks(D10), Alligator Cracks(D20), and Potholes(D40); and is intended for
developing deep learning-based methods to detect and classify road damage auto-
matically. The images in RDD2020 were captured using vehicle-mounted smart-
phones, making it useful for municipalities and road agencies to develop methods
for low-cost monitoring of road pavement surface conditions.
Classes and Label Count
D00 : 6592
D01 : 179
D10 : 4446
D11 : 45
D20 : 8381
D40 : 5627
D43 : 793
D44 : 5057
Total Labels : 31120

Figure 8.1: Damage Types

36
 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 8.2: Dataset Line Chart

8.2 Project modules

8.2.1 CNN Model
YOLO model is a object detection and classification model, which is based on CNN
Architecture. Model is trained on RDD2020 dataset, capable to classify and de-
tect Longitudinal Cracks(D00), Transverse Cracks(D10), Alligator Cracks(D20),
and Potholes(D40) damage classes. Model takes image as input and output is the
classification class, detection area(predicted bounding box) and the probability of
prediction.
Requirements:

• Google Colab

• Yolo V4

• Tensorflow

Functions

• Detection and classification of road surface damage.

AVCOE, Sangamner 37 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

8.2.2 Data Collection

Data Collection module is required to automate surveying process, the surveying in-
volves monitoring road surface conditions. Module is responsible for reading sensor
data for input, using camera and GPS sensor. Camera is responsible to capture road
surface images with at least 720X720 pixels, GPS sensor is to Geo-tag the data with
longitude and latitude. Input data is processed by the YOLO Model to output data,
detected damage class, bounding box, prediction probability. Output is Geo-tagged
and analyzed to estimate results.
Requirements

• Python

• OpenCV

• Tensorflow

Function

• Create desktop application.

• Read sensor data for input.

• Process input data for output.

• Save results in database.

8.2.3 Data Distribution

Data distribution module is responsible for creating a data distribution platform. It
is implemented by the REST-API. The API Server is responsible for accepting the
data requirement, processing, communication with the database. The API routes are
implemented to serve data based the API calls.
Requirements

• Node.js

• Express

• MySQL

• Javascript

AVCOE, Sangamner 38 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Function

• Authentication.

• Create server for API.

• Fetch data from database.

8.3 Algorithms used

8.3.1 YOLO V4 Model
YOLO is an acronym that stands for You Only Look Once. It is now the most ad-
vanced object identification algorithm available. It is a novel convolutional neural
network (CNN) that detects objects in real-time with great accuracy. This approach
uses a single neural network to process the entire picture, then separates it into parts
and predicts bounding boxes and probabilities for each component. These bounding
boxes are weighted by the expected probability. The method “just looks once” at the
image in the sense that it makes predictions after only one forward propagation run
through the neural network. It then delivers detected items after non-max suppres-
sion (which ensures that the object detection algorithm only identifies each object
once).

Figure 8.3: Yolo Model 1

AVCOE, Sangamner 39 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 8.4: Yolo Model 2

Figure 8.5: Yolo V5 Architecture

AVCOE, Sangamner 40 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

8.3.2 Steps for Model Creation

• Cloning repository and Installing dependencies for Yolo V4:

– the repository and install requirements.txt in a Python=3.6.0 environment,

including PyTorch=1.7. Models and datasets are downloaded automati-
cally from the latest YOLO v4 release.

• Create annotations for a custom dataset Using the VIA tool YOLO V4
model:
YOLO V4 modelmust be trained on labelled data in order to learn classes of
objects in that data. There are two options for creating your dataset before you
start training:

– Use Roboflow to label, prepare, and host your custom data automatically
in YOLO format.
– Manually prepare your dataset .

• Creating YAML file for training: .

Once dependencies are installed and data is ready now we need to configure a
YAML file to train the model. For that, you open any text editor and create a
file “coco.yml” inside the folder “YoloV4/data/” number of classes no: 4 class
names: [‘D00’,‘D10’,‘D20’,’D40’]

• YoloV4 training: .
Setup of the YAML file leaves nothing much to do afterwards, now we just
need to write the command and provide it with the data and weights to train
the model on our custom dataset. You need to traverse to the YoloV5 folder
through terminal or anaconda prompt.

AVCOE, Sangamner 41 Department of Computer Engineering

 Chapter 9

Software Testing And Validation

9.1 Type of Testing done
Testing is a vital part of software development, and it is important to start it as
early as possible, and to make testing a part of the process of deciding requirements.
Testing is part of a life cycle. The software development life cycle is one in which we
hear of a need, we write some code to fulfill it, and then we check to see whether you
we pleased the stake holders the users, owners, and other people who have an interest
in what the software does. We therefore design tests based on the stakeholders needs,
and run the tests before the product reaches the users. Preferably well before then,
so as not to waste our time working on something that isn’t going to do the job

• Tests represent requirements. Whether you write user stories on sticky notes
on the wall, or use cases in a big thick document, your tests should be derived
from and linked to those requirements. And as we’ve said, devising tests is a
good vehicle for discussing the requirements.

• Were not done till the tests pass. The only useful measure of completion is
when tests have been performed successfully.

Test Objective: The objective of our test plan is to find and report as many bugs as
possible to improve the integrity of our program. Although exhaustive testing is not
possible, we will exercise a broad range of tests to achieve our goal. Following test
process approach will be followed :

• Organize Project involves creating a System Test Plan, Schedule and Test Ap-
proach, and assigning responsibilities.

• Design/Build System Test involves identifying Test Cycles, Test Cases, En-
trance and Exit Criteria, Expected Results, etc. In general, test conditions/-
expected results will be identified by the Test Team in conjunction with the
Development Team. The Test Team will then identify Test Cases and the Data
required. The Test conditions are derived from the Program Specifications Doc-
ument

• Design/Build Test Procedures includes setting up procedures such as Error

Management systems and Status reporting.

42
 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

• Build Test Environment includes requesting/building hardware, software and

data set-ups.

• Execute System Tests The tests identified in the Design/Build Test Procedures
will be executed. All results will be documented and Bug Report Forms filled
out and given to the Development Team as necessary.

• Sign-off - Sign-off happens when all pre-defined exit criteria have been achieved.

UNIT TESTING: During the unit testing phase, the system is tested while it is
developed. Here all the options of the system are validated. During the first phase
of this testing the testing person tests the system by entering the valid data, or by
performing the appropriate function which the system requests for. This phase of
testing is done to verify whether the system performs all the requested functions.
WHITE BOX TEXTING: White box testing, sometimes called as glass box
testing, is a test case design method that uses control structure of the procedural
design to derive the test cases. Using the white box testing methods, the software
engineer can derive the test case that:

• Guarantee that all independent paths within module having exercised at least
once.

• Exercise all logical decisions and their true or false sides.

• Execute all loops and within their operational bounds.

• Exercise the internal data structures to ensure their validity.

BLACK BOX TESTING: We have focused on the Functional requirement and the
proceeding of the module to fulfil the requirement the requirement will be appreci-
ated by the module instances and give the desired functionality as per the expecta-
tions. Black box testing attempts to find the errors in following categories:

• Incorrect user details

• Interface problems

• Initialization Complexities

• Performance Measurement

• Error in data structures and external database error

AVCOE, Sangamner 43 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

INTEGRATION TESTING: The integration will be based on the total whole

module testing the integration of the module will be categories as follows:

• Checking the module and Data entry status

• Re-directional functionalities

• Checking the Status through ID

• Checking Chart Accuracy

• Error in data structures and external database error

Top Down Integration:

Top down integration is an incremental approach to the construction of program
structure modules and integrated by moving downwards through the control hierar-
chy beginning with main control module. Modules subordinates to the main control
module are incorporated into the structure in either the depth first or breadth first
manner as per the categories:

• Sending data to the database

• Constraint validation

• Updating the further changes in the system

• Looking for the Status

• Perform the Graphical Accuracy Test

Acceptance testing:
We performed acceptance testing to Verify whether the whole system works as
intended.
Functional testing:
Functional testing is the checking functions by emulating business scenarios,
based on functional requirements. For that we performed Blackbox testing
which is a common way to verify functions.

• Interface Problems

• Performance measurement

• External Database Error

AVCOE, Sangamner 44 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

9.2 Test cases

Table 9.1: Test Cases

No. Test Cases Test Data Expected Result Actual Result Test
Re-
sult
1. Surveyor Login Surveyor Should System should Credentials vali- Pass
be able to login validate the dated, Login suc-
with Credentials credentials and cessfully
allow access
2. Sensor Stream Mobile sensor Using the stream Sensor data read Pass
Read data should be address sensor successfully for
read data of mobile both Camera and
device should be GPS
read
3. Image Capture Image of instance From Camera Image(frame) Pass
must be captured stream frame of successfully
instance must be extracted
captured
4. Image Processing Image dimen- Image should Image dimen- Pass
sions must be be convert to sions changed
changed 408X408 size successfully
5. Gps Data (Lati- System should GPS co-ordinates GPS data read Pass
tude Longitude) read latitude and latitude and lon- successfully
longitude data gitude must be
form GPS read
6. Feature Ex- Yolo model All classes con- IModel detects Pass
traction and should detect sidered must be and classify con-
Classification and classify road detected and clas- sidered classes
damages sified by model with decent accu-
racy
7. Database Manip- Results of sur- Survey result Database manip- pass
ulation vey should be up- must be updated ulation work cor-
dated. and fetched rectly
8. REST-API server Server must fetch Server must Server able to Pass
and serve the data communicate fetch and update
to database and data based on
fetch data. requests
9. API endpoint API endpoints API endpoints API endpoints Pass
must fetch data should serve the able to fetch data
based on require- data for every successfully
ment request.

AVCOE, Sangamner 45 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Positive Test Cases

Figure 9.1: Login Successfully

Figure 9.2: Camera And GPS Connection Successful

AVCOE, Sangamner 46 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 9.3: Detection Successfully

Negative Test Cases

Figure 9.4: Login Failed

AVCOE, Sangamner 47 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 9.5: Survey Page test

Figure 9.6: Sensor test Failed

AVCOE, Sangamner 48 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 9.7: Road code missing

Figure 9.8: camera and GPS test Failed

AVCOE, Sangamner 49 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 9.9: GPS Error

9.3 Risk Management

9.3.1 Risk Identification
For risks identification, review of scope document, requirements specifications and
schedule is done. Answers to questionnaire revealed some risks. Each risk is cate-
gorized as per the categories mentioned in [?].You can refereed following risk iden-
tification questionnaire.
1. Have top software and customer managers formally committed to support the
project?
Answer : Yes

2. Are end-users enthusiastically committed to the project and the system/product

to be built?
Answer : Yes

3. Are requirements fully understood by the software engineering team and its
customers?
Answer : Yes

4. Have customers been involved fully in the definition of requirements?

Answer : Yes

AVCOE, Sangamner 50 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

5. Do end-users have realistic expectations?

Answer : Yes

6. Does the software engineering team have the right mix of skills?
Answer : Yes

7. Are project requirements stable?

Answer : Yes

8. Is the number of people on the project team adequate to do the job?

Answer : Yes

9. Do all customer/user constituencies agree on the importance of the project and

on the requirements for the system/product to be built?
Answer : Yes

9.3.2 Risk Analysis

• Technical Risk:
The probability of loss incurred through the execution of a technical process in
which the outcome is uncertain. Untested engineering, technological or manu-
facturing procedures entail some level technical risk that can result in the loss
of time, resources, and possibly harm to individuals and facilities. Like mobile
phone battery off, network error in user and server, multiple requests at time.

• Operational Risk:
Operational risk is the prospect of loss resulting from inadequate or failed pro-
cedures, systems or policies. Employee errors. Systems failures. Fraud or other
criminal activity. Any event that disrupts business processes.

• Schedule Risk:
Schedule risk is the risk that the project takes longer than scheduled. It can lead
to cost risks, as longer projects always cost more, and to performance risk, if
the project is completed too late to perform its intended tasks fully.

• Business Risk:
Business risk is the possibilities a company will have lower than anticipated
profits or experience a loss rather than taking a profit. Business risk is influ-
enced by numerous factors, including sales volume, per-unit price, input costs,
competition, and the overall economic climate and government regulations.

AVCOE, Sangamner 51 Department of Computer Engineering

 Chapter 10

Advantages, Limitations And Applications

10.1 Advantages
– Road condition monitoring process will become fast and efficient.
– This automated system will reduce human dependency of the process.
– Due to deep learning approach the system is real time.

10.2 Limitations
– Accuracy of system depends on the training dataset.
– System speed is dependent on hardware capabilities..

10.3 Applications
– Road authorities can implement this solution to collect road condition data
and can maintain roads efficiently..
– Self-driving cars can use this solution to control parameters like speed and
vehicle direction for smooth and safe ride.
– Like traffic data, road condition data collected by this solution can be
mapped on Google maps for better route selection from source to desti-
nation.
– Road condition data can be used by Government organization like “Min-
istry of Road Transport and Highways.

52
 Chapter 11

Results And Analysis

11.1 Login Page
– The Authorised Surveyor have to login with valid credentials RDD-ID and
password.
– The credentials are provided by the concern authority.

Figure 11.1: Login Page

53
 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

11.2 Survey

Figure 11.2: Survey Page 1

Figure 11.3: Survey Page 2

AVCOE, Sangamner 54 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 11.4: Survey Page 3

Figure 11.5: Survey Page 4

AVCOE, Sangamner 55 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 11.6: Detection Page 1

Figure 11.7: Detection Page 2

AVCOE, Sangamner 56 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 11.8: Detection Page 3

Figure 11.9: Detection Page 4

AVCOE, Sangamner 57 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 11.10: Detection Page 5

11.2.1 Data Distribution Platform

Figure 11.11: Survey or Information

AVCOE, Sangamner 58 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 11.12: Survey Information

Figure 11.13: Survey Data

AVCOE, Sangamner 59 Department of Computer Engineering

 Chapter 12

Conclusion And Future Scope

12.1 Conclusion
Road surface condition monitoring by the conventional method is very time
consuming, costly and slow. Due to an inefficient method of surveying road
are not being properly maintained. Our system approaches this problem by
using deep learning, out system include a CNN model (yolo model) capable of
detection and classification of the road damages from a video stream of input,
the output of the model is classification of damage and the bounding boxes of
the damage. The system collects the data and associate the data to a location
using GPS sensors output (latitude and longitude). The data collected can be
used for multiple use cases to make the data accessible an API service will be
implemented. System is totally dependent on the sensor data for the functional
requirements which makes it independent of human inspection. Elimination of
the human dependency of the process and automation of the process monitoring
and surveying the road networks become fast and efficient.
The presented solution proposes a system that is based on the CNN model
which is trained on a dataset of 26620 road images collected from Japan, India,
and the Czech Republic, and analyzes the methods that implement deep learn-
ing for automated detection and classification of road damage. This method
provide a base for quickly and cost-effectively surveying of the road network
by using mobile sensors camera and GPS.

12.2 Future Scope

• authorities can implement this solution to collect road condition data and can
maintain roads efficiently.
• Self-driving cars can use this solution to control parameters like speed and ve-
hicle direction for smooth and safe ride.
• Like traffic data, road condition data collected by this solution can be mapped
on Google maps for better route selection from source to destination.

60
 References
[1] Du, Y., Pan, N., Xu, Z., Deng, F., Shen, Y., and Kang, H.
(2020). Pavement distress detection and classification based
on yolo network. International Journal of Pavement Engi-
neering, pages 1–14.
[2] Maeda, H., Sekimoto, Y., Seto, T., Kashiyama, T., and Omata,
H. (2018). Road damage detection classification using deep
neural networks with smartphone images. Computer-Aided
Civil and Infrastructure Engineering, 33(12):1127–1141.
[3] Anand, S., Gupta, S., Darbari, V., and Kohli, S. (2018).
Crack-pot: Autonomous Road crack and pothole detection.
In 2018 Digital Image Computing: Techniques and Applica-
tions (DICTA), pages 1–6. IEEE.
[4] Fan, Z., Wu, Y., Lu, J., and Li, W. (2018). Automatic pavement
crack detection based on structured prediction with the con-
volutional neural network. arXiv preprint arXiv:1802.02208
[5] Zhang, L., Yang, F., Zhang, Y. D., and Zhu, Y. J. (2016). Road
crack detection using deep convolutional neural network. In
Image Processing (ICIP), 2016 IEEE International Confer-
ence on, pages 3708–3712. IEEE.
[6] Reagan L. Galvez; Argel A. Bandala; Elmer P. Dadios; Ryan
Rhay P. Vicerra; Jose Martin Z. Maningo. (2018). Object
Detection Using Convolutional Neural Networks. , TENCON
2018 - 2018 IEEE Region 10 Conference , IEEE.
[7] Dataset: RDD2020: An Image Dataset for Smartphone-
based Road Damage Detection and Classifica-
tion . Published: 18 March 2021 Version 1 DOI:
10.17632/5ty2wb6gvg.1 .
[8] Contributors: Deeksha Arya, Hiroya Maeda, Sanjay
Kumar Ghosh, DurgaToshniwal, HiroshiOmata, Takehiro
Kashiyama, Toshikazu Seto, Alexander Mraz, Yoshihide Seki-
moto.
[9] URL :- https://data.mendeley.com/datasets/5ty2wb6gvg/1.

61
 Annexure A

Plagiarism Report

Figure 12.1: Plagiarism Report

62
 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 12.2: Plagiarism Report

AVCOE, Sangamner 63 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 12.3: Plagiarism Report

AVCOE, Sangamner 64 Department of Computer Engineering

 Annexure B

Papers and Certificates

Figure 12.4: Certificate

65
 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 12.5: Certificate

Figure 12.6: Certificate

AVCOE, Sangamner 66 Department of Computer Engineering

 Deep Learning Approach To Detect And Classify Road Surface Damage Using Mobile Sensors.

Figure 12.7: Certificate

AVCOE, Sangamner 67 Department of Computer Engineering