Proceedings of the Third International Conference on Smart Systems and Inventive Technology (ICSSIT 2020)

IEEE Xplore Part Number: CFP20P17-ART; ISBN: 978-1-7281-5821-1

Design & implementation of real time autonomous

car by using image processing & IoT

Irfan Ahmad Karunakar Pothuganti

Department of Electrical engineering Department of Electronics and Communication Engineering
Indian Institute of technology Sri Satya Sai University of Technology & Medical sciences
Jodhpur, India. Sehore, India.
ganieirfan27@gmail.com karna119@gmail.com

Abstract— Because of the inaccessibility of Vehicle -to- profound learning procedures [13]-[18] or AI [8]-[12] to
Infrastructure correspondence in the present delivering prepare the model in an information-driven way. Profound
frameworks, (TLD), Traffic S ign Detection and path adaptive based convolutional neural system (CNN) model
identification are as yet thought to be a significant task in self- accomplishes preferable execution or results [7],[19],[20] over
governing vehicles and Driver Assistance S ystems (DAS ) or S elf the AI algorithms dependent on the Histogram of Oriented
Driving Car. For progressively exact outcome, businesses are Gradients (HOG) highlights, (for example, SVM, and the
moving to profound Neural Network Models Like Convolutional Hidden Markov Model). This is a way that CNN can extricate
Neural Network (CNN) as opposed to Traditional models like and take in increasingly unadulterated highlights from the
HOG and so forth. Profound neural Network can remove and
crude RGB channels than old calculations, for example, HOG.
take in increasingly unadulterated highlights from the Raw RGB
picture got from nature. In any case, profound neural systems In any case, calculation multifaceted nature of CNN models is
like CNN have a highly complex calculation. This paper proposes a lot greater than that of most AI calculations. Along these
an Autonomous vehicle or robot that can identify the diverse lines, in this paper, a profound neural system is proposed to
article in condition and group them utilizing CNN model and utilize distinguish different segment in condition to settle on
through this information can take some continuous choice which some significant choice which will be helpful in the field of a
can be utilized in the S elf Driving vehicle or Autonomous Car or self-driving vehicle or Autonomous Vehicle or Driving
Driving Assistant S ystem (DAS ). Assistant System (DAS).

Keywords— CNN, Deep Neural Network, Driving Assistant

This paper organizes as follows section II describes the
System (DAS), Self-Driving car. literature review, section III gives a block diagram and
workflow of the proposed system. Section IV gives
information about model for object detection. Section V
I. INTRODUCTION computer vision algorithm for line detection followed
With the improvement of different sensor procedures, (for simulation results and conclusion.
example, LIDAR, millimeter-wave radar, camera, and
differential-GPS), more profound adaptive models have come II. LITERATURE REVIEW
to up investigate on autonomous - vehicles in the previous time.
A most dependable answer for dispatch and board of Autonomous car driving was analyzed and implemented by
Autonomous Car sooner than later perhaps Vehicle-to- using CNN. Many methods were proposed and implemented
Infrastructure correspondence [1]. In the V2I correspondence in this section. In [21] authors discussed evaluating the current
framework, oneself driving vehicle can associate with the challenges in artificial intelligence and how these
general condition and with one another legitimately through the methodologies can be applied to cybersecurity in the USA,
vehicular systems [2]. In any case, because of the absence of including traffic analysis schemes. In [22], this paper gives the
the V2I framework, finders for perceiving and following advanced driving assistance system that will give past, present
vision-based traffic lights, signs, paths people on foot despite and future driving different variants and it is analyzed by
everything assume significant jobs in present self-driving using real-time embedded systems. Authors in [23], describes
frameworks or self-ruling driving colleague frameworks. methods for naturalistic data collection and algorithm
Past, PC vision-based arrangements [3]-[6] are profoundly development based on pre-scan and driving simulator. Along
influenced by the situation of cameras, the effect of ecological with this in [24], the author describes various information
light conditions, separation of articles [7], preparing capacity of technologies like IoT, AI, blockchain technology, human
the vehicle chip. In any case, a solitary arrangement of cloud and robotic techniques. Authors in [25], gave
physically fixed parameters dependent on the heuristic strategy information about frequently used algorithms for computer
isn't adaptable for confused genuine conditions, and vision in the field of medical and traffic analysis.
recognizable proof of reasonable parameters is additionally In [26], the authors described computer vision research in the
tedious. In this manner, investigate is focused on utilizing university Waikato. Hyperspectral image formation and digital

978-1-7281-5821-1/20/$31.00 ©2020 IEEE 107

Authorized licensed use limited to: Auckland University of Technology. Downloaded on October 24,2020 at 06:28:22 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Third International Conference on Smart Systems and Inventive Technology (ICSSIT 2020)
IEEE Xplore Part Number: CFP20P17-ART; ISBN: 978-1-7281-5821-1

line scan photography [26] these type of research areas are From camera, input gave to raspberry pi, in raspberry pi
analyzed by using computer vision. python language used to communicate between CNN network
and different sensors in the network. Through raspberry pi,
total data will be uploaded to the cloud platform. From the
III. PROPOSED WORK website, can be seen where the driving is. This total setup is
The proposed technology for the autonomous car by using made and shown in figure 3.
CNN is shown in figure 1 and the experimental setup shown in
figure 3. This total system has to be fixing on any motor
vehicle so that camera has to be in the direction of the road.

Fig 1. The architecture of Convolutional Neural Network

IoT cloud
RASBERRY-Pi

Pi-CAM

Sensors
Motors

Real time feed of Car using

Website

Fig 2: Block diagram of Autonom ous car using CNN

978-1-7281-5821-1/20/$31.00 ©2020 IEEE 108

Authorized licensed use limited to: Auckland University of Technology. Downloaded on October 24,2020 at 06:28:22 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Third International Conference on Smart Systems and Inventive Technology (ICSSIT 2020)
IEEE Xplore Part Number: CFP20P17-ART; ISBN: 978-1-7281-5821-1

Using a deep convolutional neural network and various image  Live feed will have data regarding the environment
processing Techniques detecting real-time objects such as seen by the robot using pi-Cam.
vehicles, Traffic Light, Traffic Signs etc. Detecting Lane using  Remote controlling of the vehicle on our Web Server
a neural network. Classifying the detected object us ing deep using IoT
neural network.  Making of a small robot with various sensors like
Procedure for line detection: Ultrasonic sensor etc. to illustrate above-mentioned
 Giving insights regarding the detected object about features.
their distance and location with regard to concern  A semi-automated robot having the ability to make
vehicle. some real-time decision (Lane detection, stop & run,
 Giving live feed of our vehicles on our web server Traffic light condition etc.)
using IoT.

Fig 3: Experimental setup for lane detection using raspberry pi.

Components used in the experimental setup as follows.

1. Raspberry pi B+ model T ABLE 1: RASP BERRY P I CAM SP ECIFICATIONS
2. Motor Driver (L298N) Still resolution 5Mega pixel
To drive DC and stepper Motors in the project, the
L298N driver module is used. It is a high power Video Resolution 2592 × 1944 pixels
driver module. Our module contains:
 L298 motor driver IC S/N ratio 36 dB
 78M05 5V regulator. Sensitivity 680/lux-sec
This motor driver module can control two DC motors
along with speed control and directional control. 7. Ultrasonic sensor (HC-SR07)
3. Power Bank (for source) The ultrasonic sensor is used in this project, to detect
4. Bread Board the object and to detect the distance between the
5. LCD (liquid crystal display) (16x2) object and source. This sensor contains a transmitter
6. Pi CAM (for video feed) and receiver which used to transmit and receive
The Raspberry Pi Camera Modules are official inaudible sound waves, which is faster than audible
products from the Raspberry Pi. In our project, the 5- sound. Based on the time to receive an echo from the
megapixel model Camera Module v2 is used. object, the distance will be calculated.
8. Potentiometer(10k ohm)
9. Resistor(1K ohm)
10. Left Motor

978-1-7281-5821-1/20/$31.00 ©2020 IEEE 109

Authorized licensed use limited to: Auckland University of Technology. Downloaded on October 24,2020 at 06:28:22 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Third International Conference on Smart Systems and Inventive Technology (ICSSIT 2020)
IEEE Xplore Part Number: CFP20P17-ART; ISBN: 978-1-7281-5821-1

11. Right motor Yolo Object Detection Algorithm

It is a single neural system that predicts jumping boxes and
All the devices in the system are connected to the groups the different item in one assessment. In the given
raspberry pi. From pi camera, input feed was given to the model procedure, can process 45 edges for each second
Rpi. Rpi analyzed by using the ultrasonic sensor to send progressively and have executed this model as convolutional
the data to the Rpi. Rpi made the decision and sends it to organize. This model partitions the picture taken as a
left and right motor. The morion and status of the car will contribution to a type of network. If the focal point of an
be known through LCD.. article falls in a network cell, the cell is liable for recognizing
the object. Each matrix cell predicts many jumping boxes with
IV. MODEL FOR OBJECT DETECTION the assured scores which reflects how much the bounding box
is sure that the article is available in the case. Each boun ding
There are several Neural Network architectures for detection:- box comprises of 5 expectations: x, y, w, h and certainty score.
Where: -
 R-CNN architectures X = Center coordinate of bound box (x-axis)
Y = Center coordinate of bound box (y-axis)
 Single Shot Detectors(SSD) H = Height of the bound box
W = Width of the bound box.
 YOLO — You Only Look Once

This paper is implemented on YOLOv3(Version 3

architecture) without going into many details as to how it
works. YOLO models are not as accurate as R-CNN models
but they are swift and are easily suitable for real-time
applications. All these images are collected in a separate
folder, named as training dataset and have labelled them by
drawing bounding boxes around every car you found .

Fig 4: YOLO V3 Network Architecture.

Fig5. Encoding architecture for YOLO V3

978-1-7281-5821-1/20/$31.00 ©2020 IEEE 110

Authorized licensed use limited to: Auckland University of Technology. Downloaded on October 24,2020 at 06:28:22 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Third International Conference on Smart Systems and Inventive Technology (ICSSIT 2020)
IEEE Xplore Part Number: CFP20P17-ART; ISBN: 978-1-7281-5821-1

by 6 numbers — pc, bx, by, bh, bw, c. 80-

Design
dimensional vector is used.
Underlying layers of the convolutional neural system separate
highlights while the completely associated layers yield B. Anchor Boxes
directions and probabilities. Our model has 24 convolutional  Anchor boxes are chosen by exploring the training
layers (CNN) in continuation with 2 completely associated data to choose reasonable height/width ratios that
layers. represent the different classes.
In this, the primary purpose is to predict the lane in the image.  The dimension for anchor boxes is the second to last
This can be done by finding the change in gradient. Ch ange in dimension in the encoding: m, nH, nW, anchors,
gradient to the color is the change in the intensity or difference classes.
in the intensity between adjacent pixels. So, first, have to  The YOLO architecture is: IMAGE (m, 840, 840, 3) -
convert our image to grayscale. The images are taken using a > DEEP CNN -> ENCODING (m, 19, 19, 5, 85).
camera with 840X840.
C. Encoding
A. Inputs and outputs In the encoding process, reduction factor 32 is used, Each box
 The input is a batch of images, and each image has contains six features pc,bx,by,bh,bw,c as shown in figure 4.
the shape (m, 840, 840, 3). The object is identified with a probability score of 0.44 as
 The output is a list of bounding boxes along with the shown in figure 5.
recognized classes. Each bounding box is represented

Fig 6: Class identification by each class box

(a) Original image

(b) Processed image

Fig 7: Shows the caany detection output from the input lane to edge detected lane

978-1-7281-5821-1/20/$31.00 ©2020 IEEE 111

Authorized licensed use limited to: Auckland University of Technology. Downloaded on October 24,2020 at 06:28:22 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Third International Conference on Smart Systems and Inventive Technology (ICSSIT 2020)
IEEE Xplore Part Number: CFP20P17-ART; ISBN: 978-1-7281-5821-1

images as shown in figure 8 and figure 9. This detection of

After this, to accurately detect as many edges possible in the edges has been utilized in the lane detection on the road. Apart
image, noises to be reduced and smoothening of the image is from being a simple model it is can only be used for only
required. The Gaussian filter has been applied for simple edge detection problems because the basic concept is
smoothening of images . Figure 7 shows how the image can be of this network is convoluting the image with the filter and it
transferred into an edge detection image. The transformation can detect edges up to some extent only. This algorithm only
of the original image to the processed image is shown in figure takes the input in Grayscale images. But in our case, this
7. Now for edge detection, the Canny edge detection method algorithm is holding up very well. It can detect the edges of
is applied to find the gradient of the image at each pixel. Now the lane in which the driver is moving. This also giving the
our image will be masked with the triangular mask which will driver a proper insight if he is moving in the lane correctly or
be our area of interest and help us to find the lane. not. In Table 2, given the parameters of training time, error
and accuracy for YOLO v3
V. RESULT S
The above-mentioned algorithm i.e. Yolo for object detection T ABLE 2: SIMULATION P ARAMETERS OF THE MODEL
has been used and got all the bounding boxes whose threshold Parameters YOLOv3
value is greater than the 0.44. As seen from the above image Time for training 9 hours
from Yolo algorithm, not able to just detect the traffic light but Accuracy on Test data 100%
also the other things like (vehicles, people etc.). This gives the Input image resolution 840X840
driver a good inference about his surroundings. This will help Loss 0.1708
the driver to know his surrounding properly in case of loss of Time for prediction 1.5 sec
attention of the driver. This algorithm proves to be the fastest Detection of object types Small, medium & large
algorithm to detect the object in the surrounding and a
valuable algorithm in the field of an autonomous car.
The experimental results of autonomous car detection were VI. CONCLUSION
shown in figure 8 and 9. Figure 5 shows the object detection From the above algorithms , can be concluded that both
whereas figure 9 shows the lane detection in real-time. algorithms are working well in their respective case scenario.
The Yolo proves to be the fast algorithm and work great in a
real-time scenario for Object Detection. Instead of classifying
a single class i.e. class like (traffic) lights, it can detect
multiple classes like (people, vehicles, traffic, and lights,
animal) giving a full and proper insight to the driver about
his/her surrounding. It also uses CNN like other object
detection but because of its speed and performance in a real-
time scenario, it out-performs other algorithms. But a part of
its huge advantages , it has some limitations like YOLO forces
solid spatial requirements on jumping box forecasts since
every lattice cell just predicts two boxes and can just have one
class. But despite this, it is great for an autonomous car. For
Lane detection also, the algorithm that is Canny Edge
Fig 8: Object detection
detection is good as it solves our purpose. But it is a very
simple algorithm which fails in a complex situation and have
used this algorithm to reduce the complexity of our mo del and
reducing both power consumption and inference time of our
application. Whenever making the application there are
mainly three important constraints (time, Power and space).
Use this simple algorithm reduces both and it is working in
our scenario so this outcast other algorithm but irrespective to
other algorithms its accuracy is less

REFERENCES

[1] O. Popescu and et al., “Automatic incident detection in intelligent

Fig 9: Lane detection transportation systems using aggregation of traffic parameters collected
From the lane detection algorithm, i.e. canny edge detection, through v2i communications,” IEEE Intelligent T ransportation Systems
Magazine, vol. 9(2), pp. 64–75, 2017.
it’s a very simple algorithm to use to detect the edges in the

978-1-7281-5821-1/20/$31.00 ©2020 IEEE 112

Authorized licensed use limited to: Auckland University of Technology. Downloaded on October 24,2020 at 06:28:22 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Third International Conference on Smart Systems and Inventive Technology (ICSSIT 2020)
IEEE Xplore Part Number: CFP20P17-ART; ISBN: 978-1-7281-5821-1

[2] R. Atallah, M. Khabbaz, and C. Assi, “Multihop v2i communications: A [16] Z. Ouyang and et al., “A cgans-based scene reconstruction model using
feasibility study, modeling, and performance analysis,” IEEE lidar point cloud,” in IEEE International Symposium on Parallel and
T ransactions on Vehicular Technology, vol. 18(2), pp. 416–430, 2017. Distributed Processing with Applications. IEEE, 2017.
[3] G. T rehard and et al., “Tracking both pose and status of a traffic light [17] K. Behrendt, L. Novak, and R. Botros, “A deep learning approach to
via an interacting multiple model filter,” in 17th International traffic lights: Detection, tracking, and classification,” in 2017 IEEE
Conference on Information Fusion (FUSION). IEEE, 2014, pp. 1 –7. International Conference on Robotics and Automation (ICRA). IEEE,
[4] A. Almagambetov, S. Velipasalar, and A. Baitassova, “Mobile 2017, pp. 1370–1377.
standards-based traffic light detection in assistive devices for individuals [18] M. Weber, P. Wolf, and J. M. Zollner, “Deeptlr: A single deep
with color-vision deficiency,” in IEEE T ransactions on Intelligent convolutional network for detection and classification of traffic lights,”
T ransportation Systems, vol. 16(3). IEEE, 2015, pp. 1305 – 1320. in 2016 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2016, pp.
[5] J. Al-Nabulsi, A. Mesleh, and A. Yunis, “T raffic light detection for 342–348.
colorblind individuals,” in 2017 IEEE Jordan Conference on Applied [19] A. Mogelmose, M. M. T rivedi, and T . B. Moeslund, “ Vision -based
Electrical Engineering and Computing T echnologies (AEECT ). IEEE, traffic sign detection and analysis for intelligent driver assistance
2017, pp. 1–6. systems: Perspectives and survey,” in IEEE Transactions on Intelligent
[6] X. Li and et al., “ Traffic light recognition for complex scene with fusion T ransportation Systems, vol. 13(4). IEEE, 2012, pp. 1484 – 1497.
detections,” in IEEE T ransactions on Intelligent T ransportation Systems, [20] M. Chen, U. Challita, and e. a. Walid Saad, “Machine learning for
vol. 19(1). IEEE, 2018, pp. 199–208. wireless networks with artificial intelligence: A tutorial on neural
[7] M. B. Jensen and et al., “Vision for looking at traffic lights: Issues, networks,” vol. arXiv:1710.02913. arXiv.incident detection in intelligent
survey, and perspectives,” in IEEE T ransactions on Intelligent [21] Soni, Vishal Dineshkumar, Challenges and Solution for Artificial
T ransportation Systems, vol. 18(2). IEEE, 2016, pp. 1800 –1815. Intelligence in Cybersecurity of the USA (June 10, 2020). Available at
[8] Y. Ji and et al., “ Integrating visual selective attention model with hog SSRN: https://ssrn.com/abstract=3624487.
features for traffic light detection and recognition,” in 2015 IEEE [22] A. Shaout, D. Colella and S. Awad, "Advanced Driver Assistance
Intelligent Vehicles Symposium (IV). IEEE, 2015, pp. 280–285. Systems - Past, present and future," 2011 Seventh International
[9] Z. Chen, Q. Shi, and X. Huang, “Automatic detection of traffic lights Computer Engineering Conference (ICENCO'2011), Giza, 2011, pp. 72-
using support vector machine,” in 2015 IEEE Intelligent Vehicles 82, doi: 10.1109/ICENCO.2011.6153935.
Symposium (IV). IEEE, 2015, pp. 37–40. [23] L. Li and X. Zhu, "Design Concept and Method of Advan ced Driver
[10] Z. Chen and X. Huang, “Accurate and reliable detection of traffic lights Assistance Systems," 2013 Fifth International Conference on Measuring
using multiclass learning and multiobject tracking,” in IEEE Intelligent T echnology and Mechatronics Automation, Hong Kong, 2013, pp. 434 -
T ransportation Systems Magazine, vol. 8(4). IEEE, 2016, pp. 28 –42. 437, doi: 10.1109/ICMT MA.2013.109.
[11] Z. Shi, Z. Zou, and C. Zhang, “Real-time traffic light detection with [24] Soni, Vishal Dineshkumar, Information T echnologies: Shaping the
adaptive background suppression filter,” in IEEE T ransactions on World under the Pandemic COVID-19 (June 23, 2020). Journal of
Intelligent T ransportation Systems. IEEE, 2016, pp. 690 –700. Engineering Sciences, Vol 11, Issue 6,June/2020, ISSN NO:0377 -9254;
DOI:10.15433.JES.2020.V11I06.43P.112 , Available at
[12] X. Du and et al., “ Vision-based traffic light detection for intelligent
SSRN: https://ssrn.com/abstract=3634361
vehicles,” in 2017 4th International Conference on Information Science
and Control Engineering (ICISCE). IEEE, 2017, pp. 1323 –1326. [25] Q. Wu, Y. Liu, Q. Li, S. Jin and F. Li, "T he application of deep learning
in computer vision," 2017 Chinese Automation Congress (CAC), Jinan,
[13] V. John and et al., “ Saliency map generation by the convolutional neural 2017, pp. 6522-6527, doi: 10.1109/CAC.2017.8243952.
network for real-time traffic light detection using template matching,” in
IEEE T ransactions on Computational Imaging, vol. 1(3). IEEE, 2015, [26] M. J. Cree et al., "Computer vision and image processing at the
pp. 159–173. University of Waikato," 2010 25th International Conference of Image
and Vision Computing New Zealand, Queenstown, 2010, pp. 1 -15, doi:
[14] S. Saini and et al., “ An efficient vision-based traffic light detection and 10.1109/IVCNZ.2010.6148863.
state recognition for autonomous vehicles,” in 2017 IEEE Intelligent
Vehicles Symposium (IV). IEEE, 2017, pp. 606 –611. [27] ‘A Comparative Study of Real T ime Operating Systems for Embedded
Systems’ International Journal of Innovative Research in Computer and
[15] J. Campbell and et al., “Traffic light status detection using movement
Communication Engineering, Vol. 5, Issue 1, June 2016, ISSN: 2320-
patterns of vehicles,” in 19th International Conference on Intelligent
9801, IMPACT FACT OR 6.7
T ransportation Systems (IT SC). IEEE, 2016, pp. 283 –288.

978-1-7281-5821-1/20/$31.00 ©2020 IEEE 113

Authorized licensed use limited to: Auckland University of Technology. Downloaded on October 24,2020 at 06:28:22 UTC from IEEE Xplore. Restrictions apply.