Object Detection Prototype

Object Detection Introduction

»» Computer vision (CV)is an interdisciplinary field that deals with

how computers can be made to gain high-level understanding
from digital images or videos.
»» Machine learning (ML) is the scientific study of algorithms and
statistical models that computer systems use to perform a specific
task without using explicit instructions, relying on patterns and
inference instead.
»» OpenCV (Open Source Computer Vision Library) is an open source
machine learning and computer vision library developed by Intel
for real-time image & video analysis and processing. Primarily
written in C++, This library has bindings for Python, Java, Matlab,
Octave etc. Open-CV combined with python makes image/video
analysis and processing astonishingly simple and for many, it can
also be the first step in the world of Computer Vision.
»» In this document we look at how raspberry pi can be used for
machine learning research using OpenCV and various object
detection models and frameworks.
»» We explored OpenCV using python and data collection setup with
amazon web server and MySQL database to analyze the data and
further visualize it on Power BI. (Refer to the diagram below)

8/23/2019 Page 1
Human Count by Date-Time
3.0
3.00 3.00 3.00 3.00

2.5

2.00 2.00 2.00 2.00 2.00

2.0
2.00
Person Count

1.5

1.00 1.00 1.00 1.00 1.00

1.0
1.00

0.5

0.00 0.00
0.0
12:17:00 PM 12:18:00 PM 12:19:00 PM 12:20:00 PM 12:21:00 PM
Date Time
1/1

Copyright © 2019 Proving Ground LLC 2

Raspberry Pi Introduction

»» Raspberry Pi is the name of a series of single-board computers made by the Raspberry Pi Foundation, a UK charity
that aims to educate people in computing and create easier access to computing education.
»» Raspberry Pi operates in the open source ecosystem: it runs Linux (a variety of distributions), and its main
supported operating system, Raspbian, is open source and runs a suite of open source software. NOOBS is an easy
operating system installer which contains Raspbian. We are using Raspbian GNU/Linux 10 (buster) version 10.
»» We have used Raspberry Pi 3 Model B+ (2018) and Arducam camera module for Raspberry Pi. Pi-camera can be
connected to the raspberry pi as an add-on accessory which will be used for object detection research.

»» Since we are using Linux system on the

Pi, it is to be noted that the terminal
or command-line is used to control the
system. Command-line allows user to
directly manipulate their system through
the use of commands.

Copyright © 2019 Proving Ground LLC 3

OpenCV Installation

»» The tutorial we followed to download OpenCV4 can be found here: https://www.pyimagesearch.com/2018/09/26/

install-OpenCV-4-on-your-raspberry-pi/

Something to Keep in Mind:

»» This will take up a significant amount of space on the Micro SD card of your Raspberry Pi.

»» This demonstration can be done utilizing SSH which allows you to operate your Raspberry Pi, but we will do this
with physical access to the Raspberry Pi terminal.

»» Follow the instructions on the tutorial. Once you have installed all the supportive libraries and pip, you will create
your virtual environment. It is important to get this step right because you will create the OpenCV in the virtual
environment. Once downloaded, OpenCV will only be accessible in this virtual environment.

»» Virtual environment is a python component that creates an isolated environment for python projects. This means
that each project can have its own dependencies, regardless of what dependencies every other project have. Virtual
environment by the name ‘cv’ is created to download OpenCV in this document.

»» It can be challenging to download OpenCV or Tensorflow on raspberry pi since it’s important to have the right
versions installed and also using the command line can be foreign concept.

»» After this step:

$$ sudo pip install virtualenv virtualenvwrapper

$$ sudo rm -rf ~/get-pip.py ~/.cache/pip

»» Open:
$$ nano ~/.profile

»» Once in the profile, copy and paste the command for the virtual environment and virtual environment wrapper to
the end of the command explanation:

$$ # virtualenv and virtualenvwrapper

$$ export WORKON_HOME=$HOME/.virtualenvs
$$ export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
$$ source /usr/local/bin/virtualenvwrapper.sh

»» The next step will access your virtual environment so that you can build the OpenCV in it.
$$ source ~/.profile

»» Continue through the tutorial until you reach:

$$ make -j4

»» Reserve a few hours for this step. This is when OpenCV will install onto your Raspberry Pi.
»» Continue on to finish the tutorial and feel free to follow along with the practice example given.

Copyright © 2019 Proving Ground LLC 4

1. Simple Object Detection

»» In this example we see how human face can be detected and from a live video feed to record the presence of
number of humans present in the frame and detect their movement. It is further used to record number of face
detected in the room at a given time which is stored in Amazon web server and visualize it on Power BI through
MySQL workbench.
»» We have followed the blog post by Pyimagesearch on Simple object detection and customized the data being
collected. You can find the link to the bold post here- https://www.pyimagesearch.com/2018/07/23/simple-object-
tracking-with-OpenCV/

»» Object Tracking process:

• Initial Object detection using CNN object detector
• Create unique ID for each detection
• Track the IDs through centroid tracker
• Upload the data collection to MySQL
• Visualize data collected on Power BI

»» Step 1 : Activate the Virtual Environment (cv) and navigate to the folder location using command line. Further deploy
the python file along with its caffe model (Caffe model is a deep learning framework). Since this example uses
argument parser, caffe model and prototxt files can be replaced with the latest suitable library available on OpenCV
online.

$$ source ~/.profile
$$ workon cv
$$ cd ~/simple-object-tracking # folder name found inside user folder pi
$$ python object_tracker.py --prototxt deploy.prototxt --model res10_300x300_ssd_iter_140000.caffemodel

»» Important snippets from the python file and further customization of the code to collect data is noted below.
»» MySQLdb is imported along with datetime to record the time to the database. MySQLdb has to be downloaded
on virtual environment (cv). Downloading mysql-client, mysql-connector, mysql-connector-python on the virtual
environment cv helped us to successfully import MySQLdb statement and execute the python script.

Copyright © 2019 Proving Ground LLC 5

»» Data being recorded are the rectangle drawn from the frame after object detection is done.
»» Cameraid is a constant value that records the pi-camera and its respective Raspberry Pi being used. In this example
we are extracted values from a single Pi.
»» Line 88-90 is collecting date and time.

»» Below snippet shows the connection to database and the data formated according to the table that is created in
MySQLdb.

HostName UserName Password DatabaseName

»» This is printing the values that is being uploaded to the database on the terminal at an regular time interval of 2
seconds. Data recorded is also being uploaded every two second

»» This example fall short in few ways. Although it is detecting human face, the centroid tracker is not accurate with
respect to object movement. Also proximity of background subtraction done is not precise to locate farther objects
with pi-camera.

Copyright © 2019 Proving Ground LLC 6

2. Filter Object Detection

»» In this example we see how person can be detected and from a live video feed to record the presence of number
of humans present in the frame and detect their movement. It is further used to record number of people detected
in the room at a given time which is stored in Amazon web server and visualize it on Power BI through MySQL
workbench.
»» We have followed the blog post by Pyimagesearch on Filter object detection and customized the data being
collected. You can find the link to the bold post here- https://www.pyimagesearch.com/2018/05/14/a-gentle-guide-
to-deep-learning-object-detection/

»» Object Tracking process:

• Initial object detection using COCO base model
• Filter person from other common objects
• Upload the data collection to MySQL
• Visualize data collected on Power BI

»» Step 1 : Activate the Virtual Environment (cv) and navigate to the folder location using command line. Further deploy
the python file along with its caffe model (Caffe model is a deep learning framework). Since this example uses
argument parser, caffe model and prototxt files can be replaced with the latest suitable library available on OpenCV
online.
»» Important snippets from the python file and further customization of the code to collect data is noted below.

$$ source ~/.profile
$$ workon cv
$$ cd ~/filter-object-detection # folder name found inside user folder pi
$$ python filter_object_detection.py --prototxt MobileNetSSD_deploy.prototxt.txt --model MobileNetSSD_deploy.
caffemodel

»» MySQLdb is imported along with datetime to record the time to the database. MySQLdb has to be downloaded
on virtual environment (cv). Downloading mysql-client, mysql-connector, mysql-connector-python on the virtual
environment cv helped us to successfully import MySQLdb statement and execute the python script.

Copyright © 2019 Proving Ground LLC 7

»» COCO (common object in context) model is a collection of 20 common objects to detect. Objects can be filtered
by setting the ignore class. In this example since we are looking for person detection, we are going to ignore every
other object except “person”.

»» Object detection is counted for data collection and since we are have ignore all the other objects we are just
counting humans.
»» Cameraid (line 78) is a constant value that records the pi-camera and its respective Raspberry Pi being used. In this
example we are extracted values from a single Pi.
»» Line 80-82 is collecting date and time.

»» Below snippet shows the connection to database and the data formated according to the table that is created in
MySQLdb.

HostName UserName Password DatabaseName

»» This is printing the values that is being uploaded to the database on the terminal at an regular time interval of half a
second. Data recorded can be set at 2 seconds which is good enough to analyze for machine learning.
»» Note that time.sleep line is used at other places to start the object detection and end the detection. It is important
to set this statement associated with mySQLdb data collection as well.

»» To stop the python file from running the object tracking click on the video frame and ‘q’ on the keyboard as it is set
as the break key in the code.

Copyright © 2019 Proving Ground LLC 8

»» This example also reads the elapsed time of the data recorded as well as approximate FPS at which the file was
executed.
»» It is to be noted that since we are using Raspberry pi for computer vision, the FPS of the executed file is extremely
slow. Maximum FPS we attained is 0.5 while using other system the same example can be executed at about 20
FPS.
»» People recognition is pretty good in this example since we are using COCO model which is widely used model
as well as the confidence is set at 0.2 which ranges from 0 to 1. Confidence is the minimum probability of the
detection set to filter out weak detection.

Copyright © 2019 Proving Ground LLC 9

MySQL Database:

»» For the next step of this, we will be working in MySQL

Workbench to read the data collected from the object detection
using OpenCV and Python from Raspberry Pi.

»» You will have to have your own MySQL Server and password at
which point you can open and load MySQL Workbench.

»» Running this script through MySQL gives us a table named

“HumanIdData”. This gives a destination for the information of
the python script.

»» In this table we are recording cameraID, date, time and humanID

or human count.

»» Opening the tab you have labeled, in this

case “HumanIdData” you should see your
values start to load into the table with a
time stamp on each value. We used two
different python scripts that each defined
which CameraID was which. Multiple
cameras associated to its system can be
used to analyze various rooms or location
to collect data.

Copyright © 2019 Proving Ground LLC 10

Power BI:
»» To take your data a step further, we can load your collected data
into Power BI to visually explain your data.

»» Select “Get Data,” from which you can pick MySQL database to
extract your data from. You will need to log in to your server
again.

»» Your data should appear in the “Fields” section of Power BI

towards the left. You can alter how you view your information in
“Visualizations” next to “Fields.”

»» The visualized data shown below is within a small time frame to

understand clearly the movement in the room in which the Pi-
camera is set up.

8/23/2019 Page 1
Human Count by Date-Time
6

4
Person Count

3.0 3.0 3.0 3.0

3

2.0 2.0 2.0 2.0 2.0

2
2.0

1.0 1.0 1.0 1.0 1.0

1
1.0

0.0 0.0
0
12:17:00 PM 12:18:00 PM 12:19:00 PM 12:20:00 PM 12:21:00 PM
Date Time
1/1

Copyright © 2019 Proving Ground LLC 11