2 DOF BALL BALANCER USING IMAGE PROCESSING

B Tech Mini-Project Report

Submitted in partial fulfillment for the award of the Degree of

Bachelor of Technology in Electrical and Electronics Engineering

by
KIRAN G KRISHNAN (Roll No.B160111EE)
STEVE EAPEN (Roll No.B160707EE)
MATHEW MARTIN (Roll No.B160318EE)
KRITIKA DUTTA (Roll No.B160264EE)

Under the guidance of

Dr. Jeevamma Jacob

Department of Electrical Engineering

NATIONAL INSTITUTE OF TECHNOLOGY CALICUT
NIT Campus P.O., Calicut - 673601, India
2019
 CERTIFICATE
This is to certify that the thesis entitled “2 DOF BALL BALANCER
USING IMAGE PROCESSING” is a bona fide record of the mini-project done by
KIRAN G KRISHNAN (Roll No.B160111EE), STEVE EAPEN (Roll No.
B160707EE), MATHEW MARTIN (Roll No. B160318EE) and KRITIKA DUTTA
(Roll No. B160264EE) under my supervision and guidance, in partial fulfillment of
the requirements for the award of Degree of Bachelor of Technology in Electrical &
Electronic Engineering from National Institute of Technology Calicut for the year
2019.

Dr.JEEVAMMA JACOB
(Guide)
Professor
Dept. of Electrical Engineering

Dr SALY GEORGE
Professor & Head
Dept. of Electrical Engineering

Place: NIT CALICUT

Date: 25/04/2019
 ACKNOWLEDGEMENT

We express our deep sense of gratitude to our respected and learned project guide,
Dr. JEEVAMMA JACOB, for the valuable help and guidance provided in the
course of completion of this project.

We are also grateful to respected Prof Dr. SALY GEORGE, Head of the
Department of Electrical Engineering and to our respected Director, Dr. SIVAJI
CHAKRAVORTI, NIT Calicut for permitting us to utilize all the necessary facilities
of the institution.

We are also thankful to all the other faculty and staff members of our department
for their kind cooperation and help.

Lastly, we would like to express our deep appreciation to all our classmates and
indebtedness to our parents for providing us the moral support and the
encouragement.

KIRAN G KRISHNAN
STEVE EAPEN
MATHEW MARTIN
KRITIKA DUTTA
 ABSTRACT

Laboratory experiments are an integral part of control education. There are lots of
educational platforms available e.g. inverted pendulum, magnetic levitation, ball and
beam system etc. Ball and plate system is an upgraded version of ball and beam
system where the position of the ball can be manipulated in two directions. The
educational model ball and plate, consists of a plate, pivoted at its centre, such
that the slope of the plate can be manipulated in two perpendicular directions.

This project is designed to balance a ball on a plate using image processing. The ball-
plate system, with 2 degrees of freedom, is using vision feedback as a sensor and DC
servo motors as actuators. This system is an open-loop, nonlinear and unstable
system. The target in this project is to create a closed-loop controller using PID to
calculate the error and to make sure fast communication between sensor and actuator,
thus correction of ball position to the centre of the plate. The communication of data
is handled by a middleware, Robot Operating System (ROS) and image processing
the handled by OpenCV. The system consists of the ball-plate system as plant, two
DC servo motors as actuators and a camera as a feedback sensor. The control system
is implemented with ARDUINO to control the servo motors. Through this project, a 2
DOF ball balancer using image processing as feedback is implemented and tested.

Keywords: Ball-Plate system; ARDUINO; openCV; PID control system; feedback

sensor; image processing; Robot Operating System
 CONTENTS

Chapter No TITLE Page no.

List of Abbreviations ii
List of Symbols iii
List of Figures iv
List of Tables vi

1 INTRODUCTION 1
1.1 Introduction 1
1.2 Aim 3
1.3 Working 3
2 BALL AND PLATE SYSTEM 5
2.1 Introduction 5
2.2 Mechanical Design 5
2.2.1 Modeling 5
2.3 Control Design 7
2.4 Simulation Results 9
2.5 Summary 11
3 HARDWARE DESCRIPTION 12
3.1 Introduction 12
3.2 Arduino (Mega) 12
3.3 Servo motor 14
3.3.1 PWM 14
3.3.2 Servo Motor (MG 995) Specifications 15
3.4 Camera 16
3.5 Lithium Polymer battery 17
3.6 Universal Joint 18
 3.7 Summary 20
4 SOFTWARE DESCRIPTION 21
4.1 Introduction 21
4.2 Arduino IDE 21
4.3 Robot Operating System 22
4.4 OpenCV 23
4.5 Summary 24
5 PROJECT DESIGN 25
5.1 Introduction 25
5.2 Mechanical Design 25
5.2.1 Ball and Plate Structure 25
5.2.2 CAD model of the system 25
5.3 Electronics Design 26
5.3.1 Arduino mega development board 26
5.3.2 MG995 Servo motor 27
5.3.3 Microsoft Lifecam HD 27
5.4 Software Design 28
5.4.1 PID Controller Design 30
5.5 Summary 31
6. RESULTS 32
7 CONCLUSIONS 33
7.1 Conclusion 33
7.2 Future Scope of work 33

REFERENCES 35
APPENDIX-1 36
APPENDIX-2 38
 List of Abbreviations

ROS – Robot Operating System

OpenCV – Open source Computer Vision

DOF – Degrees of Freedom

PID – Proportional Integral Derivative

IDE - Integrated Development Environment

DC - Direct Current

PWM - Pulse Width Modulation

LiPo - Lithium Polymer

USB - Universal Serial Bus

HSV - Hue Saturation Value

CAD – Computer Aided Design

ii
 List of Symbols

mb= mass of the ball

x = ball displacement

Fx,r = force from ball’s inertia

Fx,t= translational force generated by gravity

rb= radius of the ball

てb= torque produced

ϔb= ball angle

α = angle the plate makes with x-axis

θ = servo shaft angle

h = height of the beam

Lplate= length of the plate

g = acceleration due to gravity = 9.8 * 10-2ms-2

rarm= radius of servo motor

iii
 List of Figures

Figure No Title Page No

1.1 Flow chart of Working strategy 3
1.2 Quanser 2 DOF ball balancer 4
2.1 Schematic diagram of the ball balancer system 7
2.2 Block diagram of the ball balancing system with PID 7
2.3 Response of the servo motor 10
2.4 Response of the position of the ball 10
2.5 Response of the servo voltage 11
3.1 Arduino mega 13
3.2 Arduino mega pinout 13
3.3 PWM(duty cycle) 14
3.4 Servo motor PWM timing diagram 14
3.5 Servo motor MG995pinout 15
3.6 PWM period in the servo motor 15
3.7 Servo motor MG995 16
3.8 Microsoft lifecam 720p 17
3.9 Lithium Polymer Battery 17
3.10 Universal Joint 19
3.11 Hardware of the working model 19
4.1 Arduino 22
4.2 Robot Operating System 23
4.3 Open CV 24
5.1 Side View of the system 26
5.2 Front view of the system 26
5.3 Isometric View of the system 26
5.4 Electronics schematic diagram of the system 27
5.5 Flowchart of ROS nodes used in the system 28
5.6 ROS node graph 29

iv
5.7 Thresholded image of the ball 29
5.8 Servo angles 30
5.9 Diagram of closed loop plate-ball system 30
6.1 Response of the servo angle 32

v
 List of Tables

Table No. Title Page No.

3.1 Servo motor MG995 specifications 16

3.2 Lithium Polymer battery specifications 18

vi
 CHAPTER 1
INTRODUCTION

The ball-plate system is a nonlinear and unstable open-loop system. This system has 2
degrees of freedom (DOF). The objective is to balance the ball in a predefined
coordinate or to follow a specific centre with falling of the horizontal plate. The ball
position will be controlled indirectly by tilting the angle of the plate in a 2D direction,
x-axis and y-axis, and is controlled by two DC servo motors.

1.1 INTRODUCTION

Digital image processing is now widely popular and growing field where it’s used in
medicine, military, video production, security, tracking objects, and remote sensing.
Generally, the examination of moving objects, regardless of whether they are fire
fronts, particles, beads, or liquid interfaces, was done physically, more often than not
by measuring elements of a picture anticipated on a divider. This manual
investigation was dreary and experienced numerous inadequacies, including poor
precision and poor repeatability. One of the advantages of machine vision is the
ability of sensing moving objects from distance and without contact with the object.
However, the data in the image can be easily affected by the disturbance and changes
like sunlight and night that occur in the surrounding area of the object being sensed.
Such an issue can be overcome by using digital image processing and advanced
image filtering techniques to get rid of such noises and disturbances. The balancing of
objects is a major issue which is still under study and development especially by
robots manufacturers. The goal of this project is to develop a ball-plate balancing
system where the ball is sensed using a camera and not a resistive touch screen which
is expensive.

1
The camera will capture continuous frames that will be processed using OpenCV in
order to get the coordinates of the ball. The DC servo motors will be controlled by
ARDUINO. Using the middleware ROS, all the data are communicated between
nodes through messages.

Characterizing ball-plate system specifications is difficult since the controller is not

directly affecting ball and thus the designing of the controller and calculating its
numerical value will be difficult. Nowadays the ball and plate system has been
controlled by many control methods such as PID control, fuzzy control and so on.
However, the Proportional Integral Derivative (PID) controller proved its efficiency
in the world of controls and in the industrial field.

The feedback controller (PID) will decrease the errors in controlling the ball’s
position and trajectory. It will also control the overshoot resulted from the jump in the
reference signal and thus controlling the large input to the servo motors in order to
keep the ball on the plate. In this project, an optimal PID controller will be designed
with quick rise time, rapid settling, and negligible steady-state error.

Ball and beam and its two dimensional version, that is ball on balancing plate, have
many practical applications in real world control problems such as passenger’s
platform balancing in luxury cars, the control of exothermic chemical process
reactions where heat increase the process, control of rocket toppling control system
and aircraft vertical take–off and airplane horizontally stabilizing during landing and
in turbulent airflow and moving liquid container on road etc. Ball and beam system
having two degree of freedom and due to inherent nonlinearity it is a challenging
control problem.

2
1.2 AIM

The aim of this project is to implement the ball-plate system with visual position
feedback and PID controller so that the ball can be balanced at the centre of the plate.
The objectives of the project are:
● To apply optimal control method such as proportional integral derivative
control to control the ball-plate system
● To develop the image processing system using OpenCV.
● To implement servo motors control using ARDUINO.
● To coordinate these data and information using a middleware ROS.

1.3 WORKING

Modelling, control design and simulations were done in the 2 DOF ball balancer of
Quanser inc. But the system was not working properly for implementing the control
algorithm.Thus, a new ball balancer system was designed, fabricated and tested.
Figure 1.2 shows the image of a 2 DOF ball balancer system.

Figure 1.1: Flow chart of Working Strategy

3
The ball balancer has 2 degrees of freedom which are achieved by using servo
motors. An overhead camera is used for taking the feedback. Here the feedback is the
ball’s position. By image processing technique using OpenCV, this position of the
ball can be found out. By assigning the centre coordinates as reference point, the error
in the ball’s position can be measured. This error will be minimized by the PID
controller by adjusting the servo angles. The PID controller calculates how each servo
has to rotate (X-axis servo and Y-axis servo), so as to minimize the error and will
send this information to the ARDUINO MEGA board which actuates the servo to
achieve the required angle.

Figure 1.2 : Quanser 2 DOF ball balancer

The transfer of data like- an image from the camera, angle of the servos from the PID
controller to ARDUINO is carried out through the middleware Robot Operating
System. The flow chart is given in figure 1.1. Quanser 2 DOF ball balancer is showed
in figure 1.2.

4
 CHAPTER 2
BALL AND PLATE SYSTEM

2.1 INTRODUCTION

The Ball and plate system is a nonlinear system in which a ball has to be balanced on
a plate. Quanser 2 DOF ball balancer is a ball and plate system in which control
algorithms can be implemented for balancing the ball on the plate. Modelling and
simulations have been done in the system by implementing a PID control algorithm.

2.2 MECHANICAL DESIGN

2.2.1 Modelling
Consider figure 2.1, it shows the diagram and the force acting on the ball
Based on Newton's First Law of Motion, the sum of forces acting on the ball along
the beam equals:
mbẍ(t)=ΣF=Fx,t-Fx,r (2.1)
where mb= the mass of the ball, x= the ball displacement, Fx,r is the force from the
ball's inertia, and Fx,t is the translational force generated by gravity.
Friction and viscous damping are neglected. The force caused by gravity is as
follows:
Fx,t=mbgsinα(t) (2.2)
The force caused by the rotation of the ball is:

(2.3)
Where rb = the radius of the ball and τb =the torque which equals τb = Jb ϔb(t) where
ϒb is the ball angle.

5
Using the sector formula, x(t) = ϔb rb, we can convert between linear and angular
displacement. Then, the force acting on the ball in the x direction from its momentum
becomes:

(2.4)
Now, by substituting the rotational and translational forces, we get the nonlinear
equation of motion for the ball and beam as:

(2.5)
Solving for the linear acceleration gives:

(2.6)
Now we need to find the relationship between the servo load gear angle, l, and the
beam angle. Using the schematic diagram, consider the beam and servo angles
required to change the height of the beam by h. Taking the sine of the beam angle
gives the expression:

(2.7)
and taking the sine of the servo load shaft angle results in the equation:

(2.8)
From these we can obtain the following relationship between the beam and servo
angle:

(2.9)
Thus,

(2.10)
About angle zero, the sine function can be approximated by:

6
 (2.11)
Applying this to the nonlinear EOM we get:

(2.12)
This gives the linear equation of motion of the ball.

Figure 2.1: Schematic diagram of ball balancer system

2.3 CONTROL DESIGN

The control that will be used for each axis of the 2 DOF Ball Balancer system is
illustrated by the following block diagram:

Figure 2.2: Block Diagram of ball balancer system

7
To design the ball position control, assume that the inner servo loop is ideal and
therefore:

(2.13)

As shown in the block diagram, the outer-loop is a PV control

(2.14)

Thus ,the 2 DOF Ball Balancer closed-loop transfer function is:

(2.15)

Kp , Kd and Ki are the proportional, derivative and integral gains, respectively. We can
compute the values for these meet the settling time and overshoot specifications.

The characteristic equation of a 3rd order system is:

(2.16)

For a given overshoot specification, the damping ratio and damping frequency can be
calculated as :

(2.17)

And

8
 (2.18)
Solving for PID control gains gives,

(2.19)

(2.20)

(2.21)

2.4 SIMULATION RESULTS

Simulations were done using MATLAB-Simulink for the Quanser 2 DOF ball
balancer system. Solving the obtained transfer function in equation 2.15,
corresponding PID controller coefficients’ values were calculated. The control design
was for 10% overshoot and 4s settling time. The result of the simulation was 10.64%
overshoot and 4.2s settling time. The theoretical Kp, Kd and Ki values are 5.83
rad/m, 2.91 rads/m and 3.69 rad/ms respectively. These values were further tuned for
the ball to achieve stability and thus, Kp, Kd and Ki values were observed to be 7
rad/m, 3.6 rads/m and 6 rad/sm respectively. This change in the calculated and
observed values is because in simulation the weight of the plate is taken as negligible.
Figure 2.3 shows the response of servo, figure 2.4 shows the response of the position
of ball and figure 2.5 shows the response of servo voltage when an impulse is given.
The voltage rating of servo is 5 V. We can see that the simulated result is reaching to
the rated voltage of servo.

9
 Figure 2.3 : Response of servo angle

Figure 2.4 : Response of the position of ball

10
 Figure 2.5 : Response of servo voltage
2.5 SUMMARY

Modelling, control design and simulation were done for the Quanser 2 DOF ball
balancer. The model and transfer functions were available making it convenient to do
simulation. We also planned to implement the control algorithm in that system itself
but due to some problem with the rotary servo, it was not possible to implement the
PID control in the Quanser Ball and Plate System. Thus, a new system was made for
implementing the PID control algorithm. The details of the design, fabrication,
operation and control of the system are given in the upcoming chapters.

The Quanser system model consists of a Data Acquisition Box, a system unit, 2 rotary
servos, rotary encoders and a camera. But in the new system model, a ball and plate
system, a microcontroller, two 1800 servos and a camera are needed for implementing
the control algorithm.

11
 CHAPTER 3
HARDWARE DESCRIPTION

3.1 INTRODUCTION

The ball and plate system consists of five major hardware components, namely
ARDUINO (MEGA), servo motor, Microsoft Lifecam, Lithium polymer battery and
a universal joint.The ARDUINO (MEGA) is an open-source electronic platform
based on easy-to-use hardware and software which can be programmed easily by
connecting it to a computer via a USB cable and the ARDUINO (IDE).This board
offers digital and analog inputs and outputs as well as PWM output.The servo motor
controls the orientation of the plate.Its position is defined by the width of duty cycle
of the PWM pulses arriving from Arduino.The Microsoft Lifecam is used to obtain
the visual feedback.The Lithium Polymer battery is used as the power source while
the universal joint facilitates the movement of the plate along the X and Y axes

3.2 ARDUINO (MEGA)

Arduino is an open-source firm based on a hardware and software. It has been used in
thousands of educational and professional projects. It is a microcontroller based on
the Atmel ATmega328 which can operate at 5V. The microcontroller can be
programmed easily by connecting it to a computer via USB cable and using Arduino
software (IDE) which is built based on Java (Ladyada, 2014). This board offers
digital and analog inputs and outputs as well as PWM output. Figure 3.1 shows the
third generation of Arduino Mega and figure 3.2 shows its pinout diagram.

12
 Figure 3.1: Arduino Mega

Figure 3.2: Arduino Mega pinout

13
3.3. SERVO MOTOR

3.3.1. PWM
Pulse-width modulation (PWM) is a technique used to control the amplitude of the
digital signal in order to control AC/DC motors or even used in the communication
field. It controls the power of the voltage components by cycling the on and off
phases of a signal quickly and thus varying the value of the duty cycle. As a motor
this input will be a constant voltage since the 20 square waves are varying quickly.
PWM contribute in minimizing the loss in power in the circuit. A square waveform of
the PWM duty cycle can be shown in figure 3.3.

.
Figure 3.3: PWM (duty cycle)

For better comprehension of PWM, these diagrammatic portrayals can be utilized.

Figure 3.4 shows the waveforms as a result of the yield at various voltage
requirements.

Figure 3.4: Servo motor PWM timing diagram

14
3.3.2. Servo Motor (MG 995) Specifications
MG995 is a digital metal gear high torque servo for airplane, helicopter, RC-cars
from 10 to 6-th scale truggy and monster and many RC model
Its position is defined by the width of duty cycle of the PWM pulses arriving from
Arduino which are shown in Figure 2.5 and Figure 2.6.
This will redirect the mechanical structure towards the target.

Figure 3.5: Servo motor MG995 pinout

Figure 3.6: PWM period in servo motor

15
 Table 3.1: Servo motor MG995 specifications

Weight 1.94 oz (55.0 g)

Dimensions Length: 1.60 in (40.7 mm)

Width: 0.78 in (19.7 mm)
Height: 1.69 in (42.9 mm)

Torque 4.8V: 130.54 oz-in (9.40 kg-cm)

6.0V: 152.76 oz-in (11.00 kg-cm)

Speed 4.8V: 0.20 sec/60°

6.0V: 0.16 sec/60°

Voltage 4.8V~ 7.2V

Dead band width 4 μsec

Figure 3.7: Servo motor MG995

3.4. CAMERA

Microsoft Lifecam 720p has been chosen because of its features namely:

● CMOS Sensor Technology

● 1280 x 720 Resolution
● Up to 30 Frames per Second

16
 ● 73° Diagonal Field of View

Figure 3.8 Microsoft lifecam 720p

3.5. LITHIUM POLYMER BATTERY

Orange 1000mAh 3S 30C/60C Lithium polymer battery Pack (LiPo) batteries are
known for performance, reliability and price. It’s no surprise to us that Orange
Lithium polymer packs are the go-to pack for those in the know. Orange 1000mAh
3S 30C/60C Lithium polymer battery Pack (LiPo) batteries deliver the full rated
capacity at a price everyone can afford. Orange 1000mAh 3S 30C/60C Lithium
polymer battery Pack (LiPo) batteries are equipped with heavy duty discharge leads
to minimize resistance and sustain high current loads. Orange batteries stand up to the
punishing extremes of aerobatic flight and RC vehicles. Each pack is equipped with
gold plated connectors and JST-XH style balance connectors. All Orange Lithium
Polymer batteries packs are assembled using IR matched cells. Table 3.2 shows the
specifications of the LiPo that is being used.

Figure 3.9: Lithium Polymer battery

17
 Table 3.2: Lithium Polymer battery specifications

Product Type Lithium Polymer Battery Pack

Model No Orange 1000mAh 30C 3S1P

Capacity 1000mAh

Weight 75.0g

Dimensions 19.5*34*72(mm)

Voltage 11.1V

Discharge Plug XT-60

Max Continuous Discharge 30C(30A)

Charge Rate 1-3C Recommended, 5C Max

3.6 UNIVERSAL JOINT

A universal joint is a joint or coupling connecting rigid rods whose axes are inclined
to each other, and is commonly used in shafts that transmit rotary motion. It consists
of a pair of hinges located close together, oriented at 90° to each other, connected by
a cross shaft. The universal joint is not a constant-velocity joint. Here the U joint is
used for facilitating the motion of the plate in both the axes namely X-axis and Y
axis. Figure 3.10 shows an image of a universal joint.

The main concept of the universal joint is based on the design of gimbals, which have
been in use since antiquity. One anticipation of the universal joint was its use by the
ancient Greeks on ballistae. In Europe the universal joint is often called the Cardano

18
joint or Cardan shaft, after the Italian mathematician Gerolamo Cardano; however, in
his writings, he mentioned only gimbal mountings, not universal joints.

Figure 3.10: Universal Joint

Figure 3.11 : Hardware of the working model.

19
3.7 SUMMARY

Due to the damaged rotary servo in the Quanser system, PID control could not be
implemented in it. Thus a new system was developed and the control algorithm was
implemented.

This project uses the ARDUINO (MEGA) board instead of ARDUINO (UNO) due to
the amount of dynamic memory it provides. The servo motors draw an unusually
large amount of current (~700mA) while using a 9V battery, hence a Lithium
Polymer battery of 11.1V is used to overcome this problem. The ball and socket joint
produces more error than a universal joint and hence the latter is preferred

20
 CHAPTER 4
SOFTWARE DESCRIPTION

4.1 INTRODUCTION
The working of the ball and plate system employs three different software, namely
the ARDUINO IDE which is an user interface which allows the user to write a code
and upload it onto the arduino mega board, the middleware Robot Operating System
(ROS) which ensures the smooth communication of messages between nodes and
OPENCV which is a library of programming functions mainly aimed at real-time
computer vision.

4.2 ARDUINO IDE

The open-source Arduino Software (IDE) makes it easy to write code and upload it to
the board. It runs on Windows, Mac OS X, and Linux. The environment is written in
Java and based on Processing and other open-source software.

Arduino IDE is an open source computer hardware and software company, project,
and user community that designs and manufactures single-board microcontrollers and
microcontroller kits for building digital devices and interactive objects that can sense
and control objects in the physical and digital world. The project's products are
distributed as open-source hardware and software, which are licensed under the GNU
Lesser General Public License (LGPL) or the GNU General Public License (GPL),
permitting the manufacture of Arduino boards and software distribution by anyone.
Arduino boards are available commercially in preassembled form, or as do-it-
yourself (DIY) kits. Arduino board designs use a variety of microprocessors and
controllers. The boards are equipped with sets of digital and analog input/output (I/O)

21
pins that may be interfaced to various expansion boards or Breadboards(shields) and
other circuits. The boards feature serial communications interfaces, including
Universal Serial Bus(USB) on some models, which are also used for loading
programs from personal computers. The microcontrollers are typically programmed
using a dialect of features from the programming languages C and C++. In addition to
using traditional compiler toolchains, the Arduino project provides an integrated
development environment (IDE) based on the Processing language project. Figure 4.1
shows the logo of Arduino.

Figure 4.1: Arduino

4.3 ROBOT OPERATING SYSTEM (ROS)

Robot Operating System (ROS) is robotics middleware that is the collection of

software frameworks for robot software development. Although ROS is not an
operating system, it provides services designed for a heterogeneous computer cluster
such as hardware abstraction, low-level device control, implementation of commonly
used functionality, message-passing between processes, and package management.
Running sets of ROS-based processes are represented in a graph architecture where
processing takes place in nodes that may receive, post and multiplex sensor, control,
state, planning, actuator, and other messages. Despite the importance of reactivity and
low latency in robot control, ROS itself is not a real-time OS (RTOS). It is possible,
however, to integrate ROS with real-time code.

Both the language-independent tools and the main client libraries (C++, Python, and
Lisp) are released under the terms of the BSD license, and as such are open source

22
software and free for both commercial and research use. The majority of other
packages are licensed under a variety of open source licenses. These other packages
implement commonly used functionality and applications such as hardware drivers,
robot models, data types, planning, perception, simultaneous localization and
mapping, simulation tools, and other algorithms.

Anything connected to the ROS MASTER is known as a node. Mainly there are two
types of node- a publisher which publishes data as messages and a subscriber which
subscribes the data. Each message is referred using its topic name. Figure 4.2
illustrates the coordination by ROS.

Figure 4.2: Robot Operating System

4.4 OPENCV

OpenCV (Open Source Computer Vision Library) is an open source computer vision
and machine learning software library. OpenCV was built to provide a common
infrastructure for computer vision applications and to accelerate the use of machine
perception in the commercial products. Being a BSD-licensed product, OpenCV
makes it easy for businesses to utilize and modify the code.

23
The library has more than 2500 optimized algorithms, which includes a
comprehensive set of both classic and state-of-the-art computer vision and machine
learning algorithms. These algorithms can be used to detect and recognize faces,
identify objects, classify human actions in videos, track camera movements, track
moving objects, extract 3D models of objects, produce 3D point clouds from stereo
cameras, stitch images together to produce a high resolution image of an entire scene,
find similar images from an image database, remove red eyes from images taken
using flash, follow eye movements, recognize scenery and establish markers to
overlay it with augmented reality, etc. OpenCV has more than 47 thousand people of
user community and estimated number of downloads exceeding 14 million. The
library is used extensively in companies, research groups and by governmental
bodies. Figure 4.3 shows the main functions of openCV.

Figure 4.3: Open CV

4.5 SUMMARY

All the software employed in this project is open-source and free for commercial and
research purposes. Furthermore, the codes written in ARDUINO IDE, ROS and
OpenCV uses high-level languages (Python/C/C++) which makes it easier for the
user to comprehend the code and to edit it to suit the user’s needs.

24
 CHAPTER 5
PROJECT DESIGN

5.1 INTRODUCTION

The design part of this project mainly consists of its Mechanical, Electronics and
Software Design. The mechanical structure consists of two PVC pipes over which the
overhead camera rests. The plate is mounted on a universal joint and the two servo
motors are connected to its either sides. The electronic circuitry of this project was
implemented using Arduino Mega development board, two servo motors and an
overhead camera. The main softwares used were OpenCV, ROS and ARDUINO IDE.

5.2 MECHANICAL DESIGN

5.2.1 Ball and plate structure

The base of ball and plate system is made up of wooden plank which acts as the base.
Two PVC pipes are connected using a joint and fixed to the wooden plank for
mounting the overhead camera. A rectangular plywood is used for making the plate
which is fixed on a universal joint. Universal joint is used for facilitating the motion
of plate in X and Y axes. The servos are placed underneath the middle of plate edge.
The rotor of servo motors are connected to the plate using a firm material.

5.2.2 CAD Model of the system

CAD model of the system were made in solidworks for making manufacturing easy.
Figure 5.1 shows the side view of the system, figure 5.2 shows the front view of the
system and figure 5.3 shows the isometric view of the system.

25
 Figure 5.1: Side View of the System Figure 5.2: Front view of the System

Figure 5.3: Isometric view of the System

5.3 ELECTRONICS DESIGN

5.3.1 Arduino mega development board

The microcontroller used in Arduino mega is atmega328p. It acts as the centre of
actuation of servos. ROS and arduino are interfaced using rosserial_arduino making it
the subscriber node. Two servos are connected to the microcontroller which
subscribe the values of angles that are to be actuated by X-axis servo and Y-axis
servo. Those values will be written to the servo so that it rotates to the required angle

26
Arduino mega is used because it is has more dynamic memory when compared to
Arduino UNO. Since ROS utilizes more dynamic memory, it may leads to sync error
in development board which has less dynamic memory.

5.3.2 MG995 Servo motor

Two servo motors are used in this system fir controlling the motion of the ball in X
and Y axis. The base angle is set to 75o. Both of them are actuated by the Arduino
mega.

5.3.3 Microsoft Lifecam HD

An overhead camera is used for giving visual feedback for getting the position
coordinate of the ball.

Figure 5.4 Electronics schematic diagram of the system

27
5.4 SOFTWARE DESIGN

The main softwares used in this system are openCV, ROS and Arduino IDE. OpenCV
and ROS are a set of libraries which is coded in C++ language. OpenCV is used for
image processing application and ROS is used as a middleware for the
communication between nodes. OpenCV and ROS are interfaced using CVBRIDGE.

Here, mainly there are three nodes:

• usb_cam node - publishes the images taken by the overhead camera.

• ball_follower node - subscribes the images published by the usb_cam node
and does the required processing publishing the angle to which the servo
rotates for minimizing the error.
• Arduino – angles published by the ball_follower node is subscribed by the
arduino node and will actuate the servo accordingly. Figure 5.5 depicts the
ROS nodes used in the system. Figure 5.6 shows the ROS node graph which
depicts all the nodes and topics which are being subscribed or published.

Figure 5.5: Flowchart of ROS nodes used in the system

28
 Figure 5.6: ROS node graph

The images subscribed by ball_follower node are in RGB colour space. For
convenience, it is converted in to HSV colour space. Then the image is thresholded so
as to isolate the ball from the surrounding by converting it into a binary image. Figure
5.7 shows the thresholded image of ball. Then using moment method, the centre
coordinates of the ball can be found out. The middle point of plate is taken as the
reference point. The difference between this position of the ball and the reference
point is taken as error in the ball’s position which is fed to a PID loop. The output
from the PID loop is the angle of both the servos (X axis servo and Y axis servo) that
to be rotated to minimize the error. The angles are published by the ball_follower
node. Figure 5.8 shows the published angles. These angle values are subscribed by
the arduino node, which actuates the servo to attain the angle.

Figure 5.7 : Thresholded image of ball

29
 Figure 5.8 : Servo angles

5.4.1 PID Controller Design

The general PID control formula is as follows

(5.1)

Figure 5.9: Diagram of the closed loop plate-ball system

Figure 5.9 shows the closed loop feedback diagram.

The transfer function of the PID controller in Laplace domain is as follows

(5.2)
Where:
Kp : controller gives output which is proportional to current error e(t).

30
Ki: controller provides necessary action to eliminate the steady state error.
Kd : controller anticipates future behavior of the error. Its output depends on rate of
change of error with respect to time, multiplied by derivative constant.

5.5 SUMMARY
The mechanical structure consists of two PVC pipes over which the overhead camera
rests. The plate is mounted on a universal joint and the two servo motors are
connected to its either sides. OpenCv and ROS are a set of libraries. OpenCV is used
for image processing application and ROS is used as a middleware for the
communication between nodes.

31
 CHAPTER 6
RESULTS

The PID controller designed for the system helped the ball to be balanced at the
centre of the plate.
The total size of the frame captured by the camera is 320 pixels. Considering the
centre of the plate as the reference point, the maximum error in the position of the
ball is 160pixels. Setting the maximum angle that the servo can rotate to be 120
degrees, we get out coefficient to be 0.75. This was tuned further when
implemented on the ball-plate system for its stability. There are some disturbances in
the plate due to the vibration of servo motors. Among PID controller coefficients, Ki
has very small value making the system a PD controlled system. Overshoot and
settling time are minimized. Figure 6.1 shows the response of servo angles when the
ball is going to achieve its stable position.

Figure 6.1 : Response of servo angle

32
 CHAPTER 7
CONCLUSIONS

7.1 CONCLUSION

In this thesis, the aim was to control the position of the ball on the plate for the centre
of the plate. Instead of using electrical sensors such as resistive touch panel which is
more expensive than camera, so we have used vision sensing to track the position of
the ball on the plate. This system consists mainly of three parts, one is image
processing in order to track the ball’s trajectory, the other one is to build PID
controller which controls the servo motors of the ball-plate system and finally the
smooth communication between nodes.

We had problems in this system such as removing noise from frames captures, and
rejecting disturbances in order to maintain a static ball position balancing. The noises
were removed from the images but disturbances are still there due to the vibration of
servo motors.

In this system, a new way of communication system or middleware was used when
compared to other ball and plate systems. Integration of information and data using
ROS is a novel way.

7.2 FUTURE SCOPE OF WORK

Ball and beam system is used for teaching, understanding control characteristic and
solving difficult control problems. For many unbalanced system, present in daily life,
where size and shape not suitable for study purpose in laboratory, ball and beam

33
system can be used to learn control mechanism. Some unbalanced systems are
exothermically preceding reactions, in power generation, and in aerospace. These
types of real unbalanced systems are most difficult and dangerous control problems
and cannot be brought in laboratory. So, ball and beam system gives an advanced role
in study of this type of real unbalanced system and suitable controller design for
industrial as well as academic purposes.

In this project, only PID control was used for minimizing the error. This can be used
as a testing platform for testing different control methods like fuzzy control, neural
integrated fuzzy control, and neural integrated PID controllers.

34
 REFERENCES
[1] Ahmed Itani, “Ball Plate Balancing System using Image Processing”, Master of
Science thesis in Mechatronics Engineering, Graduate School of Applied Sciences of
Near East University.
[2] Jason Christensen, James Humble Jay Mattinson “Ball-Balancing Platform Design
Documentation”, College of Engineering UtahState University
[3] Quanser Inc. 2013,”2 DOF Ball Balancer”, Students’ Manual pp 5-9.
[4] http://wiki.ros.org/ROS/Introduction (as on 22/04/2019).
[5] https://en.wikipedia.org/wiki/OpenCV (as on 22/04/2019).
[6] Ehsan Ali and Narong Aphiratsakun, “AU ball on plate balancing robot,” IEEE
International Conference on Robotics and Biomimetics (ROBIO), held at Zhuhai,
China, 6-9 December 2015.

35
 APPENDIX 1

FINAL WORKING MODEL

Figure A : Side view of working model

Figure B : Top view of working model

36
Figure C : Front view of working model

37
 APPENDIX 2

ROS AND OPENCV CODE

#include <ros/ros.h>
#include <image_transport/image_transport.h>
#include <opencv2/highgui/highgui.hpp>
#include <cv_bridge/cv_bridge.h>
#include <sensor_msgs/image_encodings.h>
#include <opencv2/imgproc/imgproc.hpp>
#include <sstream>
#include "geometry_msgs/Point32.h"
using namespace cv;
using namespace std;

double kp=0.25, kd=1.5 , ki=0, error_x, error_y, Output_x, Output_y, errSum_x,

errSum_y, lastErr_x, lastErr_y, base_x=75, base_y=75;

void Compute()
{
errSum_x += error_x;
errSum_y += error_y;
double dErr_x = (error_x - lastErr_x);
double dErr_y = (error_y - lastErr_y);
Output_x =base_x-(kp*error_x + ki*errSum_x + kd*dErr_x);
Output_y =base_y+(kp*error_y + ki*errSum_y + kd*dErr_y);
lastErr_x = error_x;
lastErr_y = error_y;

38
}

class ballfollower
{
ros::NodeHandle nh;
ros::Publisher cmd_pub;
image_transport::ImageTransport it;
image_transport::Subscriber image_sub;
image_transport::Publisher image_pub;

public:
ballfollower()
:it(nh)
{
image_sub=it.subscribe("usb_cam/image_raw",
1,&ballfollower::imageCallback,this);
image_pub = it.advertise("image/detected", 1);
cmd_pub = nh.advertise<geometry_msgs::Point32>("follow_cmd", 1000);
}
void imageCallback(const sensor_msgs::ImageConstPtr& msg)
{
Mat original_frame, hsv_frame, threshold_frame;
int x_centre,y_centre;
int posX, posY, iLastX, iLastY;
cv_bridge::CvImagePtr cv_ptr;
namespace enc = sensor_msgs::image_encodings;
try
{
cv_ptr = cv_bridge::toCvCopy(msg, sensor_msgs::image_encodings::BGR8);
}

39
catch (cv_bridge::Exception& e)
{
ROS_ERROR("cv_bridge exception: %s", e.what());
return;
}
original_frame = cv_bridge::toCvCopy(msg, "bgr8")->image;
cvtColor(original_frame, hsv_frame, COLOR_BGR2HSV);
inRange(hsv_frame, Scalar(0, 96, 182), Scalar(179, 255, 255), threshold_frame);

//morphological opening (removes small objects from the foreground)

erode(threshold_frame,threshold_frame, getStructuringElement(MORPH_ELLIPSE,
Size(5, 5)) );
dilate(threshold_frame,threshold_frame, getStructuringElement(MORPH_ELLIPSE,
Size(5, 5)) );
//morphological closing (removes small holes from the foreground)
dilate(threshold_frame,threshold_frame, getStructuringElement(MORPH_ELLIPSE,
Size(5, 5)) );
erode(threshold_frame,threshold_frame, getStructuringElement(MORPH_ELLIPSE,
Size(5, 5)) );

Moments oMoments = moments(threshold_frame);

double dM01 = oMoments.m01;
double dM10 = oMoments.m10;
double dArea = oMoments.m00;
x_centre = dM10 / dArea;
y_centre = dM01 / dArea;
posY = original_frame.rows / 2;
posX = original_frame.cols / 2;
geometry_msgs::Point32 message;
error_x=x_centre-posX;

40
error_y=y_centre-posY;
Compute();
message.x=Output_x;
message.y=Output_y;
cmd_pub.publish(message);
sensor_msgs::ImagePtr msge = cv_bridge::CvImage(std_msgs::Header(), "mono8",
threshold_frame).toImageMsg();
image_pub.publish(msge);
}
};

int main(int argc, char** argv)

{
ros::init(argc,argv,"ball_follower_ros");
class ballfollower bf;
ros::spin();
return 0;
}

41
ROS ARDUINO CODE

#include <ros.h>
#include <geometry_msgs/Point32.h>
#include <Servo.h>

ros::NodeHandle nh;
Servo myservo_x;
Servo myservo_y;
int pos_x, pos_y;
void messageCb( const geometry_msgs::Point32& msg)

pos_x=(int)msg.x;
pos_y=(int)msg.y;
myservo_x.write(pos_x);
myservo_y.write(pos_y);

ros::Subscriber<geometry_msgs::Point32> sub("follow_cmd", &messageCb );

void setup()

myservo_x.attach(9);
myservo_y.attach(8);
nh.initNode();

42
nh.subscribe(sub);

void loop()
{
nh.spinOnce();
delay(1);
}

43