International Journal of Swarm

lige
Intel nce an
rm d
Patil, Int J Swarm Intel Evol Comput 2017, 6:1
wa

Ev
Intelligence and Evolutionary
ournal of S

oluti
DOI: 10.4172/2090-4908.1000154
onary C
Computation
al J

ISSN: 2090-4908
om
on

pu
tati ati
on Intern

Research Article
Research Article Open Access

Robot Manipulator Control Using PLC with Position Based and Image Based
Algorithm
Madhav D Patil*
Department of Electrical and Computer Engineering, University of Bridgeport, USA

Abstract
Programmable Logic Controller is making big role in automation and robotics. This paper described about
the design and implementation of Artificial intelligence algorithms for controlling the robotic arm or manipulator
hardware platform using Programmable Logic Controller (PLC). The controlled robot is 5 degrees of freedom (DOF)
manipulator with a closed kinematic chain, designed for high-performance pick and place applications. The control
software is fully developed on a commercial PLC system, using its standard programming tools and the multi-tasking
features of its operating system. In particular, the paper analyse in detail the drawbacks and the advantages related
to the choice of standard PLCs in this kind of applications, compared to the much common choice of specialized
hardware or industrial personal computers, with particular emphasis on the computational performances obtained
with the proposed control architecture.

Keywords: Artificial intelligence; Vision system; Programmable 61131-3) allow to easily implement any kind of algorithm that can be
logic controller; Robot manipulator/Arm; Industrial automation used in robot control [5].

Introduction Moreover, even if most PLCs support only their proprietary IDE,
with that specific IDE it is always possible to upload the user software
Robot manipulators are largely used in packaging plants, especially from the controller and modify it, which is not possible on PC-based
for pick and place operations and box filling. In particular, packaging of systems whose software, is a compiled executable [6]. Therefore, PLC
small food products, like cookies or candies, requires high performance programs can be updated and extended by end-users even without
robots with short cycle times and precise motion control, even if their the original source code, which makes PLCs “open” systems. Despite
workspace is relatively small [1]. Such performances can be obtained this feature, PLC-based control systems for robot manipulators are
by means of lightweight parallel-drive or delta-like kinematics, whose quite rare and, in general, limited to simple Cartesian or gantry-like
advantages in terms of reduced moving masses and inertias are well- structures [7].
known. The robots include a vision system, in order to identify and
localize products transported in unordered manner and in a large A robot arm is the combination of links and joints in the form of
number on the conveyors. The robot controller is actually a quite closed a chain with one end is fixed while the other end is free. The joints are
system that customers are not able to extend without specific support either prismatic or revolute, driven by actuators. In order to move the
from the manufacturer. Since the integration of a robotic system with free end, also called the end effector, along a certain path, most, if not
additional, application-specific, tools and features often requires the all, of the joints are to be moved [8]. In doing so it is necessary to solve
development of software or hardware closely matched with the basic the inverse kinematics equation. In the case of redundant manipulator,
robot motion control system, customers often search for openness of inverse kinematics is much more difficult when compared to a non-
robot controllers. In fact, several open architectures for robot control redundant manipulator whose kinematics is not so complicated [7].
can be mentioned, either developed within academic projects or from There are traditional methods such as algebraic solutions, geometric
commercial robot manufacturer [2-4]. In most cases, these architectures solutions and iterative solutions in order to solve the inverse kinematics
take advantage of hardware platforms based on Personal Computers problem. However, these methods are time-consuming and suffer from
(PC). On one hand, the use of PCs for robot control reduces software numerical problems. Recently, the particle swarm optimization (PSO)
development costs, thanks to high level languages and well-designed has been successfully applied to various optimization problems. This
Integrated Development Environments (IDE). On the other hand, new optimization algorithm combines social psychology principles in
standard PCs do not have the reliability required by harsh industrial socio-cognition human agents and evolutionary computations [9].
environments, so that the use of more protective cases and more robust
electronic components actually increase the final cost of PC based robot In this paper, we aim to show that the capabilities of a commercial
controllers.
Programmable Logic Controllers (PLCs) are instead a kind of
control devices playing a dominant role in industrial automation, *Corresponding author: Patil MD, Department of Electrical and Computer
Engineering, University of Bridgeport, Bridgeport, CT, USA, Tel: 203-572-4551;
thanks to their robustness, reasonable cost and ease of use [1].
E-mail: mpatil@my.bridgeport.edu
Electronic components inside PLCs are selected for their reliability
and solid presence on the market, rather than for their performance. Received April 07, 2017; Accepted April 19, 2017; Published April 26, 2017
For this reasons, PLCs are commonly considered as low-level systems, Citation: Patil MD (2017) Robot Manipulator Control Using PLC with Position
whose main purpose is to control simple Boolean signals (i.e. discrete Based and Image Based Algorithm. Int J Swarm Intel Evol Comput 6: 154. doi:
10.4172/2090-4908.1000154
control and sequencing) and to supervise the safety and integrity
of plants and operators. However, modern PLCs have most of the Copyright: © 2017 Patil MD. This is an open-access article distributed under the
terms of the Creative Commons Attribution License, which permits unrestricted
times sufficient computational bandwidth to execute also complex use, distribution, and reproduction in any medium, provided the original author and
mathematical calculations and their programming languages (i.e. IEC source are credited.

Int J Swarm Intel Evol Comput, an open access journal

ISSN: 2090-4908 Volume 6 • Issue 1 • 1000154
Citation: Patil MD (2017) Robot Manipulator Control Using PLC with Position Based and Image Based Algorithm. Int J Swarm Intel Evol Comput 6:
154. doi: 10.4172/2090-4908.1000154

Page 2 of 8

PLC and its multi-tasking operating system are sufficiently powerful

to implement a robot control system including inverse kinematics, on-
line trajectory planning. The integration of vision systems with Robot
manipulator and PLC used for implementing Imaged Based Intelligence
algorithm also to find out the object or goal position for Position Based
algorithm. The particle swarm optimization is used for Position based
algorithm as it is characterized as a simple concept, which is both easy
to implement and computationally efficient [9,10]. The robotic platform
described in the paper is designed and developed in Programmable
Logic Controller laboratory at University of Bridgeport, CT, USA. The
rest of the paper is organized as follows: Sec. II describes the problem
definition; Sec. III gives some details about the solution, while Sec. IV
describes the conclusion and Sec. V gives the future scope.
Problem definition
A wave of an industrial automation given the profound impact Figure 1: Robot arm with joint angle.
on technologies and electronics leads to the revolution. Due to
this industrial revolution, technological advances now allow the
actions such as for open and close the gripper, take snap shot of parts,
manufacturer to invent and create new ideas, product in mass
start/stop the conveyor, start/stop the motor etc.
production with enhanced quality [11]. The most of industrial
processes uses manipulator arms for picking and placing the objects D. Transition Model
or products in close proximity of previously placed objects. The task
The transition model for the manipulator is depending on the action
assigned to the robot arms can be performed repeatedly with high
taken. At the start the manipulator returns to the homing position. So
accuracy and with precise movements of joint angles [12]. The common
we will consider the initial state as the homing position, but not all
industrial manipulator is often referred as robot arm with joints and
the time. The camera mounted on the end effector will continuously
angles as shown in figure 1. The robot arm used in our application is an
feed the current position coordinates of the robot manipulator to the
articulated arm consists of all revolute joints. The articulated robot arm
controller and the algorithm controls the motion of robot arm. The
has maximum flexibility and can reach over and under the objects. As
complete transition model is shown in figure (Figure 2).
all joints are revolute these robots can cover large workspace and joints
are easy to seal. The robot manipulators are assigned to accomplish E. Goal Test
the specific task in unstructured environment with minimal joint
movements and best shortest path (Figure 1). The goal test of this model is to pick and place the objects/parts
with respective area for further processing by detecting metal and non-
The robot motion path planning has been studied more than two metallic parts and avoiding the obstacle or collision of the robot arm
decades and there are so many contributions made on this problem with any other part of the system.
[13]. Deriving the best possible inverse kinematic solution for end
effector is challenging and difficult. Some of the challenging aspects in F. Path Cost
designing reverse kinematic solution are: The path cost of this application is varies according to the product
A. States Space cost, which company parts you are using for the system assembling, and
other factors.
As the manipulator is design for pick and place application, the
manipulator pick up the part and place it in relative positions such as The complete flow chart of the application designed and
good part position or bad part or rejection area according to the sensory implemented as given in figure (Figure 3).
data. So the states finding for the robot manipulator is finite. The arm To achieve the goal of the application, the position trajectories
has its work space and it can reach to those positions by different paths. calculation of robot manipulator is the most difficult task, as it has to
B. Initial State avoid the obstacle and reach to the product picking [14]. As stated in
above problem transition state, for this robot manipulator there are
The initial state for the robot manipulator could be any state lots of possible path to reach object. Among those paths, the best path
depends on the signal send by the sensors to the arm. But at the start of will have the minimal joint movements and shortest distance within
the program the manipulator always go to the home or nesting position. its workspace. There are lots of algorithms and techniques have been
developed for finding the inverse kinematics of the manipulator. The
C. Action
complexity of finding the kinematics solution increases with number of
The manipulator action is depend on the perceived signal of the joints or Degree Of Freedom (DOF). The robot manipulator position,
sensors (camera, part detect sensor, etc.). The manipulator takes action path planning and motion control in 3 dimensional workspace
after the signal send to the controller by sensors and controller decides become a key factor for control system design engineers and robot
the movement of arm. So manipulator moves toward the destination manufacturer. The robot arm should be self-proficient, flexible, low
area by avoiding the obstacles in the path as well as within the time limit power consumptions, fully efficient. The typical task for robot arm in
and with respective speed. To reach to the goal or object manipulator has industry is to move its end effector from initial position to some desired
to find out the best possible path with minimal joint angle movement position in working environment with least residual vibration, minimal
within its work space. The end effector will try to reach the object as torque, obstacle avoidance and collision free kinematics, shortest time
soon as x and y coordinates are calculated. There are also some other interval and/or distance of a desired path [15].

Int J Swarm Intel Evol Comput, an open access journal

ISSN: 2090-4908 Volume 6 • Issue 1 • 1000154
Citation: Patil MD (2017) Robot Manipulator Control Using PLC with Position Based and Image Based Algorithm. Int J Swarm Intel Evol Comput 6:
154. doi: 10.4172/2090-4908.1000154

Page 3 of 8

quantity depending on the distance between the target and the camera
Home which is difficult to evaluate. A control in the image plane results also
Position to be strongly nonlinear and coupled when mapped on the joint space
of the robot and may cause problems when crossing points which are
singular for the kinematics of the manipulator. 
The pseudo code for the position based algorithm is as given below,
Signal Signal not
received from received from For each position
controller Controller
{
Initialize position
Pick up Not pick }
part up part
Do until maximum iterations or minimum error criteria
{
For_each position
Bad Good
Part part {
Area Area
Calculate joint angle kinematics
If_ Joint1 better than Joint2
Metal Nonmetal
{
part part
Set pBest = current Joint angle
}
Start Pick up
conveyor part If pBest is better than gBest
{
1st
2 nd
Place at Set gBest = pBest
position position reject area
}
Figure 2: Transition model of the system.
}

Solution methodology }

Dealing with complex, higher level control system with continuous For each position
interactive subsystem in dynamic environment is difficult and requires {
sophisticated and intellectual controller with continuously process
optimization. We proposed the solution for above stated problem in Calculate joint Velocity
this section. The proposed solution has been applied and tested on Use gBest and Velocity to update position Data
robot manipulator with 5 DOF. The experimental set up is explained in
next section. In this experiment we are using position based algorithm }
in combination with Image based algorithm to finding the best possible The main purpose or goal of this study is to use the visual feedback
solution for our stated problem. As we are implementing Artificial from images captured by the camera and the Cartesian space co-
Intelligence in the robot arm control system based on PLC, we have to ordinates of the target object which ultimately controls the motion of
use search algorithms, inference engine etc. The system is completely the robot to perform a task. The starting position and coordinate frame
knowledge based as uses the sensors, camera and applies the action on boundary is taught to the robot arm. From the calculated co-ordinates
the robot arm and actuators. Solving the inverse kinematics becomes of the target object, the surface model in 3 dimensional co-ordinate
complex and nonlinear as the number of degrees of freedom of the systems will be constructed for robot manipulator. The sensor data
manipulator increases. The position based search algorithm uses the will be used for knowledge base and will be used to avoid collision as
visual data provided by the imaged based algorithm and calculates well as can be used to find optimal shortest possible path or trajectory
the joint velocity and angles to form the inverse kinematic solution by checking each point in its workspace. The sensors can give a signal
in 3 dimensional workspace [16]. By adding vision or imaged based when contact is made with obstacles, detect metal or non-metal objects,
algorithm, the robotic control system is more flexible, adaptable and or measure a force being applied. Due to this knowledge provided
increases the accuracy in the joints movement (Figure 4). by sensory units to the system, robot path can be planned before its
execution to the target position.
The principal advantage of using position-based control is the
chance of defining tasks in a standard Cartesian frame. On the contrary, Firstly, a camera is mounted on a manipulator end effector and
the image-based control is less sensitive to calibration errors; however, it catches a 2-D image, a true potential can be exactly calculated. We
it is required the online calculation of the image Jacobian, that is, a assign the two dimensional coordinate system with the x-axis and

Int J Swarm Intel Evol Comput, an open access journal

ISSN: 2090-4908 Volume 6 • Issue 1 • 1000154
Citation: Patil MD (2017) Robot Manipulator Control Using PLC with Position Based and Image Based Algorithm. Int J Swarm Intel Evol Comput 6:
154. doi: 10.4172/2090-4908.1000154

Page 4 of 8

Int J Swarm Intel Evol Comput, an open access journal

ISSN: 2090-4908 Volume 6 • Issue 1 • 1000154
Citation: Patil MD (2017) Robot Manipulator Control Using PLC with Position Based and Image Based Algorithm. Int J Swarm Intel Evol Comput 6:
154. doi: 10.4172/2090-4908.1000154

Page 5 of 8

Figure 3: Flow chart of the complete process.

Figure 5: Camera mount position.

Figure 4: Block diagram of PLC based robot arm control system.

In order to determine the position of the target object in the image
y-axis forming a basis for the image plane and the z-axis perpendicular plane, camera will capture the images and through which only the
to the image plane. The origin located at distance ʎ behind the image central point (only a single pixel) have to be identified, without knowing
plane as shown in figure (Figure 5). this point’s co-ordinates in the attached Cartesian reference system. This

Int J Swarm Intel Evol Comput, an open access journal

ISSN: 2090-4908 Volume 6 • Issue 1 • 1000154
Citation: Patil MD (2017) Robot Manipulator Control Using PLC with Position Based and Image Based Algorithm. Int J Swarm Intel Evol Comput 6:
154. doi: 10.4172/2090-4908.1000154

Page 6 of 8

coordinates will be feed to the Position Based Algorithm for calculating

the joints angle and trajectory. The position finding algorithm outputs
the x-coordinate and y-coordinate in the image frame along with a
scaling factor (ʎ). The scaling factor is related to the dimensions of the
gripper and is used to get an idea of the elevation of the gripper. This
helps in making the gripper co-planar with the target object.
Hardware setup
We implemented our proposed algorithms and optimal path
planning schemes on a PLC based Robot arm control system for
sorting, pick and place application as shown in Figure (Figure 6).
Following are the main components of the project:
1. Mitsubishi Fx3G Programmable Logic Controller.
2. Mitsubishi RV-M1 Mitsubishi Robot along with Movemaster
Drive Unit.
3. Mitsubishi touch screen E-1061 HMI (Human Machine
Interface).
4. VGA Camera.
5. Cognex Vision Camera unit.
6. A servomotor for controlling belt conveyor and another Figure 7: Nomenclature of robot arm.
servomotor for displacing a component as a kicker
7. Sensor system comprising of laser sensors, capacitive sensor,
track sensor and proximity.
8. Two pneumatic cylinders which act as component pusher.
The Mitsubishi RV-M1 robot is driven with robot drive unit
Movemaster EX from which also provides interface with Mitsubishi
PLC with help of 50 pin connector and another interface for robot teach
pendant. This drive unit stores the code for the robot. The nomenclature
of the robot arm is shown in figure (Figure 7).
The specification of Movemaster robot arm is as given in figure
(Figure 8).
The Programmable Logic Controller used in this experimental set
up is Mitsubishi Fx3G and its basic specification are given below,
Power Supply: 100-240 V AC/ 12-24 V DC, 50/60 Hz

Figure 8: Specification of robot arm.

No. of Outputs: 16.

No. of Inputs: 24.
Input form: sink / source with photocoupler isolation
Input / Output response: Approx 10 ms
Digital Output: Relay Transistor.
Program Cycle Period: 0.55-1 us
Memory: 32K steps EEPROM.
Software: GX Developer.
USB Port, Profibus or Ethernet Connection
Figure 6: PLC system setup.

Int J Swarm Intel Evol Comput, an open access journal

ISSN: 2090-4908 Volume 6 • Issue 1 • 1000154
Citation: Patil MD (2017) Robot Manipulator Control Using PLC with Position Based and Image Based Algorithm. Int J Swarm Intel Evol Comput 6:
154. doi: 10.4172/2090-4908.1000154

Page 7 of 8

Results
The application we used for testing these two algorithm with PLC
is Pick and Place application with Quality Inspection. As an initial step
we localize the object using a traditional Haar classifier. We trained
Mark is Mark is Not
multiple models for recognition various objects in the scene. At time present in Nut absent in Nut
of training process each Haar object detection model will be fed with
respective positive samples or in class samples and out of class samples.
We then evaluate the performance of the trained model using test data.
Good Part Bad Part
The processing is performed on a windows machine using Matlab
Software. We chose this method (Haar classifier) as it is one of the Figure 12: Job 4 test product images.

successful and simple object detection method used widely in machine

vision technologies. Some of the sample training images are shown in Start node Stack = empty
figures (Figure 9-12).
End node Stack = empty
The pseudo code for Experimental Setup is as given below-
All other variables initialized.
1. Initialize:
Load the mode into Start node Stack equivalent to sNODE out
degree each time incrementing the start index. Start index to point to
sNODE the Root, on top of stack. Now pop the top of stack and assign
to the Depth First Search (DFS) proceed to 2
2. Begin (At start node Get the unvisited adjacent)
Screw is present Screw is absent
while (there are bNODEs proceed with PBA and IBA algorithm and
generate Step code for the branch till you reach a tNODE or xNODE)
if ( xNODE proceed to 3)
Good Product Bad Product
else if ( tNODE go to 7)
Figure 9: Job 1 test product images.
else go to 10
3. At xNODE check node status;
if (we are meeting this xNODE for the first time) go to 4
else (top of end index points to this xNODE pop End node Stack
and go to 5)
New Optimal Guidance for Mountaineer Lifesaving Mission
4. Make this xNODE as an end node, if bad part is detected on
Cognex vision system Stack equivalent to its in degree each time
incrementing the end index. Then go to 5
Good Product Bad Product Bad Product 5. Process this xNODE and increment the in degree processing
Figure 10: Job 2 test product images. count for this node.
If (the in degree processing count equal to the xNODE in degree)
go to 6
else if (Start node Stack not empty) get the start node ID from stack
by popping Start node Stack assign ID to DFS then go to 2
else go to10.
6. Make this node a start node, put its ID on Start node Stack
equivalent to its out degree
if (Start node Stack not empty,) pop the Start node Stack assign ID
to DFS and then go to 2
else go to 10.
7. Process this tNODE then
if (Start node Stack is empty ) proceed to 8
Good Product Bad Product else pop the Start node Stack assign to DFS and then go to 2
Figure 11: Job 3 test product images. 8. if (End node Stack empty and statistic check correct) proceed to 9

Int J Swarm Intel Evol Comput, an open access journal

ISSN: 2090-4908 Volume 6 • Issue 1 • 1000154
Citation: Patil MD (2017) Robot Manipulator Control Using PLC with Position Based and Image Based Algorithm. Int J Swarm Intel Evol Comput 6:
154. doi: 10.4172/2090-4908.1000154

Page 8 of 8

else go to 10 based algorithm in combination with Image based algorithm. The

implementation is carried out on Movemaster Robot Arm with PLC for
9. Good metal product successfully processed to packaging/loading
sorting metal and non-metal objects, pick and place application. The
area
proposed combination algorithm reduces the computation time and
exit (Normal) positioning error for finding the target in real time.
10. Bad non-metal product. Encountered an error, send to rejection References
area. 1. Patil M, Sheth A, Toporovsky PJ (2011) Implementation of a robotic conveyor
assembly line with vision camera using two different coding techniques and
Exit comparison of their performance parameters. In proc 2011 ASEE Northeast
Section Annual Conference.
Following are the test images that were used as a target image to
2. AB AAT (2006) IRB 340 datasheet.
evaluate proposed methodology. We assigned 4 different jobs. Each job
has different products with unique features to distinguish whether it is 3. The orocos project: Open robot control software.
good or bad product. 4. Macchelli A, Melchiorri C (2002) A real-time control system for industrial robots
and control applications based on real-time Linux. In 15th IFAC world congress,
JOB 1 Barcelona, Spain.

We used relay for job 1, it will check whether screw in the socket 5. Asada H, Youcef-Toumi K (1983) Analysis and design of semidirect drive robot
arms. In: Proc Amer C Conf, San Francisco, CA.
is present or not. If cognex checker detects screw on the right position
then it will consider it as good part and send it to accept position else it 6. Kao I, Gong C (1997) Robot based computer integrated manufacturing as
will send it to reject position (Figure 9). applied in manufacturing automation. Robot and Comput Int Manuf 13: 157-
167.
JOB 2 7. Bonfe M, Vignali M, Fiorini M (2009) PLC-based control of a robot manipulator
with closed kinematic chain. In Proc IEEE Int Conf Robotics and Automation.
We used holder for job 2, checker will check whether holding
pins are assembled in holder during manufacturing process or not. If 8. Whitney DE, Lozinski CA, Rourke JM (1986) Industrial robot forward calibration
method and results. J Dyn Sys Meas Control 108: 1-8.
all holding pins are present in holder then checker will consider it as
good product else it will consider it as bad product and send the good 9. Rini DP, Shamsuddin SM, Yuhaniz SS (2011) Particle swarm optimization:
Technique, system and challenges. Int J Comput Appl 14: 19.
product to accept position and bad product to reject position (Figure
10). 10. Hashimoto K, Kimoto T, Ebine T, Kimura H (1991) Manipulator control with
image-based visual servo. In Proc IEEE Int Conf Robotics and Automation.
JOB 3 11. Patil M, Toporovsky Y, Lawrence H, Sobh T (2012) Integration of vision system,
intelligent Robo Actuator, HMI and PLC to design a universal quality inspection
Purell hand sanitizer is used in job 3. In this job we check the label or control machine. I-managers J Mech Eng 2:17.
of Purell brand name in the hand sanitizer container. Container with
Purell brand name is considered as a good product and without Purell 12. Khatib O, Quinlan S, Williams D (1997) Robot planning and control. Robot
Auton Syst 21: 249-261.
brand name is considered as a bad product (Figure 11).
13. Glavina B (1990) Solving findpath by combination of goal-directed and
JOB 4 randomized search. In Proc IEEE Int Conf Robotics and Automation. Pp: 1718-
1723.
We used nut for job 4. Nut with two marks is considered as good
14. Feng-Yih H, Li-Chen F (1995) A new design of adaptive fuzzy hybrid force/
product and nut without marks will be considered as a bad part (Figure position controller for robot manipulators. In Proc IEEE Int Conf Robotics and
12). Automation 1:863-868.

15. Ogunlowore O (2013) Realtime motion planning for manipulator robots

Conclusion under dynamic environments: An optimal control approach. In mechanical
engineering, University of Waterloo, Canada.
In this paper we proposed an algorithms and implementation
method for calculating the inverse kinematic solution and trajectory 16. Palmieri G, Palpacelli M, Battistelli M, Callegar M (2012) A comparison between
planning for industrial robot manipulator using the Position position-based and image-based dynamic visual servoings in the control of a
translating parallel manipulator. J Robotics.

Int J Swarm Intel Evol Comput, an open access journal

ISSN: 2090-4908 Volume 6 • Issue 1 • 1000154