Image Processing Based Robotics PDF
Image Processing Based Robotics PDF
IT-BHU
VARANASI
karan.bajaj.cse09@itbhu.ac.in | karanhindustan.blogspot.com
Karan Bajaj
Getting Started
Image Processing comprises of any form of signal processing for which the input is an image. Not very surprising! How do we use image processing in Robotics? The Robot we would be using has an eye. Now where is this eye located? Its generally a webcam that may be permanently attached at some specific location, or attached to the Robot itself. This webcam inputs images, which are sent to the computer via the webcam connecting wire. The concept of Image processing now comes in, and the acquired image is processed using MATLAB programming (introduced later in the tutorial). Now we need to inform the Robot about the processed details. For this, we use a port of the computer, which sends information from the computer to the Robot. The Robot contains a microcontroller which would decode the information send via the port. This information is worked upon, and signals are sent to the motors. We would work on understanding these aspects STEP by STEP as follows 1. Image acquisition Image taken in from webcam to be brought into the environment (MATLAB) where it will be processed 2. Image processing Processing of the image using the environment (MATLAB) 3. Using the port Interfacing using Serial ports 4. Motor control Controlling DC motors and Stepper motors
2
robotics.cca@itbhu.ac.in
Karan Bajaj
MATLAB
MATLAB stands for Matrix Laboratory. It is a numerical computation environment and 4th generation programming language that uses matrices to do hell lot of things! As I mentioned we would be using MATLAB, it makes sense that the work of processing the Image would be done in MATLAB. MATLAB uses matrices to store multimedia entities. Hence, our images would also be stored as matrices.
robotics.cca@itbhu.ac.in
Karan Bajaj
MATLAB environment
We would be using MATLAB 7.8.0 (R2009a) for the tutorial. A typical MATLAB 2009 window would look like the image above. It is divided into four major sections Command Window We write all the commands in this section Current Directory Shows us the contents of the current directory Workspace Displays the details of variables being used Command History Shows the list of commands recently used The toolbar gives option for setting a custom current directory. You were informed that MATLAB does hell lotta stuff using matrices. You can yourself find the tutorials to all that using the Help button
4
robotics.cca@itbhu.ac.in
We would be restricting ourselves to the Image Acquisition tool and the Image Processing tool for this tutorial. Now lets start working with MATLAB. All the commands would be written on the >> symbol you can see in the Command Window. Type k = 9 and Press ENTER
robotics.cca@itbhu.ac.in
Karan Bajaj
You can see that MATLAB automatically stores k as a matrix of size 1x1 with value 9. This action also reflects in the Command History. 1D and 2D matrices can be stored as follows
To access any element in the matrix, we can use indices corresponding to that location. Indices in MATLAB start from 1. For instance
The value stored at second row and first column is 5 We can do simple mathematical operations using the matrix variables -
robotics.cca@itbhu.ac.in
Karan Bajaj
ans is the default variable of MATLAB, and is used if the assignment operator is not present. We can assign the result of this operation using the = and a variable on the left.
Trigonometric functions
sin(1) value in radians is passed sind(45) value in degrees is passed asin(1) inverse sin in radians asind(1) inverse sin in degrees Similarly, we can use cos(), cosd(), tan(), tand(), etc General commands clc Clear the Command Window clear Reset the Workspace and delete all the variables stored
7
robotics.cca@itbhu.ac.in
robotics.cca@itbhu.ac.in
Karan Bajaj Syntax if <condition> statements else statements end while <condition> statements end for <initialization and condition> statements end
zeros(k) : Returns matrix of dimension k-by-k filled with zeros zeros(p,q) : Returns matrix of dimension p-by-q filled with zeros ones(k) : Returns matrix of dimension k-by-k filled with ones ones(p,q) : Returns matrix of dimension p-by-q filled with ones
robotics.cca@itbhu.ac.in
Karan Bajaj Working with Images We can read images in MATLAB as follows
imread() function
Now the variable img stores the image cameraman.tif as a matrix. One would expect an error message on typing this directly, but MATLAB has some default images present, among which cameraman.tif is one. If one wonders the value of the matrix formed, try NOT using the semicolon once!
robotics.cca@itbhu.ac.in
11
robotics.cca@itbhu.ac.in
For each pixel in a colour image, we have a value of the RGB colour triplet. This triplet indicates the intensity of Red, Green and Blue on a scale of 0 to 255 (256 colours). RGB value of an image can be very easily obtained using the Data Cursor tool For instance, in the toolbox of the Figure 1 window of MATLAB. has RGB value as 255, 128 and 0.
You can verify this by using the data cursor tool, which shows -
Making m-files
It sounds quite sensible when I say that we do not write the complete code in Command Window. MATLAB codes are stored in special files called m files. You can use the icon for opening the Editor
12
robotics.cca@itbhu.ac.in
For running the m file, write onto the Command Window, the name of the m file without the extension
13
robotics.cca@itbhu.ac.in
Functions in MATLAB are also stored in an m-file, with an exception that the first executable statement of the file must be of the form function [out1, out2, ...] = <function_name>(in1, in2, ...) The out1, out2, etc are the return values of the function. Unlike C,C++, etc, a MATLAB function can return multiple values. in1, in2, etc are the input parameters required by the function. The function is written in the m-file as follows
Save the m-file with the name of the function stat.m in our case
The function can be executed from the Command Window in the following manner
The vector a is passed to the function stat as parameter, and return values m and std are obtained.
14
robotics.cca@itbhu.ac.in
Karan Bajaj
Coming on to the aspects of Image processing we would be primarily using 1. Obtaining the Image 2. Isolating the area of Interest 3. Obtaining the desired outputs
IMAGE ACQUISITION
Hardware Information For obtaining an image, we would obviously be having an image acquisition device (like a webcam) onto our PC/laptop. MATLAB has software called adaptors for accessing these devices. You can obtain this information by using this command >> imaqhwinfo Can be visualized as im-aq-hw-info (image-acquisition-hardwareinformation) It produces the following output on my system
The adapter that MATLAB indicates on my system is winvideo as shown above. We can now use this adapter to gallop into further details! For getting some more information onto the default colour format, device name, etc, use the following command 15
robotics.cca@itbhu.ac.in
Karan Bajaj
In case you have multiple image acquisition devices, you may use further DeviceIDs like 2, 3, etc in the statement
These are different formats of capturing the video. The 1024x768 indicates the resolution of the video in that particular mode. Defining Video Input Object
winvideo is the adapter name, 1 is the DeviceID of the device we would be using, and YUY2_320x240 is one of the supported format.
16
robotics.cca@itbhu.ac.in
Karan Bajaj
This gives all details of the Video Input Object. Now, how to switch the PREVIEW ON?
And the Video Preview window opens with the resolution you specified in the object creation statement -
Preview can also be viewed using different formats (Resolutions) from the Supported Formats list.
17
robotics.cca@itbhu.ac.in
We use the getshapshot() function for capturing an image from our image input object -
The object appears so weird, you think either your webcam has gone berserk, or its some evil leprechaun!
Nothing to worry, guys! This happened because we took picture in YCbCr family of colour spaces, which is YUY2_... format. You can grab some more details about YCbCr here - http://en.wikipedia.org/wiki/YCbCr We would have to convert this to RGB as follows -
18
robotics.cca@itbhu.ac.in
Karan Bajaj
Saving an Image file Similar to imread() command, we have the imwrite() for writing an image variable onto the disk in form of an image file.
Image file stored IMPORTANT Getting a snapshot requires multiple steps switching ON the camera, sufficient light entering in it, capturing taking place. So there should be a pause of some duration between both the statements, when the code is written in m-file.
19
robotics.cca@itbhu.ac.in
Karan Bajaj
vid=videoinput('winvideo',1, 'YUY2_160x120'); preview(vid); pause(3); % Provides the required pause while(1) img=getsnapshot(vid); % Do all image processing and analysis here end
We now have an idea about how to acquire an image. Let us work on processing the acquired image now!
IMAGE PROCESSING
As the image is now with us as input, our main task is to work on it. Typical Image processing problems include locating a particular object with some colour, or counting objects with the same colour, etc. Let us see a simple example Suppose we have an image shapes.tif
We can store this image in a variable img using the imread() function, and check for the RGB values at different positions. For the image taken, the RGB at Red region is (255, 0, 0), at Green region is (0, 255, 0), and at Blue region is (0, 0, 255).
20
robotics.cca@itbhu.ac.in
Save this file as BB.m Running the program in the Command Window
Figure Window
21
robotics.cca@itbhu.ac.in
Karan Bajaj
What we did in the program was to declare a zero matrix with dimensions same as that of the test image. For every pixel location, we test if the regions RED value is 255 and GREEN and BLUE values are 0. If yes, we mark that place as 1 (white), otherwise it remains 0 (black) in the resultant binary image. A piece of cake! But real time problems consist of not a uniform colour, but a distribution of intensities! Let us take another example Suppose we have an image flower.tif in our MATLAB directory
22
robotics.cca@itbhu.ac.in
Karan Bajaj
The image is processed using the above program. Lets name the file BB2.m On executing the program
We have approximately localized the violet colour region from the real world image. This choice of intensity selection has to be done manually and with great care, as the final binary image will be computed accordingly. For if(im(i,j,1)>50 && im(i,j,2)>50 && im(i,j,3)>120);
23
robotics.cca@itbhu.ac.in
As we saw in the first binary picture of the flower, the resultant image is distorted, with presence of holes, irregularities, etc. This can be treated using the imclose() function. But for using the imclose() function, we need to define a structuring element first, using the strel command as follows
Modified Output
24
robotics.cca@itbhu.ac.in
Karan Bajaj
Another important function used for filling all holes in a binary or grayscale image is imfill() function Using the imfill() function along with the program
Resultant binary image that is produced does not contain any hole
Similar to imclose() function, we have the imopen() function, which performs morphological opening of the grayscale or binary image. Example
I = imread('snowflakes.png'); imshow(I)
25
robotics.cca@itbhu.ac.in
Karan Bajaj
What actually happens here is that all snowflakes with radius less than 5 pixels were removed by using the structuring element of type disk with size 5 pixel, which is used. Image Noise removal is very essential for making the data to be computed error free, and hence, our program need to be efficient at it.
26
robotics.cca@itbhu.ac.in
Karan Bajaj
Some more important commands used for Image Processing are regionprops() STATS = regionprops(BW, properties) It measures a set of properties for each connected component (object) in the binary image, BW. The image BW is a logical array; it can have any dimension.
For example, if you want to locate centroid of any object in a picture you can use the following command s = regionprops(BW, 'centroid'); Web Link: http://www.mathworks.in/help/toolbox/images/ref/regionprops.html
bwlabel() [L, num] = bwlabel(BW, n) It returns a matrix L, of the same size as BW, containing labels for the connected objects in BW. The variable n can have a value of either 4 or 8, where 4 specifies 4-connected objects and 8 specifies 8connected objects. If the argument is omitted, it defaults to 8.
27
robotics.cca@itbhu.ac.in
It returns in num the number of connected objects found in BW. Web Link: http://www.mathworks.in/help/toolbox/images/ref/bwlabel.html
imcrop() I = imcrop creates an interactive Crop Image tool associated with the image displayed in the current figure, called the target image. The Crop Image tool is a moveable, resizable rectangle that you can position interactively using the mouse. I2 = imcrop(I) displays the image I in a figure window and creates a cropping tool associated with that image. I can be a grayscale image, a truecolor image, or a logical array. The cropped image returned, I2, is of the same type as I. Web Link: http://www.mathworks.in/help/toolbox/images/ref/imcrop.html
bwboundaries() B = bwboundaries(BW) traces the exterior boundaries of objects, as well as boundaries of holes inside these objects, in the binary image BW. B = bwboundaries(BW,conn) specifies the connectivity to use when tracing parent and child boundaries. conn can have either of the following scalar values.
robotics.cca@itbhu.ac.in
Karan Bajaj
In laptops/computers of today, we generally do not have parallel ports, and some modern laptops (like mine!) do not have Serial port also. But they do have multiple USB (Universal Serial BUS) ports. So we use USB to Parallel, or USB to Serial port converters USB to Serial Port USB to Parallel Port
29
robotics.cca@itbhu.ac.in
DB25 pinout
30
robotics.cca@itbhu.ac.in
Karan Bajaj
Before you being working with parallel port, do check if the parallel port in your system is operational, by following these steps My Computer>System Properties>Device Manager>Ports You will find the name of your parallel port from here. Let us assume it is LPT1 >> parport = digitalio (parallel, LPT1); We have now defined an object named parport. Port address may be obtained using these statements >> get (parport, PortAddress) >> daqhwinfo (parallel); Now we need to define PINS 2-9 as OUTPUT pins using the addline() function >> addline(parport, 0:7, out) Now, depending upon the motion of the robot desired, you can put the data in a matrix using logical() function >> dout = logical([1 0 0 1 0 0 0 1]); Using the putvalue() function, we can output this value >>putvalue(parport, dout); We can also send the decimal (or octal/hexadecimal) value directly >>dat = 259; >>putvalue(parport, dat); While using the parallel port, we do not need any microcontroller, as outputs can be directly received by the motors from the parallel port using H-bridge (L293D or L298 IC Motor driver ICs) for driving the motor in both directions clockwise and anticlockwise.
31
robotics.cca@itbhu.ac.in
Serial transfer means that if you want to transfer a byte of information, then Serial port will transfer the 8 bits bit-by-bit (one bit at a time). As seen from the above diagram, the transmission happens from PIN3 (Tx), and receiving happens at PIN2 (Rx). Now, we would require a microcontroller with UART (Universal Asynchronous Receiver and Transmitter) to be present. Most of the uC that we use in Robotics (like ATMEGA 16L, PIC, etc) have UART, which needs to be initialized for Serial data transfer. The standard used for Serial communication is RS-232 (Recommended Standard 232). This defines the voltage values that would be equivalent to logic ZERO and logic ONE. Valid signals are positive and negative 3 to 15 volts.
32
robotics.cca@itbhu.ac.in
Karan Bajaj
Now the problem that arises is that our microcontroller works on TTL standards, and hence would not work on RS-232. So, here we introduce IC MAX 232, which would help us in this situation. The connection diagram of MAX 232 with Serial port, for our purpose is given is follows
PINs 11 and 12 can now be used for communication via pins of uC meant for Serial transfer of data. Before you start using the Serial Port, check in the Device Manager (Port section), whether you can see COM ports available. If yes, then your Serial port is ready for some action! Let us assume your Serial port device name to be COM1 We can create a MATLAB object using the following commands >> ser = serial (COM1, BaudRate, 9600); with Baud Rate = 9600 bps (bits per second)
33
robotics.cca@itbhu.ac.in
Karan Bajaj
Baud Rate - In telecommunications and electronics, baud is synonymous to symbols per second or pulses per second. While setting this value in the above command, we must make sure that the value is the same as set while configuring the microcontroller on-board the robot. The Serial data format includes 1 Start bit 5 to 8 Data bits 1 Stop bit Parity bit (may be there) Additional Stop bit (may be there)
Asynchronous transmission using the UART character So the data that we are sending must be of 8bits. Let us suppose that we keep codes in hexadecimal A Forward B Backward C Turn left D Turn right No we open the Serial port and start sending data through it >> fopen (ser); >> fwrite (ser, A); >> fwrite (ser, C); Signal for moving the robot one unit FORWARD, then turning LEFT.
34
robotics.cca@itbhu.ac.in
Karan Bajaj
This data has to be sent to the microcontroller. So now we need to program the microcontroller to listen to the COM1 port. Let us take ATMEGA16L as our microcontroller. We would generate the program to be burnt, using CodeVision AVR software. A typical window of CVAVR looks like this
Select Project and click on OK. From the new menu, select the first option, and click on OK
35
robotics.cca@itbhu.ac.in
Karan Bajaj
Now comes the most important part configuring the microcontroller for use. The window that opens shows several tabs on the left
Select ATmega16L (or any chip you are using) from the Chip tab. We now need to configure the microcontroller for using its USART in asynchronous mode, and along with that set the PORTs of the uC which would be used for signaling the motors.
36
robotics.cca@itbhu.ac.in
Karan Bajaj
Make the necessary settings in this submenu putting a tick on the Transmitter option, and Receiver also if it is being used in your case. The Baud Rate is by default set to 9600. Notice that this is the same as the rate set by us while defining the Serial port object in MATLAB. Decide and set the communication parameters, and set the Mode as asynchronous. PORTs can be set using the Ports tab
37
robotics.cca@itbhu.ac.in
Karan Bajaj Once, all settings are done, Click on Program>Generate, Save and Exit
Write the same names in the Save windows that come. The program now appears in front of you with all configuration settings done.
Browse down the window with the program code to find this
You can now write your code here for accepting data from Serial port, checking it across a set of operations (like moving left, forward, blinking LED, etc), and perform the required action using the respective PORTs that you already have set as input/output. Output from here would now drive the motors.
38
robotics.cca@itbhu.ac.in
Karan Bajaj
MOTOR CONTROL
Now all the data acquisition and processing is complete. All we need to do is to RESPOND to the instructions that the microcontroller gives. We would do this by interfacing the Output PORT of the microcontroller with the motors. The problem that arises is the output potential of the port of microcontroller is not enough to drive a motor, hence, we need to use a Motor driver IC (L293D, L298) for doing the same. We would use these with both DC motors and Stepper motors as given below.
DC Motors
As already mentioned above, the DC motor would require an IC for working with the microcontroller output. We can use L293D for this purpose
39
robotics.cca@itbhu.ac.in
Karan Bajaj
Pin diagram of L293D Each L293D has four Input-Output pairs. For making a motor run both directions, we need to use two Input-Output pairs per motor. Hence, we can use ONE L293D for controlling two DC motors
40
robotics.cca@itbhu.ac.in
Karan Bajaj
Shown above is the connection diagram of the two DC motors with one L293D IC.
Vs-pin stands for Motor Voltage and is the voltage with which the motors would be driven. We can use a 12V DC connection on this pin. Enable pins must be set HIGH for enabling the working of both motors. These pins can also be used to DEACTIVATE the motor functionality under certain circumstances as directed by the algorithm. For various INPUT conditions at IN1, IN2 and IN3, IN4, we see the following pattern per motor -
DC motors are simple, but they lack accuracy. In case we want to use DC motors, we have to use an efficient feedback mechanism to overcome and compensate the loss of accuracy.
41
robotics.cca@itbhu.ac.in
It is a brushless, electric motor that can divide a full rotation into a large number of steps. The motor's position can be controlled precisely without any feedback mechanism. How it works?
II
III
IV
Hence, a complete rotation happens after four such steps. The angle which the Stepper motor rotates with after completion of any step above is called the Resolution of the Stepper motor. Steppers are available in many Resolutions, ranging from 30 to 1.8 degrees per step. Stepper motor can be either Unipolar or Bipolar.
42
robotics.cca@itbhu.ac.in
Karan Bajaj
As you can see, each stepper has four coils. Hence we have four connections per stepper motor. So, one L293D can drive one bipolar 2 phase stepper motor. The connection diagram of the same is given as follows
We have the four Controller Pins going to the OUTPUT port of the microcontroller. And the data coming through these pins will be controlling the operation of the Stepper. We saw in a diagram above that there is a specific sequence in which coils should get activated, so as to ensure proper completion of a cycle. The pulse sequence has to be in the following order Step Step Step Step 1 2 3 4 Coil 1A High Low Low High Coil 2A High High Low Low Coil 1B Low High High Low Coil 2B Low Low High High
robotics.cca@itbhu.ac.in
A Unipolar stepper motor has one winding with center tap per phase. The special thing about Unipolar Steppers is that we can reverse the direction of rotation of the motor without changing the direction of current.
For Unipolar Stepper, we use IC ULN2003/2004 instead of L293D, as it also has a COM port for serving as the Common connection of the coils.
44
robotics.cca@itbhu.ac.in
Karan Bajaj
Thank you for going through the tutorial. If you have any queries or suggestions that you wish to share, feel free to contact me at karanbajaj23@gmail.com karan.bajaj.cse09@itbhu.ac.in
45
robotics.cca@itbhu.ac.in