PRATEEK AGRAWAL* et al.

[IJESAT] INTERNATIONAL JOURNAL OF ENGINEERING SCIENCE & ADVANCED TECHNOLOGY

ISSN: 22503676
Volume-2, Issue-2, 245 249

MOUSE MOVEMENT THROUGH FINGER BY IMAGE GRABBING USING SIXTH SENSE TECHNOLOGY
Prateek Agrawal1, Kunal Gupta2
M.Tech. Student, Computer Science Department, Amity School of Science & Technology, Amity University, Noida, Uttar Pradesh, India, prateekagrawal21@gmail.com 2 Assistant Professor, Computer Science Department, Amity School of Science & Technology, Amity University, Noida, Uttar Pradesh, India, kgupta@amity.edu
1

Abstract
Abundant amount of input devices are used to interact with the computer world or more precisely saying to digital world and very less through gestures made by body movements. Concepts of assistive technology are one of them used for controlling the input from mouse movements, like by detecting the eye movements of a user with the help of eye tracking system, hand gestures through wearable devices, etc. The gesture recognition devices are already developed but we focus on bridging the crevice between the two different worlds i.e. Physical and Digital world using sixth sense technology. Sixth Sense Technology is the technology in which the standard inputs get replaced with the wearable gesture interface that aggrandize physical world around us with digital world and lets us to use the natural hand gestures that interplay with the information related to it. Our focus is in moving mouse cursor on the screen without using any hardware which is used very often now-a-days i.e. mouse. We use the newly born technology named Sixth Sense Technology for this purpose. We implement computer mouse movement through finger by image grabbing using Sixth Sense Technology which gets processed in MATLAB without using gesture recognition.

Index Terms: Sixth Sense Technology, Colour Recognition, MATLAB, Image Processing, Mouse, Keyboard.
---------------------------------------------------------------***--------------------------------------------------------------------

1. INTRODUCTION
Sixth Sense Technology is a revolutionary way to augment the physical world directly without using dedicated electronic chips. Sixth Sense is a set of wearable devices that acts as a gestural interface and aggrandize the physical world around us with digital information and lets the users to use natural hand gestures to interact with the digital information through it. This technology gaining its popularity strength because of the usability, simplicity and ability to work independently in todays scenario. Many other modern technologies are available which are widely used like touch screen that not only saves the utilization time but increases the ease of usability too. Its implementation consists of five main components and a microphone which is optional and can be used for speech recognition purpose, that collectively acts as a system in itself and each device has its important role in the system. The devices include a webcam or digital camera, coloured caps or markers, a phone or laptop, a projector and a mirror. Camera is

used to capture the object in sight range and follow the users hand gestures, sending the data to phone or laptop connected with it. Camera acts as a digital eye connecting the user to the digital world. Coloured caps or markers are attached at the finger tips of the user. Marking the users fingers with different colours helps the webcam to recognize the gestures made by the fingers. The movements and arrangements of these markers are grasped as gestures that act as an interaction instructions for the projected application interfaces. A phone or laptop with web enabled services is used as the processing device that processes the input video data send by the camera. Other softwares in it search the web and interpret the gestures. The projector projects the visual information enabling the surfaces and physical objects to be used as interfaces. It itself contains a battery with the battery life of around three hours. The tiny LED projector displays the data send from the phone on any surface on view like any object, wall or a person. Mirror is used to display the projected images or videos simply by reflecting it to the desired object or surface.

IJESAT | Mar-Apr 2012

Available online @ http://www.ijesat.org 245

PRATEEK AGRAWAL* et al. [IJESAT] INTERNATIONAL JOURNAL OF ENGINEERING SCIENCE & ADVANCED TECHNOLOGY This method bridgesup the crevice between the physical world and digital world but still has bottle necks such as position of projector, position of camera for capturing the gestures and the accuracy of the projected output depends on the accuracy of the input taken by the camera. This hindrance increases the use of commands along with the hand gestures as the position of camera is a major constraint in the image capturing and projected output that directly affects the efficiency and accuracy. So to remove this constraint, actions which we regularly perform in our daily life are converted to commands. Speech also has its importance in this method as they can be stored in a database in the integrated circuit and corresponding actions are performed when the speech is recognised from the user. Mouse is the most popular input device now-a-days used for the human interaction with the computer systems to interact with the digital world through users hand. The popularity gained its strength from the days when the development of GUIs based operating systems started like Microsoft Windows 95, Windows 98, Macintosh, Windowsamiga, Symbian OS, and many more but still it is dependent on the system. The device i.e. mouse changes the relative position of itself, with respect to the base surface on which it is lying, transforms the motion from the form of two-dimensional coordinates to the device controller to begin the cursor movement on the screen. The mouse clicks, that generates the signals on pressing the mouse buttons acts as an input to the pre-programmed structures that finally fires an event or triggers an activity accordingly. Earlier it was done by using electronic chips but here we emphasize on image recognition instead of electronic chips. Other applications can also be developed, using similar approach, like taking the live pictures from the physical world and saving them to the system using hand gestures, checking current time, watching videos, drawing scenery and many more. This increases the intuitiveness of the system with the physical world where user can paint the physical world using the digital information in his/her own way. The following figures show the examples of the applications of the same. (a)

ISSN: 22503676
Volume-2, Issue-2, 245 249

(b)

(c)

(d) Fig-1: (a), (b), (c), and (d) shows different applications of Sixth Sense Technology

2. PROPOSED METHOD
The methodology used is based on the Sixth Sense Technology where user have some devices which together acts as a system. Our aim is to move mouse cursor as the user moves his/her fingers. For this purpose, three components of Sixth Sense are used i.e. Camera, Coloured Caps and MATLAB installed in Laptop. The following figure shows the hardware devices we have used for experimental purpose.

(a)

IJESAT | Mar-Apr 2012

Available online @ http://www.ijesat.org 246

PRATEEK AGRAWAL* et al. [IJESAT] INTERNATIONAL JOURNAL OF ENGINEERING SCIENCE & ADVANCED TECHNOLOGY

ISSN: 22503676
Volume-2, Issue-2, 245 249

(b)

camera into frames of images or slicing of video is done in the form of images. These images that are obtained from the slicing of video are then processed for colour recognition process. The output of the colour recognition process are the images that contains only those colours of which colour caps are present at the finger tips of the user. Neither the fingers of user are shown in the output images nor are any background colours there in the output images from the colour recognition process. For this purpose, RGB values of the colour caps are set prior in the code so that no other colour will be detected in the image after colour recognition except the caps colours. The output images are displayed in continuation and at the same speed as the speed at which slicing of video is done, so that it looks like a continuous movie in which the input is physical world and the output is only those colours which are present at the fingertips of the user. The colour is then associated with the mouse cursor in code so that whenever the colour moves in the output image from one position to another, the mouse cursor gets attached at the same position where the colour is now displayed.

(c) Fig-2: (a) shows web camera used for taking input from physical world. (b) shows colour markers or colour caps at the fingertips and (c) shows a laptop installed with MATLAB used as the process device. The approach works in a continuous manner where camera takes the live video, sending to the laptop, and MATLAB installed in laptop processes the input and recognizes the colours at the finger tips of the user. Following figure shows the algorithm we used in our approach to move mouse cursor on screen using sixth sense technology thus providing a bridge between physical world and digital world.

3. IMPLEMENTATION & RESULTS

The image processing and colour recognition was simulated in the MATLAB to process the image from the live video from camera to obtain the final image showing required colours only. The image processing and colour recognition part was performed with the machine with i5 processor clocked at 2.40 GHz equipped with 4 MB cache memory, 4 GB RAM and standard on board graphics processing unit of Intel motherboard. The algorithm returns the image showing the desired colours only that are placed at the fingertips of the user. For this purpose we use a sample image. The sample image and its output image after colour recognition are shown below.

Fig-3: Proposed Methodology Here, in our proposed methodology, first interaction with the physical world is done by camera. Camera takes the video and starts recording the live video and in continuation of recording it sends the live video to MATLAB which is already installed in laptop which is connected with the camera. In MATLAB, code is prepared which convert the incoming live video from

Fig-4.1: Sample image

IJESAT | Mar-Apr 2012

Available online @ http://www.ijesat.org 247

PRATEEK AGRAWAL* et al. [IJESAT] INTERNATIONAL JOURNAL OF ENGINEERING SCIENCE & ADVANCED TECHNOLOGY Fig-4.1 is the sample image we took to accomplish the colour recognition process. On this image we took only yellow colour to be recognized at first.

ISSN: 22503676
Volume-2, Issue-2, 245 249

The above figure contains yellow colour also, so it can be done for any colour and it is users choice to select the colour of his/her own choice. For this only thing need to be done is to change the RGB pixel threshold values for that colour which can be checked from internet.The above colour recognition process took 0.031 seconds to recognize three colours in a single image. Since the above colour recognition till shows some other colours of the background of the imgae, hence working on the live video during colour recognition process, it will show background colours also which can create ambiguation for the cursor to detect the exact colour or colours to get associated with to change its position. For this purpose, some more optimization and RGB pixel threshold values need to be modified in such a way that it will take only and only those colours which user want to show.The figure shown below demonstrate the optimized colour recognition output image showing only three colours and background is taken to be black so that visualization doesnt create any kind of ambiguation to the user.

Fig-4.2: Output image obtained after colour recognition process done only for one colour i.e. yellow. Above figure shows only the yellow colour in output image. For this we took RGB values to be greater than 200, greater than 200 and less than 125 (i.e. >200,>200 and <125). The running time for this was 0.027 seconds. When the RGB values for the colours are changed, it gives more colours for the same sample image of balls. The RGB values for- red colour was fixed to be greater than 100 (for red part) but less than 125 (for green part) and 125 (for blue part) (i.e. >100, <125, <125), similarly for green colour it was fixed to be less than 50, greater than 50 and less than 50 (i.e. <50, >50, <50)for red, green and blue parts, in the same way blue part was fixed to be less than 100, less than 100 and greater than 150 for red, green and blue RGB pixel values. Since we took another colour yellow instead of blue, to accomplish this, we took different RGB pixel value to detect or recognize yellow colour. For yellow colour we took greater than 150, greater than 165 and less than 100 RGB values (i.e. >150, <165, >100) which gives the output shown in figure below.

Fig-4.4: Optimized output image obtained after colour recognition process done for three colours i.e. yellow, red and green. To get the above output image, RGB values are set to greater than 200, less than 125 and less than 125 (i.e. >200, <125, <125) for red colour, less than 125, greater than 200 and less than 125 (i.e. <125, >200, <125) for green colour, less than 125, less than 125 and greater than 200 (i.e. <125, <125, >200) for blue colour and greater than 200, greater than 200 and less than 100 (i.e. >200, >200, <100) for yellow colour. The above colour recognition was successfully done in 0.022 seconds hence better execution time with better colour recognition and better visualization of the colours in the image.

Fig-4.3: Output image obtained after colour recognition process done for three colours i.e. yellow, red and green.

IJESAT | Mar-Apr 2012

Available online @ http://www.ijesat.org 248

PRATEEK AGRAWAL* et al. [IJESAT] INTERNATIONAL JOURNAL OF ENGINEERING SCIENCE & ADVANCED TECHNOLOGY

ISSN: 22503676
Volume-2, Issue-2, 245 249

4. CONCLUSION
The use of image processing and colour recognition in MATLAB for the implementation of our proposed approach proved to be practically successful and the movement of mouse cursor is achieved with a good precision. The approach has huge potential once it gets further optimized, as its time complexity is higher, with the help of hardware having better specifications. This approach has much high potential for future advanced applications which can have the ability to change the mobile world also. The overall cost of this method is within USD 150 but its commercial implementation can be easily made in affordable price range for the common man.

BIOGRAPHIES
Prateek Agrawal received B. Tech. degree from Uttar Pradesh Technical University, Uttar Pradesh and currently pursuing M. Tech. in Computer Science and Engineering from Amity University, Noida.

5. FUTURE WORKS
Sixth Sense has the ability to remove the gap between the two worlds and observing the implementation of the system it is clear that the designed system has a huge potential to be used in immense and can be said that the proliferation of this technology may be there within this 2020 and sufficient awareness of this technology will lead to further development of any technology.

Kunal Gupta received B.Tech. degree from Punjab University Patiala, M.Tech. Degree from Punjab Engineering College, Chandigarh and currently pursuing PhD from DTU and have 5 year Academic experience and from last 2 years working with Amity University, Noida.

ACKNOWLEDGEMENTS
I would like to express heartfelt gratitude towards my faculty guide Mr. Kunal Gupta, Faculty, Computer Science & Engineering Department, Amity University for his guidance and help in completing this work successfully with good results. I would also like to express my gratitude to my Head of the Department Prof. Dr. (Col) Kaiser Singh for his timely support and help.

REFERENCES
[1] S. Sadhana Rao Sixth Sense Technology Proceedings of the International Conference on Communication and Computational Intelligence 2010, Kongu Engineering College, Perundurai, Erode, T.N.,India.27 29 December,2010.pp.336-339. [2] http://www.ted.com/talks/pranav_mistry_the_thrilling_po tential_of_sixthsense_technology.html Last visited on 10th Dec11 [3] http://www.commoncraft.com/video/augmented-reality Last visited on 10th Dec11 [4] http://www.ted.com/talks/pattie_maes_demos_the_sixth_s ense.html Last visited on 10th Dec11

IJESAT | Mar-Apr 2012

Available online @ http://www.ijesat.org 249