Eyesight Sharing in Blind Grocery Shopping: Remote P2P Caregiving through Cloud Computing

Abstract

Product recognition continues to be a major access barrier for visually impaired (VI) and blind individuals in modern supermarkets. R&D approaches to this problem in the assistive technology (AT) literature vary from automated vision-based solutions to crowdsourcing applications where VI clients send image identification requests to web services. The former struggle with run-time failures and scalability while the latter must cope with concerns about trust, privacy, and quality of service. In this paper, we investigate a mobile cloud computing framework for remote caregiving that may help VI and blind clients with product recognition in supermarkets. This framework emphasizes remote teleassistance and assumes that clients work with dedicated caregivers (helpers). Clients tap on their smartphones’ touchscreens to send images of products they examine to the cloud where the SURF algorithm matches incoming image against its image database. Images along with the names of the top 5 matches are sent to remote sighted helpers via push notification services. A helper confirms the product’s name, if it is in the top 5 matches, or speaks or types the product’s name, if it is not. Basic quality of service is ensured through human eyesight sharing even when image matching does not work well. We implemented this framework in a module called EyeShare on two Android 2.3.3/2.3.6 smartphones. EyeShare was tested in three experiments with one blindfolded subject: one lab study and two experiments in Fresh Market, a supermarket in Logan, Utah. The results of our experiments show that the proposed framework may be used as a product identification solution in supermarkets.