Work in Progress

ApplianceReader: A Wearable, Crowdsourced, Vision-based System to Make Appliances Accessible

Authors:

Anhong Guo,

Xiang 'Anthony' Chen,

Jeffrey P. BighamAuthors Info & Claims

CHI EA '15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems

Pages 2043 - 2048

https://doi.org/10.1145/2702613.2732755

Published: 18 April 2015 Publication History

Get Access

Abstract

Visually impaired people can struggle to use everyday appliances with inaccessible control panels. To address this problem, we present ApplianceReader - a system that combines a wearable point-of-view camera with on-demand crowdsourcing and computer vision to make appliance interfaces accessible. ApplianceReader sends photos of appliance interfaces that it has not seen previously to the crowd, who work in parallel to quickly label and describe elements of the interface. Computer vision techniques then track the user's finger pointing at the controls and read out the labels previously provided by the crowd. This enables visually impaired users to interactively explore and use appliances without asking the crowd repetitively. ApplianceReader broadly demonstrates the potential of hybrid approaches that combine human and machine intelligence to effectively realize intelligent, interactive access technology today.

References

[1]

The braille literacy crisis in america. facing the truth, reversing the trend, empowering the blind. National Federation of the Blind, Jernigan Institute, March 2009.

Google Scholar

[2]

Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, ACM (2010), 333--342.

Digital Library

Google Scholar

[3]

Bigham, J. P., Jayant, C., Miller, A., White, B., and Yeh, T. Vizwiz:: Locateit-enabling blind people to locate objects in their environment. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on, IEEE (2010), 65--72.

Crossref

Google Scholar

[4]

Kane, S. K., Frey, B., and Wobbrock, J. O. Access lens: a gesture-based screen reader for real-world documents. In Proc. CHI, ACM (2013), 347--350.

Digital Library

Google Scholar

[5]

Lasecki, W. S., Thiha, P., Zhong, Y., Brady, E., and Bigham, J. P. Answering visual questions with conversational crowd assistants. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility, ACM (2013), 18.

Digital Library

Google Scholar

[6]

Manduchi, R., and Coughlan, J. (computer) vision without sight. Communications of the ACM 55, 1 (2012), 96--104.

Digital Library

Google Scholar

[7]

Zhong, Y., L. W. B. E., and Bigham, J. Regionspeak: Quick comprehensive spatial descriptions of complex images for blind users. In Proc. CHI, ACM (2015).

Digital Library

Google Scholar

Cited By

View all

Tschakert HLang FWieland MSchmidt AMachulla T(2023)A Dataset and Machine Learning Approach to Classify and Augment Interface Elements of Household Appliances to Support People with Visual ImpairmentProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584038(77-90)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584038
Nirjhar EKim JWinslow JChaspari TAhn C(2023)Sensor-based detection of individual walkability perception to promote healthy communitiesSmart Health10.1016/j.smhl.2023.10041429(100414)Online publication date: Sep-2023
https://doi.org/10.1016/j.smhl.2023.100414
Lei WFan MThang J(2022)“I Shake The Package To Check If It’s Mine”Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502063(1-15)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502063
Show More Cited By

Index Terms

ApplianceReader: A Wearable, Crowdsourced, Vision-based System to Make Appliances Accessible
1. Human-centered computing
  1. Human computer interaction (HCI)
2. Social and professional topics
  1. Professional topics
    1. Computing profession
      1. Assistive technologies
  2. User characteristics
    1. People with disabilities

Recommendations

Facade: Auto-generating Tactile Interfaces to Appliances
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

Common appliances have shifted toward flat interface panels, making them inaccessible to blind people. Although blind people can label appliances with Braille stickers, doing so generally requires sighted assistance to identify the original functions ...
VizLens: A Robust and Interactive Screen Reader for Interfaces in the Real World
UIST '16: Proceedings of the 29th Annual Symposium on User Interface Software and Technology

The world is full of physical interfaces that are inaccessible to blind people, from microwaves and information kiosks to thermostats and checkout terminals. Blind people cannot independently use such devices without at least first learning their layout,...
Making Real-World Interfaces Accessible Through Crowdsourcing, Computer Vision, and Fabrication
W4A '17: Proceedings of the 14th International Web for All Conference

The world is full of physical interfaces that are inaccessible to blind people, from microwaves and information kiosks to thermostats and checkout terminals. Blind people cannot independently use such devices without at least first learning their layout,...

Comments

Information & Contributors

Information

Published In

CHI EA '15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems

April 2015

2546 pages

ISBN:9781450331463

DOI:10.1145/2702613

General Chairs:
Bo Begole
Huawei, USA
,
Jinwoo Kim
Yonsei University, Korea
,
Program Chairs:
Kori Inkpen
Microsoft Research, USA
,
Woontack Woo
KAIST, Korea

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 April 2015

Check for updates

Author Tags

Qualifiers

Work in progress

Funding Sources

National Science Foundation

Conference

CHI '15

Sponsor:

SIGCHI

CHI '15: CHI Conference on Human Factors in Computing Systems

April 18 - 23, 2015

Seoul, Republic of Korea

Acceptance Rates

CHI EA '15 Paper Acceptance Rate 379 of 1,520 submissions, 25%;

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
215
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)2

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Tschakert HLang FWieland MSchmidt AMachulla T(2023)A Dataset and Machine Learning Approach to Classify and Augment Interface Elements of Household Appliances to Support People with Visual ImpairmentProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584038(77-90)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584038
Nirjhar EKim JWinslow JChaspari TAhn C(2023)Sensor-based detection of individual walkability perception to promote healthy communitiesSmart Health10.1016/j.smhl.2023.10041429(100414)Online publication date: Sep-2023
https://doi.org/10.1016/j.smhl.2023.100414
Lei WFan MThang J(2022)“I Shake The Package To Check If It’s Mine”Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502063(1-15)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502063
J. P. Amorim VC. Silva MA. R. Oliveira R(2019)Software and Hardware Requirements and Trade-Offs in Operating Systems for Wearables: A Tool to Improve Devices’ PerformanceSensors10.3390/s1908190419:8(1904)Online publication date: 22-Apr-2019
https://doi.org/10.3390/s19081904
Guo AKim JChen XYeh THudson SMankoff JBigham JMark GFussell SLampe Cschraefel mHourcade JAppert CWigdor D(2017)FacadeProceedings of the 2017 CHI Conference on Human Factors in Computing Systems10.1145/3025453.3025845(5826-5838)Online publication date: 2-May-2017
https://dl.acm.org/doi/10.1145/3025453.3025845
Guo AChen XQi HWhite SGhosh SAsakawa CBigham JRekimoto JIgarashi TWobbrock JAvrahami D(2016)VizLensProceedings of the 29th Annual Symposium on User Interface Software and Technology10.1145/2984511.2984518(651-664)Online publication date: 16-Oct-2016
https://dl.acm.org/doi/10.1145/2984511.2984518
Fusco GTekin EGiudice NCoughlan JYesilada YBigham J(2015)Appliance DisplaysProceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility10.1145/2700648.2811392(405-406)Online publication date: 26-Oct-2015
https://dl.acm.org/doi/10.1145/2700648.2811392

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Facade: Auto-generating Tactile Interfaces to Appliances

VizLens: A Robust and Interactive Screen Reader for Interfaces in the Real World

Making Real-World Interfaces Accessible Through Crowdsourcing, Computer Vision, and Fabrication