research-article

FootNotes: Geo-referenced Audio Annotations for Nonvisual Exploration

Authors:

Alexander J. Fiannaca,

Melanie Kneisel,

Edward Cutrell,

Meredith Ringel MorrisAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 2, Issue 3

Article No.: 109, Pages 1 - 24

https://doi.org/10.1145/3264919

Published: 18 September 2018 Publication History

Abstract

The majority of information in the physical environment is conveyed visually, meaning that people with vision impairments often lack access to the shared cultural, historical, and practical features that define a city. How can someone who is blind find out about the sleek skyscrapers that dot a modern city's skyline, historic cannons that have been remade into traffic pillars, or ancient trees that uproot a neighborhood's sidewalks? We present FootNotes, a system that embeds rich textual descriptions of objects and locations in OpenStreetMap, a popular geowiki. Both sighted and blind users can annotate the physical environment with functional, visual, historical, and social descriptions. We report on the experience of ten participants with vision impairments who used a spatialized audio application to interact with these annotations while exploring a city. By sharing rich annotations of physical objects and areas, FootNotes helps people thoroughly explore a new location or serendipitously discover previously unknown features of familiar environments.

Supplementary Material

gleason (gleason.zip)

Supplemental movie, appendix, image and software files for, FootNotes: Geo-referenced Audio Annotations for Nonvisual Exploration

Download
56.86 MB

References

[1]

Aira. 2017. Aira | Instant Access to Information. https://aira.io

[2]

Frank Bentley and Santosh Basapur. 2012. StoryPlace.Me: The Path from Studying Elder Communication to a Public Location-based Video Service. In CHI '12 Extended Abstracts on Human Factors in Computing Systems (CHIEA '12). ACM, New York, NY, USA, 777--792.

Digital Library

[3]

Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Aubrey Tatarowicz, Brandyn White, Samuel White, and Tom Yeh. 2010. VizWiz: Nearly Real-time Answers to Visual Questions. In Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) (W4A '10). ACM, New York, NY, USA, Article 24, 2 pages.

Digital Library

[4]

Jeffrey P. Bigham, Ryan S. Kaminsky, Richard E. Ladner, Oscar M. Danielsson, and Gordon L. Hempton. 2006. WebInSight:: Making Web Images Accessible. In Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '06). ACM, New York, NY, USA, 181--188.

Digital Library

[5]

Erin Brady, Meredith Ringel Morris, and Jeffrey P. Bigham. 2015. Gauging Receptiveness to Social Microvolunteering. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1055--1064.

Digital Library

[6]

Erin L. Brady, Yu Zhong, Meredith Ringel Morris, and Jeffrey P. Bigham. 2013. Investigating the Appropriateness of Social Network Question Asking As a Resource for Blind Users. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW '13). ACM, New York, NY, USA, 1225--1236.

Digital Library

[7]

OpenStreetMap Wiki contributors. 2017. Changeset. https://wiki.openstreetmap.org/wiki/Changeset

[8]

OpenStreetMap Wiki contributors. 2017. Changeset. https://wiki.openstreetmap.org/wiki/Changeset

[9]

OpenStreetMap Wiki contributors. 2017. Elements. https://wiki.openstreetmap.org/wiki/Elements

[10]

OpenStreetMap Wiki contributors. 2017. Key:oneway. https://wiki.openstreetmap.org/wiki/Key:oneway

[11]

Daniel Dardailler. 1997. The ALT-server (" An eye for an alt").

[12]

Navid Fallah, Ilias Apostolopoulos, Kostas Bekris, and Eelke Folmer. 2012. The User As a Sensor: Navigating Users with Visual Impairments in Indoor Spaces Using Tactile Landmarks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 425--432.

Digital Library

[13]

Navid Fallah, Ilias Apostolopoulos, Kostas Bekris, and Eelke Folmer. 2013. Indoor Human Navigation Systems: A Survey. Interacting with Computers () 25, 1 (2013), 21--33.

[14]

Alexander Fiannaca, Ilias Apostolopoulous, and Eelke Folmer. 2014. Headlock: A Wearable Navigation Aid That Helps Blind Cane Users Traverse Large Open Spaces. In Proceedings of the 16th International ACM SIGACCESS Conference on Computers 8 Accessibility (ASSETS '14). ACM, New York, NY, USA, 19--26.

Digital Library

[15]

Elizabeth FitzGerald. 2012. Creating user-generated content for location-based learning: an authoring framework. Journal of Computer Assisted Learning 28, 3 (2012), 195--207.

Digital Library

[16]

REGINALD G. GOLLEDGE, ROBERTA L. KLATZKY, JACK M. LOOMIS, JON SPEIGLE, and JEROME TIETZ. 1998. A geographical information system for a GPS based personal guidance system. International Journal of Geographical Information Science 12, 7 (1998), 727--749. arXiv:https://doi.org/10.1080/136588198241635

[17]

W. G. Griswold, P. Shanahan, S. W. Brown, R. Boyer, M. Ratto, R. B. Shapiro, and T. M. Truong. 2004. ActiveCampus: experiments in community-oriented ubiquitous computing. Computer 37, 10 (Oct 2004), 73--81.

Digital Library

[18]

João Guerreiro, Dragan Ahmetovic, Kris M. Kitani, and Chieko Asakawa. 2017. Virtual Navigation for Blind People: Building Sequential Representations of the Real-World. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '17). ACM, New York, NY, USA, 280--289.

Digital Library

[19]

Mordechai Haklay. 2010. How Good is Volunteered Geographical Information? A Comparative Study of OpenStreetMap and Ordnance Survey Datasets. Environment and Planning B: Planning and Design 37, 4 (2010), 682--703. arXiv:https://doi.org/10.1068/b35097

[20]

Kotaro Hara, Shiri Azenkot, Megan Campbell, Cynthia L. Bennett, Vicki Le, Sean Pannella, Robert Moore, Kelly Minckler, Rochelle H. Ng, and Jon E. Froehlich. 2013. Improving Public Transit Accessibility for Blind Riders by Crowdsourcing Bus Stop Landmark Locations with Google Street View. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '13). ACM, New York, NY, USA, Article 16, 8 pages.

Digital Library

[21]

Kotaro Hara, Vicki Le, and Jon Froehlich. 2013. Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 631--640.

Digital Library

[22]

Bill Holton. 2014. BlindSquare App for iOS: A Solution for Accessible Navigation. Issue 7. https://www.afb.org/afbpress/pubnew.asp?DocID=aw150704

[23]

Hernisa Kacorri, Kris M. Kitani, Jeffrey P. Bigham, and Chieko Asakawa. 2017. People with Visual Impairment Training Personal Object Recognizers: Feasibility and Challenges. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 5839--5849.

Digital Library

[24]

Shaun K. Kane, Chandrika Jayant, Jacob O. Wobbrock, and Richard E. Ladner. 2009. Freedom to Roam: A Study of Mobile Device Adoption and Accessibility for People with Visual and Motor Disabilities. In Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '09). ACM, New York, NY, USA, 115--122.

Digital Library

[25]

Aniket Kittur and Robert E. Kraut. 2008. Harnessing the Wisdom of Crowds in Wikipedia: Quality Through Coordination. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (CSCW '08). ACM, New York, NY, USA 37--46.

Digital Library

[26]

Jacobus Lock, Grzegorz Cielniak, and Nicola Bellotto. 2017. Portable navigations system with adaptive multimodal interface for the blind. In The 2017 AAAI Spring Symposium Series. AAAI, The AAAI Press, Palo Alto, CA, USA, 395--400.

[27]

Microsoft. 2017. Seeing AI | Talking camera app for those with a visual impairment. https://www.microsoft.com/en-us/seeing-ai/

[28]

Per Persson, Fredrik Espinoza, Petra Fagerberg, Anna Sandin, and Rickard Coster. 2003. GeoNotes: A Location-Based Information System for Public Spaces. In Designing Information Spaces: The Social Navigation Approach, Krishna Hook, David Benyon, and Alan J. Munro (Eds.). Springer London, London, 151--173.

Digital Library

[29]

Reid Priedhorsky, Mikhil Masli, and Loren Terveen. 2010. Eliciting and Focusing Geographic Volunteer Work. In Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work (CSCW '10). ACM, New York, NY, USA 61--70.

Digital Library

[30]

Reid Priedhorsky and Loren Terveen. 2008. The Computational Geowiki: What, Why, and How. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (CSCW '08). ACM, New York, NY, USA, 267--276.

Digital Library

[31]

Microsoft Research. 2018. Microsoft Soundscape, https://www.microsoft.com/en-us/research/product/soundscape/

[32]

Hironobu Takagi, Shinya Kawanaka, Masatomo Kobayashi, Takashi Itoh, and Chieko Asakawa. 2008. Social Accessibility: Achieving Accessibility Through Collaborative Metadata Authoring. In Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '08). ACM, New York, NY, USA, 193--200.

Digital Library

[33]

Fernando Torre, S. Andrew Sheppard, Reid Priedhorsky, and Loren Terveen. 2010. Bumpy, Caution with Merging: An Exploration of Tagging in a Geowiki. In Proceedings of the 16th ACM International Conference on Supporting Group Work (GROUP '10). ACM, New York, NY, USA, 155--164.

Digital Library

[34]

Luis von Ahn, Shiry Ginosar, Mihir Kedia, Ruoran Liu, and Manuel Blum. 2006. Improving Accessibility of the Web with a Computer Game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06). ACM, New York, NY, USA, 79--82.

Digital Library

[35]

Yuhang Zhao, Cynthia L. Bennett, Hrvoje Benko, Edward Cutrell, Christian Holz, Meredith Ringel Morris, and Mike Sinclair. 2018. Enabling People with Visual Impairments to Navigate Virtual Reality with a Haptic and Auditory Cane Simulation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 116, 14 pages.

Digital Library

Cited By

Kaniwa YKuribayashi MKayukawa SSato DTakagi HAsakawa CMorishima S(2024)ChitChatGuide: Conversational Interaction Using Large Language Models for Assisting People with Visual Impairments to Explore a Shopping MallProceedings of the ACM on Human-Computer Interaction10.1145/36764928:MHCI(1-25)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676492
Gupta CSridhar SMatthies DJouffrais CNanayakkara S(2024)SonicVista: Towards Creating Awareness of Distant Scenes through SonificationProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596098:2(1-32)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659609
Sheshadri SHara K(2024)Conversational LocalizationProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314047:4(1-32)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631404
Show More Cited By

Index Terms

FootNotes: Geo-referenced Audio Annotations for Nonvisual Exploration
1. Human-centered computing
2. Social and professional topics
  1. User characteristics
    1. People with disabilities

Recommendations

Towards a real-time system for finding and reading signs for visually impaired users
ICCHP'12: Proceedings of the 13th international conference on Computers Helping People with Special Needs - Volume Part II

Printed text is a ubiquitous form of information that is inaccessible to many blind and visually impaired people unless it is represented in a non-visual form such as Braille. OCR (optical character recognition) systems have been used by blind and ...
Implementing Assistive Technology on Wearable Computers

The article presents the author's work in developing assistive technology for disabled users. He makes a case for wearable computers and focuses on design strategies that accommodate the users' changing needs. He points out that what might seem ...
A Refreshable Tactile Display Effectively Supports Cognitive Mapping Followed by Orientation and Mobility Tasks: A Comparative Multi-modal Study Involving Blind and Low-vision Participants
MAHCI '19: Proceedings of the 2nd Workshop on Multimedia for Accessible Human Computer Interfaces

We investigate the role of refreshable tactile display in supporting the learning of cognitive maps, followed by actual exploration of a real environment that matches that map. We test both blind and low-vision persons and compare displaying maps in ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 2, Issue 3

September 2018

1536 pages

EISSN:2474-9567

DOI:10.1145/3279953

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 September 2018

Accepted: 01 September 2018

Revised: 01 May 2018

Received: 01 February 2018

Published in IMWUT Volume 2, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
320
Total Downloads

Downloads (Last 12 months)36
Downloads (Last 6 weeks)2

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kaniwa YKuribayashi MKayukawa SSato DTakagi HAsakawa CMorishima S(2024)ChitChatGuide: Conversational Interaction Using Large Language Models for Assisting People with Visual Impairments to Explore a Shopping MallProceedings of the ACM on Human-Computer Interaction10.1145/36764928:MHCI(1-25)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676492
Gupta CSridhar SMatthies DJouffrais CNanayakkara S(2024)SonicVista: Towards Creating Awareness of Distant Scenes through SonificationProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596098:2(1-32)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659609
Sheshadri SHara K(2024)Conversational LocalizationProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314047:4(1-32)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631404
Hwang HJung HGiudice NBiswas JLee SKim D(2024)Towards Robotic Companions: Understanding Handler-Guide Dog Interactions for Informed Guide Dog Robot DesignProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642181(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642181
Karimi PBrady EMartin-Hammond ABolchini D(2023)“I stepped into a puddle”: Non-Visual Texting in Nomadic ContextsProceedings of the 20th International Web for All Conference10.1145/3587281.3587285(32-43)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3587281.3587285
Jain GTeng YCho DXing YAziz MSmith B(2023)"I Want to Figure Things Out": Supporting Exploration in Navigation for People with Visual ImpairmentsProceedings of the ACM on Human-Computer Interaction10.1145/35794967:CSCW1(1-28)Online publication date: 16-Apr-2023
https://dl.acm.org/doi/10.1145/3579496
He TNiu QLiu N(2023)GC-LocProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694956:4(1-27)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3569495
Gordon JFiannaca AKneisel MCutrell EMiller AGonzalez-Franco M(2023)Hearing the Way Forward: Exploring Ambient Navigational Awareness with Reduced Cognitive Load through Spatial Audio-ARExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585800(1-7)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585800
Alamri A(2023)Development of Ontology-Based Indoor Navigation Algorithm for Indoor Obstacle Identification for the Visually Impaired2023 9th International Conference on Engineering, Applied Sciences, and Technology (ICEAST)10.1109/ICEAST58324.2023.10157934(38-42)Online publication date: 1-Jun-2023
https://doi.org/10.1109/ICEAST58324.2023.10157934
Zheng BLiu FZhang MZhou TCui SYe YGuo Y(2023)Image captioning for cultural artworks: a case study on ceramicsMultimedia Systems10.1007/s00530-023-01178-829:6(3223-3243)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1007/s00530-023-01178-8
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents