Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3441852.3471225acmconferencesArticle/Chapter ViewAbstractPublication PagesassetsConference Proceedingsconference-collections
research-article

Disability-first Dataset Creation: Lessons from Constructing a Dataset for Teachable Object Recognition with Blind and Low Vision Data Collectors

Published: 17 October 2021 Publication History

Abstract

Artificial Intelligence (AI) for accessibility is a rapidly growing area, requiring datasets that are inclusive of the disabled users that assistive technology aims to serve. We offer insights from a multi-disciplinary project that constructed a dataset for teachable object recognition with people who are blind or low vision. Teachable object recognition enables users to teach a model objects that are of interest to them, e.g., their white cane or own sunglasses, by providing example images or videos of objects. In this paper, we make the following contributions: 1) a disability-first procedure to support blind and low vision data collectors to produce good quality data, using video rather than images; 2) a validation and evolution of this procedure through a series of data collection phases and 3) a set of questions to orient researchers involved in creating datasets toward reflecting on the needs of their participant community.

Supplementary Material

VTT File (TP5715.vtt)
VTT File (TP5715.vtt)
Supplemental materials (5715-file2.zip)
MP4 File (TP5715.mp4)
Video presentation
MP4 File (TP5715.mp4)
Video presentation

References

[1]
Dustin Adams, Sri Kurniawan, Cynthia Herrera, Veronica Kang, and Natalie Friedman. 2016. Blind photographers and viz snap: A long-Term study. ASSETS 2016 - Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility, 201–208. https://doi.org/10.1145/2982142.2982169
[2]
Dustin Adams, Lourdes Morales, and Sri Kurniawan. 2013. A qualitative study to support a blind photography mobile application. ACM International Conference Proceeding Series, 1–8. https://doi.org/10.1145/2504335.2504360
[3]
Subeida Ahmed, Cecily Morrison, Harshadha Balasubramanian, Abigail Sellen, Simone Stumpf, and Martin Grayson. 2020. Investigating the Intelligibility of a Computer Vision System for Blind Users. (2020), 11. https://doi.org/10.1145/3377325.3377508
[4]
Dragan Ahmetovic, Daisuke Sato, Uran Oh, Tatsuya Ishihara, Kris Kitani, and Chieko Asakawa. 2020. ReCog: Supporting Blind People in Recognizing Personal Objects. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–12. https://doi.org/10.1145/3313831.3376143
[5]
Cynthia L Bennett. 2020. Authentic Intelligence: A Blind Researcher Bringing Wisdom to the Future of Technology Innovations. https://www.bennettc.com/wp-content/uploads/2020/07/NFB_2020_Speech_Final.pdf Transcript of a talk at the NFB, Retrieved September, 2020.
[6]
Cynthia L. Bennett, E. Jane, Martez E. Mott, Edward Cutrell, and Meredith Ringel Morris. 2018. How teens with visual impairments take, edit, and share photos on social media. Conference on Human Factors in Computing Systems - Proceedings 2018-April, 1–12. https://doi.org/10.1145/3173574.3173650
[7]
Cynthia L Bennett and Daniela K Rosner. 2019. The Promise of Empathy: Design, Disability, and Knowing the” Other”. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13. https://doi.org/10.1145/3290605.3300528
[8]
Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samuel White, and Tom Yeh. 2010. VizWiz: Nearly real-time answers to visual questions. UIST 2010 - 23rd ACM Symposium on User Interface Software and Technology, 333–342. https://doi.org/10.1145/1866029.1866080
[9]
Jeffrey P. Bigham, Chandrika Jayant, Andrew Miller, Brandyn White, and Tom Yeh. 2010. VizWiz::LocateIt - Enabling blind people to locate objects in their environment. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, CVPRW 2010, 65–72. https://doi.org/10.1109/CVPRW.2010.5543821
[10]
Erin L Brady, Yu Zhong, Meredith Ringel Morris, and Jeffrey P Bigham. 2013. Investigating the appropriateness of social network question asking as a resource for blind users. In Proceedings of the 2013 conference on Computer supported cooperative work. 1225–1236. https://dl.acm.org/doi/abs/10.1145/2441776.2441915
[11]
Danielle Bragg, Naomi Caselli, John W Gallagher, Miriam Goldberg, Courtney J Oka, and William Thies. 2021. ASL Sea Battle: Gamifying Sign Language Data Collection. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–13.
[12]
Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa Verhoef, Christian Vogler, and Meredith Ringel Morris. 2019. Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19). Association for Computing Machinery, New York, NY, USA, 16–31. https://doi.org/10.1145/3308561.3353774
[13]
John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, and Richard Turner. 2020. Tasknorm: Rethinking batch normalization for meta-learning. In International Conference on Machine Learning. PMLR, 1153–1164.
[14]
Morrison Cecily, Edward Cutrell, Martin Grayson, Thieme Anja, Alex S. Taylor, Geert Roumen, Camilla Longden, Rita Marques, Abigail Sellen, and Sebastian Tschiatschek. 2021. Social Sensemaking with AI: Designing an Open-ended AI experience with a Blind Child. In The 2021 CHI Proceedings on Human Factors in Computing Systems. 1–12.
[15]
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. 2017. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence 40, 4(2017), 834–848.
[16]
Microsoft Corporation. 2016. SeeingAI. https://www.microsoft.com/en-us/ai/seeing-ai Retrieved September, 2020.
[17]
deeplearning.ai. 2019. AI For Everyone | Coursera. https://www.coursera.org/learn/ai-for-everyone?utm_source=gg&utm_medium=sem&utm_content=08-AIforEveryone-ROW&campaignid=9727679885&adgroupid=99187762066&device=c&keyword=artificial%20intelligence%20and%20machine%20learning&matchtype=b&network=g&devicemodel=&adpostion=&creativeid=428167449287&hide_mobile_promo&gclid=CjwKCAjw_qb3BRAVEiwAvwq6Vm_0ODrr8pKM8y3rrBvUoBlD7920KDsNb-JC1ajRb9uzeEEwbAFHtxoCzIQQAvD_BwE#syllabus
[18]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks.
[19]
C Grady. [n.d.]. Ethical and practical considerations of paying research participants. Department of Clinical Bioethics Clinical Center/NIH ([n. d.]).
[20]
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, and Meredith Ringel Morris. 2019. Toward Fairness in AI for People with Disabilities: A Research Roadmap. (7 2019). http://arxiv.org/abs/1907.02227
[21]
Danna Gurari, Qing Li, Chi Lin, Yinan Zhao, Anhong Guo, Abigale Stangl, and Jeffrey P. Bigham. 2019. VizWiz-Priv: A Dataset for Recognizing the Presence and Purpose of Private Visual Information in Images Taken by Blind People. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[22]
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3608–3617.
[23]
Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P Bigham. 2018. A data-driven analysis of workers’ earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–14.
[24]
Susumu Harada, Daisuke Sato, Dustin W. Adams, Sri Kurniawan, Hironobu Takagi, and Chieko Asakawa. 2013. Accessible photo album: Enhancing the photo sharing experience for people with visual impairment. Conference on Human Factors in Computing Systems - Proceedings, 2127–2136. https://doi.org/10.1145/2470654.2481292
[25]
Julia Himmelsbach, Stephanie Schwarz, Cornelia Gerdenitsch, Beatrix Wais-Zechmann, Jan Bobeth, and Manfred Tscheligi. 2019. Do We Care About Diversity in Human Computer Interaction: A Comprehensive Content Analysis on Diversity Dimensions in Research. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–16.
[26]
Jonggi Hong, Kyungjun Lee, June Xu, and Hernisa Kacorri. 2020. Crowdsourcing the Perception of Machine Teaching. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.
[27]
Chandrika Jayant, Hanjie Ji, Samuel White, and Jeffrey P. Bigham. 2011. Supporting blind photography. ASSETS’11: Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility, 203–210. https://doi.org/10.1145/2049536.2049573
[28]
Hernisa Kacorri. 2017. Teachable machines for accessibility. ACM SIGACCESS Accessibility and Computing119 (2017), 10–18.
[29]
Hernisa Kacorri, Utkarsh Dwivedi, Sravya Amancherla, Mayanka K. Jha, and Riya Chanduka. 2020. IncluSet: A Data Surfacing Repository for Accessibility Datasets. ASSETS 2020 - 22nd International ACM SIGACCESS Conference on Computers and Accessibility, 6.
[30]
Hernisa Kacorri, Kris M. Kitani, Jeffrey P. Bigham, and Chieko Asakawa. 2017. People with visual impairment training personal object recognizers: Feasibility and challenges. Conference on Human Factors in Computing Systems - Proceedings 2017-May, 5839–5849. https://doi.org/10.1145/3025453.3025899
[31]
Amlan Kar, Nishant Rai, Karan Sikka, and Gaurav Sharma. 2017. Adascan: Adaptive scan pooling in deep convolutional neural networks for human action recognition in videos. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3376–3385.
[32]
Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money: Worker motivation in crowdsourcing-a study on Mechanical Turk. 11 (2011), 1–11.
[33]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105.
[34]
Todd Kulesza, Margaret Burnett, Weng Keen Wong, and Simone Stumpf. 2015. Principles of Explanatory Debugging to personalize interactive machine learning. International Conference on Intelligent User Interfaces, Proceedings IUI 2015-January, 126–137. https://doi.org/10.1145/2678025.2701399
[35]
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. 2011. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society, Vol. 33.
[36]
MIT Open Learning. 2019. aik12-MIT. https://aieducation.mit.edu/ Retrieved April, 2021.
[37]
Kyungjun Lee, Jonggi Hong, Simone Pimento, Ebrima Jarjue, and Hernisa Kacorri. 2019. Revisiting blind photography in the context of teachable object recognizers. ASSETS 2019 - 21st International ACM SIGACCESS Conference on Computers and Accessibility, 83–95. https://doi.org/10.1145/3308561.3353799
[38]
Daniela Massiceti, Lida Theodorou, Luisa Zintgraf, Matthew Tobias Harris, Simone Stumpf, Cecily Morrison, Edward Cutrell, and Katja Hofmann. 2021. ORBIT: A real-world few-shot dataset for teachable object recognition collected from people who are blind or low vision. https://doi.org/10.25383/city.14294597.v1 City, University of London, Dataset.
[39]
Daniela Massiceti, Luisa Zintgraf, John Bronskill, Matthew Tobias Harris, Edward Cutrell, Cecily Morrison, Katja Hofmann, and Simone Stumpf. 2021. ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition. arXiv preprint arXiv:2104.03841(2021).
[40]
Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences., 38 pages. https://doi.org/10.1016/j.artint.2018.07.007
[41]
Meredith Ringel Morris. 2020. AI and accessibility. Commun. ACM 63 (5 2020), 35–37. Issue 6. https://doi.org/10.1145/3356727
[42]
Joon Sung Park, Danielle Bragg, Ece Kamar, and Meredith Ringel Morris. 2021. Designing an Online Infrastructure for Collecting AI Data From People With Disabilities. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 52–63.
[43]
Jennifer Pearson, Simon Robinson, Thomas Reitmaier, Matt Jones, Shashank Ahire, Anirudha Joshi, Deepak Sahoo, Nimish Maravi, and Bhakti Bhikne. 2019. StreetWise: Smart speakers vs human help in public slum settings. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.
[44]
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. 2019. Fast and Flexible Multi-Task Classification using Conditional Neural Adaptive Processes. In Advances in Neural Information Processing Systems.
[45]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, 2015. Imagenet large scale visual recognition challenge. International journal of computer vision 115, 3 (2015), 211–252.
[46]
Manaswi Saha, Alexander J. Fiannaca, Melanie Kneisel, Edward Cutrell, and Meredith Ringel Morris. 2019. Closing the Gap: Designing for the Last-Few-Meters Wayfinding Problem for People with Visual Impairments. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility (Pittsburgh, PA, USA) (ASSETS ’19). Association for Computing Machinery, New York, NY, USA, 222–235. https://doi.org/10.1145/3308561.3353776
[47]
Elizabeth B-N Sanders and Pieter Jan Stappers. 2008. Co-creation and the new landscapes of design. Co-design 4, 1 (2008), 5–18.
[48]
Ashley Shew. 2020. Ableism, Technoableism, and Future AI. IEEE Technology and Society Magazine 39 (3 2020), 40–50+85. Issue 1. https://doi.org/10.1109/MTS.2020.2967492
[49]
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems.
[50]
Adrianna Surmiak. 2020. Ethical Concerns of Paying Cash to Vulnerable Participants: The Qualitative Researchers’ Views. The Qualitative Report 25, 12 (2020), 4461–4480.
[51]
Mingxing Tan and Quoc Le. 2019. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In International Conference on Machine Learning. 6105–6114.
[52]
David Touretzky, Christina Gardner-McCune, Cynthia Breazeal, Fred Martin, and Deborah Seehorn. 2019. A year in K-12 AI education. AI Magazine 40 (12 2019), 88–90. Issue 4. https://doi.org/10.1609/aimag.v40i4.5289
[53]
Shari Trewin, Sara Basson, Michael Muller, Stacy Branham, Jutta Treviranus, Daniel Gruen, Daniel Hebert, Natalia Lyckowski, and Erich Manser. 2019. Considerations for AI fairness for people with disabilities. AI Matters 5, 3 (2019), 40–63.
[54]
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, 2019. Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.03096(2019).
[55]
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, 2016. Matching networks for one shot learning.
[56]
Meredith Whittaker, Meryl Alper, Cynthia L Bennett, Sara Hendren, Liz Kaziunas, Mara Mills, Meredith Ringel Morris, Joy Rankin, Emily Rogers, Marcel Salas, 2019. Disability, Bias, and AI. AI Now Institute, November(2019).
[57]
Stephanie Wilson, Abi Roper, Jane Marshall, Julia Galliers, Niamh Devane, Tracey Booth, and Celia Woolf. 2015. Codesign for people with aphasia through tangible design languages. CoDesign 11, 1 (2015), 21–34. https://doi.org/10.1080/15710882.2014.997744 arXiv:https://doi.org/10.1080/15710882.2014.997744
[58]
Shaomei Wu and Lada Adamic. 2014. Visually impaired users on an online social network. Conference on Human Factors in Computing Systems - Proceedings, 3133–3142. https://doi.org/10.1145/2556288.2557415
[59]
Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. (2020), 1–13. https://doi.org/10.1145/3313831.3376301
[60]
Chen Zhu, Xiao Tan, Feng Zhou, Xiao Liu, Kaiyu Yue, Errui Ding, and Yi Ma. 2018. Fine-grained video categorization with redundancy reduction attention. In Proceedings of the European Conference on Computer Vision (ECCV). 136–152.
[61]
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. 2019. Fast context adaptation via meta-learning. In International Conference on Machine Learning. PMLR, 7693–7702.

Cited By

View all
  • (2024)Artificial IntelligenceTransforming Special Education Through Artificial Intelligence10.4018/979-8-3693-5538-1.ch017(459-520)Online publication date: 20-Sep-2024
  • (2024)"I Want to Publicize My Stutter": Community-led Collection and Curation of Chinese Stuttered Speech DataProceedings of the ACM on Human-Computer Interaction10.1145/36870148:CSCW2(1-27)Online publication date: 8-Nov-2024
  • (2024)Identifying Crucial Objects in Blind and Low-Vision Individuals' NavigationProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3688538(1-8)Online publication date: 27-Oct-2024
  • Show More Cited By

Index Terms

  1. Disability-first Dataset Creation: Lessons from Constructing a Dataset for Teachable Object Recognition with Blind and Low Vision Data Collectors
          Index terms have been assigned to the content through auto-classification.

          Recommendations

          Comments

          Information & Contributors

          Information

          Published In

          cover image ACM Conferences
          ASSETS '21: Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility
          October 2021
          730 pages
          ISBN:9781450383066
          DOI:10.1145/3441852
          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

          Sponsors

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          Published: 17 October 2021

          Permissions

          Request permissions for this article.

          Check for updates

          Author Tags

          1. AI
          2. accessibility
          3. blind and low vision users
          4. datasets
          5. teachable object recognition

          Qualifiers

          • Research-article
          • Research
          • Refereed limited

          Conference

          ASSETS '21
          Sponsor:

          Acceptance Rates

          ASSETS '21 Paper Acceptance Rate 36 of 134 submissions, 27%;
          Overall Acceptance Rate 436 of 1,556 submissions, 28%

          Contributors

          Other Metrics

          Bibliometrics & Citations

          Bibliometrics

          Article Metrics

          • Downloads (Last 12 months)181
          • Downloads (Last 6 weeks)11
          Reflects downloads up to 28 Jan 2025

          Other Metrics

          Citations

          Cited By

          View all
          • (2024)Artificial IntelligenceTransforming Special Education Through Artificial Intelligence10.4018/979-8-3693-5538-1.ch017(459-520)Online publication date: 20-Sep-2024
          • (2024)"I Want to Publicize My Stutter": Community-led Collection and Curation of Chinese Stuttered Speech DataProceedings of the ACM on Human-Computer Interaction10.1145/36870148:CSCW2(1-27)Online publication date: 8-Nov-2024
          • (2024)Identifying Crucial Objects in Blind and Low-Vision Individuals' NavigationProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3688538(1-8)Online publication date: 27-Oct-2024
          • (2024)Misfitting With AI: How Blind People Verify and Contest AI ErrorsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675659(1-17)Online publication date: 27-Oct-2024
          • (2024)AccessShare: Co-designing Data Access and Sharing with Blind PeopleProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675612(1-16)Online publication date: 27-Oct-2024
          • (2024)From Information Seeking to Empowerment: Using Large Language Model Chatbot in Supporting Wheelchair Life in Low Resource SettingsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675609(1-18)Online publication date: 27-Oct-2024
          • (2024)ProgramAlly: Creating Custom Visual Access Programs via Multi-Modal End-User ProgrammingProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676391(1-15)Online publication date: 13-Oct-2024
          • (2024)From Fitting Participation to Forging Relationships: The Art of Participatory MLProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642775(1-9)Online publication date: 11-May-2024
          • (2024)“To us, it is still foreign”: AI and the disabled in the Global SouthAI & SOCIETY10.1007/s00146-024-02149-xOnline publication date: 2-Dec-2024
          • (2023)A Guide to Data Collection for Computation and Monitoring of Node Energy ConsumptionBig Data and Cognitive Computing10.3390/bdcc70301307:3(130)Online publication date: 11-Jul-2023
          • Show More Cited By

          View Options

          Login options

          View options

          PDF

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format.

          HTML Format

          Figures

          Tables

          Media

          Share

          Share

          Share this Publication link

          Share on social media