Abstract
Augmented Reality (AR) is a technology which enhances physical environments by superimposing digital data on top of a real-world view. AR has multiple applications and use cases, bringing digital data into the physical world enabling experiences such as training staff on complicated machinery without the risks that come with such activities. Numerous other uses have been developed including for entertainment, with AR games and cultural experiences now emerging. Recently, AR has been used for developing assistive technologies, with applications across a range of disabilities. To achieve the high-quality interactions expected by users, there has been increasing integration of AR with Machine Learning (ML) algorithms. This integration offers additional functionality to increase the scope of AR applications. In this paper we present the potential of integrating AR with ML algorithms for developing assistive technologies, for the use case of locating objects in the home context.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
World Report on Disability Summary. https://www.who.int/publications/i/item/WHO-NMH-VIP-11.01. Accessed 24 Feb 2022
Hung, S.-W., Chang, C.-W., Ma, Y.-C.: A new reality: exploring continuance intention to use mobile augmented reality for entertainment purposes. Technol. Soc. 67, 101757 (2021). https://doi.org/10.1016/j.techsoc.2021.101757
Li, X., Yi, W., Chi, H.-L., Wang, X., Chan, A.P.C.: A critical review of virtual and augmented reality (VR/AR) applications in construction safety. Autom. Constr. 86, 150–162 (2018). https://doi.org/10.1016/j.autcon.2017.11.003
“Assistive technology.” https://www.who.int/news-room/fact-sheets/detail/assistive-technology. Accessed 24 Feb 2022
“Labour Force Survey - Office for National Statistics.” https://www.ons.gov.uk/surveys/informationforhouseholdsandindividuals/householdandindividualsurveys/labourforcesurvey. Accessed 22 Feb 2022
“New Home gb - eSight.” https://esighteyewear.com/gb. Accessed 24 Feb 2022
“Envision Glasses.” https://www.letsenvision.com/envision-glasses. Accessed 24 Feb 2022
Laviole, J., Thevin, L., Albouys-Perrois, J., Brock, A.: Nectar. In: Proceedings of the Virtual Reality International Conference - Laval Virtual, April 2018, pp. 1–6 (2018). https://doi.org/10.1145/3234253.3234317
Chuang, T.-K., et al.: Deep trail-following robotic guide dog in pedestrian environments for people who are blind and visually impaired - learning from virtual and real worlds. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), May 2018, pp. 5849–5855. https://doi.org/10.1109/ICRA.2018.8460994
Albouys-Perrois, J., Laviole, J., Briant, C., Brock, A.M.: Towards a multisensory augmented reality map for blind and low vision people. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, April 2018, vol. 2018, pp. 1–14 (2018). https://doi.org/10.1145/3173574.3174203
Assistive Devices for People with Hearing or Speech Disorders | NIDCD. https://www.nidcd.nih.gov/health/assistive-devices-people-hearing-voice-speech-or-language-disorders. Accessed 22 Feb 2022
Tam, V., Gelsomini, M., Garzotto, F.: Polipo. In: Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction, March 2017, pp. 11–20 (2017). https://doi.org/10.1145/3024969.3025006
Stone, B.G., Mills, K.A., Saggers, B.: Online multiplayer games for the social interactions of children with autism spectrum disorder: a resource for inclusive education. Int. J. Incl. Educ. 23(2), 209–228 (2019). https://doi.org/10.1080/13603116.2018.1426051
EASY READING – Keeping the user at the digital original. https://www.easyreading.eu/. Accessed 24 Feb 2022
Pradhan, A., Mehta, K., Findlater, L.: Accessibility came by accident. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, April 2018, vol. 2018-April, pp. 1–13 (2018). https://doi.org/10.1145/3173574.3174033
Oliveira, R., Arriaga, P., Santos, F.P., Mascarenhas, S., Paiva, A.: Towards pro-social design: a scoping review of the use of robots and virtual agents to trigger prosocial behaviour. Comput. Hum. Behav. 114, 106547 (2021). https://doi.org/10.1016/j.chb.2020.106547
Coughlan, J.M., Miele, J.: AR4VI: AR as an accessibility tool for people with visual impairments. In: 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), October 2017, pp. 288–292 (2017). https://doi.org/10.1109/ISMAR-Adjunct.2017.89
Cavus, N., Al-Dosakee, K., Abdi, A., Sadiq, S.: The utilization of augmented reality technology for sustainable skill development for people with special needs: a systematic literature review. Sustainability 13(19), 10532 (2021). https://doi.org/10.3390/su131910532
Quintero, J., Baldiris, S., Rubira, R., Cerón, J., Velez, G.: Augmented reality in educational inclusion. A systematic review on the last decade. Front. Psychol. 10, 1835 (2019). https://doi.org/10.3389/FPSYG.2019.01835/BIBTEX
Yoon, C., et al.: Leveraging augmented reality to create apps for people with visual disabilities. In: The 21st International ACM SIGACCESS Conference on Computers and Accessibility, October 2019, pp. 210–221 (2019). https://doi.org/10.1145/3308561.3353788
Tian, Y., Yang, X., Yi, C., Arditi, A.: Toward a computer vision-based wayfinding aid for blind persons to access unfamiliar indoor environments. Mach. Vis. Appl. 24(3), 521–535 (2013). https://doi.org/10.1007/s00138-012-0431-7
Huang, J., Kinateder, M., Dunn, M.J., Jarosz, W., Yang, X.-D., Cooper, E.A.: An augmented reality sign-reading assistant for users with reduced vision. PLoS ONE 14(1), e0210630 (2019). https://doi.org/10.1371/journal.pone.0210630
Bakshi, A.M., Simson, J., de Castro, C., Yu, C.C., Dias, A.: Bright: an augmented reality assistive platform for visual impairment. In: 2019 IEEE Games, Entertainment, Media Conference (GEM), June 2019, pp. 1–4 (2019). https://doi.org/10.1109/GEM.2019.8811556
lo Bianco, M., Pedell, S., Renda, G.: Augmented reality and home modifications. In: Proceedings of the 28th Australian Conference on Computer-Human Interaction - OzCHI 2016, November 2016, pp. 499–507 (2016). https://doi.org/10.1145/3010915.3010929
Géron, A.: Hands-on machine learning with Scikit-Learn, Keras and TensorFlow: concepts, tools, and techniques to build intelligent systems (2019). https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/. Accessed 22 Feb 2022
Dabran, I., Avny, T., Singher, E., ben Danan, H.: Augmented reality speech recognition for the hearing impaired. In: 2017 IEEE International Conference on Microwaves, Antennas, Communications and Electronic Systems (COMCAS), November 2017, vol. 2017-November, pp. 1–4 (2017). https://doi.org/10.1109/COMCAS.2017.8244731
Zhang, Z., Geiger, J., Pohjalainen, J., Mousa, A.E.-D., Jin, W., Schuller, B.: Deep learning for environmentally robust speech recognition. ACM Trans. Intell. Syst. Technol. 9(5), 1–28 (2018). https://doi.org/10.1145/3178115
Gales, M., Young, S.: The application of hidden markov models in speech recognition. Found. Trends® Sig. Process. 1(3), 195–304 (2007). https://doi.org/10.1561/2000000004
Rabiner, L., Juang, B.: An introduction to hidden Markov models. IEEE ASSP Mag. 3(1), 4–16 (1986). https://doi.org/10.1109/MASSP.1986.1165342
Abdel-Hamid, O., Mohamed, A.R., Jiang, H., Deng, L., Penn, G., Yu, D.: Convolutional neural networks for speech recognition. IEEE Trans. Audio Speech Lang. Process. 22(10), 1533–1545 (2014). https://doi.org/10.1109/TASLP.2014.2339736
Nassif, A.B., Shahin, I., Attili, I., Azzeh, M., Shaalan, K.: Speech recognition using deep neural networks: a systematic review. IEEE Access 7, 19143–19165 (2019). https://doi.org/10.1109/ACCESS.2019.2896880
Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement, April 2018. http://arxiv.org/abs/1804.02767. Accessed 22 Feb 2022
Barakat, B., Steponenaite, A., Lall, G.S., Arshad, K., Wassell, I.J., Keates, S.: Assistive technology for the visually impaired: optimizing frame rate (freshness) to improve the performance of real-time objects detection application. In: Antona, M., Stephanidis, C. (eds.) HCII 2020. LNCS, vol. 12189, pp. 479–492. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49108-6_34
Barakat, B., et al.: 6G opportunities arising from internet of things use cases: a review paper. Future Internet 13(6), 159 (2021). https://doi.org/10.3390/fi13060159
Get Microsoft HoloLens - Microsoft Store en-GB. https://www.microsoft.com/en-gb/p/microsoft-hololens/9nblggh4qwnx?activetab=pivot:overviewtab. Accessed 25 Feb 2022
Microsoft. MixedReality-WebRTC Documentation. https://microsoft.github.io/MixedReality-WebRTC/. Accessed 25 Feb 2022
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Barakat, B., Hall, L., Keates, S. (2022). Integrating Machine Learning with Augmented Reality for Accessible Assistive Technologies. In: Antona, M., Stephanidis, C. (eds) Universal Access in Human-Computer Interaction. User and Context Diversity. HCII 2022. Lecture Notes in Computer Science, vol 13309. Springer, Cham. https://doi.org/10.1007/978-3-031-05039-8_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-05039-8_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-05038-1
Online ISBN: 978-3-031-05039-8
eBook Packages: Computer ScienceComputer Science (R0)