Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Study and Development of Machine Learning Models Designed for Extended Reality Interactivity in Real-Time

  • Conference paper
  • First Online:
HCI International 2024 – Late Breaking Papers (HCII 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15377))

Included in the following conference series:

  • 101 Accesses

Abstract

The continuous development of hardware has allowed immersive technologies to be accessible for application in various daily tasks. It is currently possible to access immersive experiences through mobile devices, head-mounted displays, and other technologies. Extended reality (XR) is the term that encompasses a spectrum of various immersive and interactive technologies. Interactions are part of the hard core of immersive experiences and provide the possibility of interacting with the world around us. However, there are currently some SDKs intended for machine learning, such as ML-Kit. This SDK includes functionalities such as face and object detection and tracking, pose detection, and so on. However, there is a diverse set of functionalities, which are not yet grouped within some reusable structure (API or SDK). Many of these functionalities have proposed solutions using deep learning techniques. A characteristic of these techniques is that they tend to be complex in their space-time dimension. For this and other reasons, they are usually more difficult techniques to adapt for real-time use and other limiting characteristics of devices that reproduce immersive experiences. This work aims to study a viable architecture in which a set of ML methods that can be adapted to work in XR environments can be organized and grouped. These methods will have special characteristics since they must be adapted to provide real-time responses to users’ interactions with the environment. That is, they must adjust to the characteristics and limitations of immersive environments. Within the study, some of the main SDKs for immersive environments are analyzed and several ML methods are analyzed, related to the areas of computer vision and speech recognition, which should be included within the proposed architecture.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Gupta, Y.P., Mukul, Gupta, N.: Deep learning model-based multimedia retrieval and its optimization in augmented reality applications. Multimed Tools Appl. 82, 8447–8466 (2023). https://doi.org/10.1007/s11042-022-13555-y

  2. Amin, D., Govilkar, S.: Comparative study of augmented reality Sdk’s. Int. J. Comput. Sci. Appl. 5, 11–26 (2015). https://doi.org/10.5121/ijcsa.2015.5102

    Article  Google Scholar 

  3. ML Kit|Google for Developers (2024). https://developers.google.com/ml-kit

  4. Alaskar, H., Saba, T.: Machine learning and deep learning: a comparative review. In: Singh Mer, K.K., Semwal, V.B., Bijalwan, V., Crespo, R.G. (eds.) Proceedings of Integrated Intelligence Enable Networks and Computing. Algorithms for Intelligent Systems. Springer, Singapore (2021). https://doi.org/10.1007/978-981-33-6307-6_15

  5. Kwon, H., et al.: XRBench: an extended reality (XR) machine learning benchmark suite for the metaverse. In: Proceedings of the 6th MLSys Conference, Miami Beach, FL, USA, 2023.2211.08675, arXiv, http://arxiv.org/abs/2211.08675/ (2023)

  6. Sahu, C., Young, C., Rai, R.: Artificial intelligence (AI) in augmented reality (AR)-assisted manufacturing applications: a review. Int. J. Prod. Res. 59(16), 4903–4959 (2021). https://doi.org/10.1080/00207543.2020.1859636

    Article  Google Scholar 

  7. Cao, J., Lam, K., Lee, L., Liu, X., Hui, P., Su, X.: Mobile Augmented Reality: User Interfaces, Frameworks, and Intelligence (2021). https://doi.org/10.1145/3557999

  8. Liberatore, M., Wagner, W.: Virtual, mixed, and augmented reality: a systematic review for immersive systems research. Virtual Reality 25, 773–799 (2021). https://doi.org/10.1007/s10055-020-00492-0

    Article  Google Scholar 

  9. Orji, J., Chan, G., Orji, R.: Augmented reality and machine learning in health: a systematic review, 59–67 (2023). https://doi.org/10.1145/3603421.3603430,

  10. Karacif, E., Gurer, E.A.: Decision support system proposal on the usage of extended reality SDKs. In: Architecture Symposium 16th DDAS (MSTAS) - Special Issue 2022 23, 17–30 (2022). https://doi.org/10.18038/estubtda.1165368

  11. Google. ARCore. https://developers.google.com/ar. Accessed 30 Mar 2023

  12. Syahputra, M., Hardywantara, F., Andayani, U.: Augmented reality virtual house model using ARCore technology based on android. J. Phys. Conf. Ser. (2018)

    Google Scholar 

  13. ARKit|Apple Developer Documentation (2023). https://developer.apple.com/documentation

  14. Ashour, Z., Yan, W.: BIM-powered augmented reality for advancing human-building interaction. eCAADe 1, 169–178 (2020)

    Google Scholar 

  15. AR Foundation|5.1.3 (2024). ttps://docs.unity3d.com/Packages/com.unity.xr.arfoundation@5.1/manual/index.html

    Google Scholar 

  16. Chaudhry, T., Juneja, A., Rastogi, S.: AR foundation for augmented reality in unity. Int. J. Adv. Eng. Manag. (IJAEM) 3(1), 662–667 (2021). www.ijaem.net ISSN: 2395–5252 https://doi.org/10.35629/5252-0301662667

  17. PTC. Vuforia Documentation. https://www.ptc.com/en/products/vuforia. Accessed 30 Mar 2023

  18. Yao, J., Lin, Y., Zhao, Y., Chang-lin, L., Yuan, P.: Augmented reality technology-based wind environment visualization. In: CAADRIA, pp. 369–377 (2018)

    Google Scholar 

  19. Gül, L.F.: Studying architectural massing strategies in co-design mobile augmented reality tool versus 3d virtual world. eCAADe 35 (2017)

    Google Scholar 

  20. Goepel, G.: Augmented construction: Impact and opportunity of mixed reality integration in architectural design implementation. In: ACADIA, pp. 430–437 (2019)

    Google Scholar 

  21. Upadhyay, G., Aggarwal, D., Bansal, A., Bhola, G.: Augmented reality and machine learning based product identification in retail using Vuforia and MobileNets. In: 2020 International Conference on Inventive Computation Technologies (ICICT), Coimbatore, India, pp. 479–485 (2020). https://doi.org/10.1109/ICICT48043.2020.9112490

  22. Lamb, P.: ARToolKit (2004). http://www.hitl.washington.edu/artoolkit/

  23. Amin, D., Govilkar, S.: Comparative study of augmented reality SDK’S. Int. J. Comput. Sci. Appl. (IJCSA) 5(1) (2015). https://doi.org/10.5121/ijcsa.2015.5102

  24. Lehman, S.M., Alrumayh, A.S., Kolhe, K., Ling, H., Tan, C.: Hidden in plain sight: exploring privacy risks of mobile augmented reality applications. ACM Trans. Priv. Secur. 25(4), 1–35 (2022). https://doi.org/10.1145/352402

    Article  Google Scholar 

  25. Chen, C., et al.: Privacy computing meets metaverse: necessity, taxonomy and challenges. Ad Hoc Netw. 103457 (2024). ISSN 1570–8705. https://doi.org/10.1016/j.adhoc.2024.103457 (https://www.sciencedirect.com/science/article/pii/S1570870524000684)

  26. Sharma, M., Kushwaha, P., Kumari, P., Kumari, P., Yadav, R.: Machine learning techniques in data fusion: a review. In: Sharma, H., Shrivastava, V., Bharti, K.K., Wang, L. (eds.) Communication and Intelligent Systems. ICCIS 2022. Lecture Notes in Networks and Systems, vol. 686. Springer, Singapore (2023). https://doi.org/10.1007/978-981-99-2100-3_31

  27. Prince, S.: Understanding Deep Learning. MIT Press (2024). https://mitpress.mit.edu

  28. Binns, R.: Algorithmic accountability, and public reason. Philosophy Technol. 31(4), 543–556 (2018)

    Article  Google Scholar 

  29. Grennan, L., et al.: Why businesses need explainable AI—and how to deliver it. McKinsey (2022)

    Google Scholar 

  30. Heikkilä, M.: Why business is booming for military AI startups. MIT Technol. Rev. 7 (2022)

    Google Scholar 

  31. David, H.: Why are there still so many jobs? the history and future of workplace automation. J. Econ. Perspect. 29(3), 3–30 (2015)

    Article  Google Scholar 

  32. Tegmark, M.: Life 3.0: Being human in the age of artificial intelligence. Vintage (2018)

    Google Scholar 

  33. Smith, M., Miller, S.: The ethical application of biometric facial recognition technology. AI Soc. 37(167–175), 2022 (2022)

    Google Scholar 

  34. Barrett, L.: Ban facial recognition technologies for children — and everyone else. Boston Univ. J. Sci. Technol. Law 26(2), 223–285 (2020)

    Google Scholar 

  35. Boulemtafes, A., Derhab, A., Challal, Y.: A review of privacy-preserving techniques for deep learning. Neurocomputing 384, 21–45 (2020)

    Article  Google Scholar 

  36. Wolford, B.: Editor in Chief, GDPR EU. General Data Protection Regulation (GDPR) (2024). https://gdpr.eu/what-is-gdpr/

Download references

Acknowledgments

This paper was presented as part of the results of the Project “SIDIA-M_AR_Internet_For_Bondi”, carried out by the Institute of Science and Technology - SIDIA, in partnership with Samsung Eletrônica da Amazônia LTDA, in accordance with the Information Technology Law n.8387/91 and article at the. 39 of Decree 10,521/2020.

Author information

Authors and Affiliations

  1. Sidia Institute of Science and Technology. Av. Darcy Vargas, 654, 69055-035, Manaus, Brasil

    Geovana Amorim Abensur, Agustín Alejandro Ortiz Díaz, Sergio Cleger Tamayo & Delrick Nunes De Oliveira

Authors
  1. Geovana Amorim Abensur
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Agustín Alejandro Ortiz Díaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sergio Cleger Tamayo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Delrick Nunes De Oliveira
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Agustín Alejandro Ortiz Díaz .

Editor information

Editors and Affiliations

  1. U.S. Army Research Laboratory, Adelphi, MD, USA

    Jessie Y. C. Chen

  2. U.S. Army Combat Capabilities, Orlando, FL, USA

    Gino Fragomeni

  3. Smart Future Initiative, Frankfurt, Germany

    Norbert A. Streitz

  4. Kyushu University, Fukuoka, Japan

    Shin'ichi Konomi

  5. DePaul University, Chicago, IL, USA

    Xiaowen Fang

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Abensur, G.A., Díaz, A.A.O., Cleger Tamayo, S., Nunes De Oliveira, D. (2025). Study and Development of Machine Learning Models Designed for Extended Reality Interactivity in Real-Time. In: Chen, J.Y.C., Fragomeni, G., Streitz, N.A., Konomi, S., Fang, X. (eds) HCI International 2024 – Late Breaking Papers. HCII 2024. Lecture Notes in Computer Science, vol 15377. Springer, Cham. https://doi.org/10.1007/978-3-031-76812-5_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-76812-5_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-76811-8

  • Online ISBN: 978-3-031-76812-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation