Towards Automatic Oracle Prediction for AR Testing: Assessing Virtual Object Placement Quality under Real-World Scenes
Authors: 
Xiaoyi Yang, Yuxing Wang, Tahmid Rafi, Dongfang Liu, Xiaoyin Wang, Xueling ZhangAuthors Info & Claims
Xiaoyi Yang, Yuxing Wang, Tahmid Rafi, Dongfang Liu, Xiaoyin Wang, Xueling ZhangAuthors Info & ClaimsPages 717 - 729
Abstract
Augmented Reality (AR) technology opens up exciting possibilities in various fields, such as education, work guidance, shopping, communication, and gaming. However, users often encounter usability and user experience issues in current AR apps, often due to the imprecise placement of virtual objects. Detecting these inaccuracies is crucial for AR app testing, but automating the process is challenging due to its reliance on human perception and validation. This paper introduces VOPA (Virtual Object Placement Assessment), a novel approach that automatically identifies imprecise virtual object placements in real-world AR apps. VOPA involves instrumenting real-world AR apps to collect screenshots representing various object placement scenarios and their corresponding metadata under real-world scenes. The collected data are then labeled through crowdsourcing and used to train a hybrid neural network that identifies object placement errors. VOPA aims to enhance AR app testing by automating the assessment of virtual object placement quality and detecting imprecise instances. In our evaluation of a test set of 304 screenshots, VOPA achieved an accuracy of 99.34%, precision of 96.92% and recall of 100%. Furthermore, VOPA successfully identified 38 real-world object placement errors, including instances where objects were hovering between two surfaces or appearing embedded in the wall.
References
[1]
2015. Vuforia. https://www.ptc.com/en/products/vuforia
[2]
2015. Wikitude. https://www.wikitude.com/
[3]
2018. Apple ARKit. https://developer.apple.com/augmented-reality/
[4]
2018. MarketsandMarkets. https://www.marketsandmarkets.com/Market-Reports/augmented-reality-virtual-reality-market-1185.html
[5]
2018. Microsoft Hololens. https://www.microsoft.com/en-us/hololens
[6]
2018. Oculus Rift. https://www.oculus.com/
[7]
2020. Sceneform. https://developers.google.com/sceneform/develop
[8]
2020. Unity MARS. https://unity.com/products/unity-mars
[9]
2021. Google AR Core. https://developers.google.com/ar
[10]
2021. Recording and playback introduction. https://developers.google.com/ar/develop/recording-and-playback
[11]
2022. AR Apps. http://bit.ly/GooglePlay_ArApps
[12]
2022. AR Games. http://bit.ly/GooglePlay_ArGames
[13]
2023. Amazon’s AR Tool is a useful way to see products before you buy them. https://www.dailydot.com/debug/amazon-ar-view-app/
[14]
2023. Android Debug Bridge. https://developer.android.com/tools/adb
[15]
2023. Avoid The Dreaded Ikea Fight With This New App. https://www.fastcompany.com/90143908/avoid-killing-your-relationship-at-ikea-with-this-new-app
[16]
2023. Filters on Google Play. https://developer.android.com/google/play/filters Google Play Filter
[17]
2023. Get discovered on Google Play search. https://support.google.com/googleplay/android-developer/answer/4448378?hl=en Google Play Search
[18]
2023. Get the lighting right. https://developers.google.com/ar/develop/lighting-estimation
[19]
2023. Google Translate,. https://play.google.com/store/apps/details?id=com.google.android.apps.translate&hl=en_US&gl=US
[20]
2023. Hit-tests place virtual objects in the real world. https://developers.google.com/ar/develop/hit-test
[21]
2023. HitResult. https://developers.google.com/ar/reference/java/com/google/ar/core/HitResult
[22]
2023. IKEA Place,. https://www.ikea.com/au/en/customer-service/mobile-apps/say-hej-to-ikea-place-pub1f8af050
[23]
2023. IL2CPP Overview. https://docs.unity3d.com/Manual/IL2CPP.html IL2CPP
[24]
2023. Light Meter LM-3000. https://apps.apple.com/us/app/light-meter-lm-3000/id1554264761
[25]
2023. Pokémon Go,. https://pokemongolive.com/?hl=en
[26]
2023. Screenshot touch. https://play.google.com/store/apps/details?id=com.mdiwebma.screenshot&hl=en_US&gl=US
[27]
2023. Snapchat Lenses,. https://lens.snapchat.com/?locale=en-US
[28]
2023. VOPA: Virtual Object Placement Assessment. https://sites.google.com/view/vopa-for-artesting/home VOPA
[29]
2023. Working with Anchors. https://developers.google.com/ar/develop/anchors
[30]
Amazon. 2005. Amazon Mechanical Turk. https://www.mturk.com
[31]
Amazon. 2005. Amazon Mechanical Turk. https://www.mturk.com/
[32]
Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering, 41, 5 (2015), 507–525. https://doi.org/10.1109/TSE.2014.2372785
[33]
Ke Chen, Yufei Li, Yingfeng Chen, Changjie Fan, Zhipeng Hu, and Wei Yang. 2021. Glib: towards automated test oracle for graphically-rich applications. 1093–1104.
[34]
Tsong Y Chen, Shing C Cheung, and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543.
[35]
Yiming Cui, Liqi Yan, Zhiwen Cao, and Dongfang Liu. 2021. Tf-blender: Temporal feature blender for video object detection. 8138–8147.
[36]
Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. 2021. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34 (2021), 3965–3977.
[37]
Google Developers. 2021. Frame. https://developers.google.com/ar/reference/java/com/google/ar/core/Frame
[38]
Google Developers. 2021. Pose. https://developers.google.com/ar/reference/java/com/google/ar/core/Pose
[39]
Google Developers. 2022. Overview of ARCore and supported development environments. https://developers.google.com/ar/develop
[40]
Google Developers. 2022. Perform hit-tests in your Android app. https://developers.google.com/ar/develop/java/hit-test/developer-guide
[41]
Google Developers. 2023. Fundamental concepts. https://developers.google.com/ar/develop/fundamentals
[42]
Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K Lahiri. 2022. TOGA: a neural method for test oracle generation. 2130–2141.
[43]
Hugh Durrant-Whyte and Tim Bailey. 2006. Simultaneous localization and mapping: part I. IEEE robotics & automation magazine, 13, 2 (2006), 99–110.
[44]
Afonso Fontes and Gregory Gay. 2021. Using machine learning to generate test oracles: a systematic literature review. 1–10.
[45]
Gordon Fraser and Andreas Zeller. 2010. Mutation-driven generation of unit tests and oracles. 147–158.
[46]
Henry Fuchs, Mark A Livingston, Ramesh Raskar, Kurtis Keller, Jessica R Crawford, Paul Rademacher, Samuel H Drake, and Anthony A Meyer. 1998. Augmented reality visualization for laparoscopic surgery. 934–943.
[47]
Nidhi Gowdra, Roopak Sinha, Stephen MacDonell, and WeiQi Yan. 2021. Maximum Categorical Cross Entropy (MCCE): A noise-robust alternative loss function to mitigate racial bias in Convolutional Neural Networks (CNNs) by reducing overfitting.
[48]
Margherita Grandini, Enrico Bagli, and Giorgio Visani. 2020. Metrics for multi-class classification: an overview. arXiv preprint arXiv:2008.05756.
[49]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. https://doi.org/10.48550/ARXIV.1512.03385
[50]
Hirokazu Kato and Mark Billinghurst. 1999. Marker tracking and hmd calibration for a video-based augmented reality conferencing system. 85–94.
[51]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
[52]
Pingfan Kong, Li Li, Jun Gao, Kui Liu, Tegawendé F Bissyandé, and Jacques Klein. 2018. Automated testing of android apps: A systematic literature review. IEEE Transactions on Reliability, 68, 1 (2018), 45–66.
[53]
William B Langdon, Shin Yoo, and Mark Harman. 2017. Inferring automatic test oracles. 5–6.
[54]
Dongfang Liu, Yiming Cui, Wenbo Tan, and Yingjie Chen. 2021. Sg-net: Spatial granularity network for one-stage video instance segmentation. 9816–9825.
[55]
Dongfang Liu, Yiming Cui, Liqi Yan, Christos Mousas, Baijian Yang, and Yingjie Chen. 2021. Densernet: Weakly supervised visual localization using multi-scale feature aggregation. 35, 6101–6109.
[56]
Lutz Lorenz, Philipp Kerschbaum, and Josef Schumann. 2014. Designing take over scenarios for automated driving: How does augmented reality support the driver to get back into the loop? 58, 1681–1685.
[57]
Michael R Lyu, Irwin King, TT Wong, Edward Yau, and PW Chan. 2005. Arcade: Augmented reality computing arena for digital entertainment. 1–9.
[58]
Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: Multi-objective automated testing for android applications. 94–105.
[59]
Winter Mason and Sidharth Suri. 2011. How to use mechanical turk for cognitive science research. 33.
[60]
Zeljko Medenica, Andrew L Kun, Tim Paek, and Oskar Palinko. 2011. Augmented reality vs. street views: a driving simulator study comparing two emerging navigation aids. 265–274.
[61]
Shabnam Mirshokraie, Ali Mesbah, and Karthik Pattabiraman. 2015. JSeft: Automated JavaScript unit test generation. 1–10.
[62]
Keiron O’Shea and Ryan Nash. 2015. An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458.
[63]
Mauro Pezzè and Cheng Zhang. 2014. Chapter One - Automated Test Oracles: A Survey. Advances in Computers, Vol. 95. Elsevier, 1–48.
[64]
Iulian Radu. 2012. Why should my students use AR? A comparative review of the educational impacts of augmented-reality. 313–314.
[65]
Tahmid Rafi, Xueling Zhang, and Xiaoyin Wang. 2022. PredART: Towards Automatic Oracle Prediction of Object Placements in Augmented Reality Testing. 1–13.
[66]
Hassan Ramchoun, Youssef Ghanou, Mohamed Ettaouil, and Mohammed Amine Janati Idrissi. 2016. Multilayer perceptron: Architecture optimization and training.
[67]
Franziska Roesner, Tadayoshi Kohno, and David Molnar. 2014. Security and privacy for augmented reality systems. Commun. ACM, 57, 4 (2014), 88–96.
[68]
Jim Scheibmeir and Yashwant K. Malaiya. 2019. Quality Model for Testing Augmented Reality Applications. 0219–0226. https://doi.org/10.1109/UEMCON47517.2019.8992974
[69]
Engineering ToolBox. 2004. Illuminance - Recommended Light Level. https://www.engineeringtoolbox.com/light-level-rooms-d_708.html
[70]
Connor Tumbleson and Ryszard WiÅŻniewski. 2017. Apktool-A tool for reverse engineering 3rd party, closed, binary Android apps.
[71]
Xiaoyin Wang. 2022. VRTest: an extensible framework for automatic testing of virtual reality scenes. 232–236.
[72]
Wikipedia. 2023. Simultaneous localization and mapping. https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping
[73]
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, and Simon Kornblith. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. 23965–23998.
[74]
Razieh Nokhbeh Zaeem, Mukul R Prasad, and Sarfraz Khurshid. 2014. Automated generation of oracles for testing user-interaction features of mobile apps. 183–192.
[75]
Yan Zheng, Xiaofei Xie, Ting Su, Lei Ma, Jianye Hao, Zhaopeng Meng, Yang Liu, Ruimin Shen, Yingfeng Chen, and Changjie Fan. 2019. Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. 772–784.
[76]
Yi Zhong, Mengyu Shi, Youran Xu, Chunrong Fang, and Zhenyu Chen. 2023. Iterative Android automated testing. Frontiers of Computer Science, 17, 5 (2023), 175212.
Index Terms
- Towards Automatic Oracle Prediction for AR Testing: Assessing Virtual Object Placement Quality under Real-World Scenes
Recommendations
PredART: Towards Automatic Oracle Prediction of Object Placements in Augmented Reality Testing
ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software EngineeringWhile the emerging Augmented Reality (AR) technique allows a lot of new application opportunities, from education and communication to gaming, current augmented apps often have complaints about their usability and/or user experience due to placement ...
Miniature AR: Multi-view 6DOF Virtual Object Visualization for a Miniature Diorama
VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and TechnologyWe describe a miniature diorama AR system called ‘Miniature AR’ which can be applied to a mechanical diorama to extend the content’s feasibility by overlapping virtual objects on the complex diorama structure. The previous AR researches for the diorama ...
Can Haptic Feedback on One Virtual Object Increase the Presence of Another Virtual Object?
VRST '22: Proceedings of the 28th ACM Symposium on Virtual Reality Software and TechnologyThis paper investigated whether increased presence from experiencing haptic feedback on one virtual object can transfer to another virtual object. Two similar studies were run in different environments: an immersive virtual environment and a mixed ...
Comments
Information & Contributors
Information
Published In
September 2024
1928 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 September 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ISSTA '24
Sponsor:
ISSTA '24: 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis
September 16 - 20, 2024
Vienna, Austria
Acceptance Rates
Overall Acceptance Rate 58 of 213 submissions, 27%
Upcoming Conference
ISSTA '25
- Sponsor:
- sigsoft
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 42Total Downloads
- Downloads (Last 12 months)42
- Downloads (Last 6 weeks)42
Reflects downloads up to 04 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in