SIM2VR: Towards Automated Biomechanical Testing in VR
Authors: 
Florian Fischer, 
Aleksi Ikkala, 
Markus Klar, Arthur Fleig, Miroslav Bachinski, 
Roderick Murray-Smith, Perttu Hämäläinen, 
Antti Oulasvirta, 
Jörg MüllerAuthors Info & Claims
Florian Fischer, Aleksi Ikkala, Markus Klar, Arthur Fleig, Miroslav Bachinski, Roderick Murray-Smith, Perttu Hämäläinen, Antti Oulasvirta, Jörg MüllerAuthors Info & ClaimsArticle No.: 79, Pages 1 - 15
Abstract
Automated biomechanical testing has great potential for the development of VR applications, as initial insights into user behaviour can be gained in silico early in the design process. In particular, it allows prediction of user movements and ergonomic variables, such as fatigue, prior to conducting user studies. However, there is a fundamental disconnect between simulators hosting state-of-the-art biomechanical user models and simulators used to develop and run VR applications. Existing user simulators often struggle to capture the intricacies of real-world VR applications, reducing ecological validity of user predictions. In this paper, we introduce sim2vr, a system that aligns user simulation with a given VR application by establishing a continuous closed loop between the two processes. This, for the first time, enables training simulated users directly in the same VR application that real users interact with. We demonstrate that sim2vr can predict differences in user performance, ergonomics and strategies in a fast-paced, dynamic arcade game. In order to expand the scope of automated biomechanical testing beyond simple visuomotor tasks, advances in cognitive models and reward function design will be needed.
Supplemental Material
MP4 File
Video figure
- Download
- 36.54 MB
PDF File
Supplementary Material PDF (re-upload/updated version)
- Download
- 628.89 KB
PDF File
Appendix/Supplementary Manuscript with additional explanations and figures
- Download
- 628.89 KB
References
[1]
M. A. Ayoub, M. M. Ayoub, and A. G. Walvekar. 1974. A Biomechanical Model for the Upper Extremity using Optimization Techniques. Human Factors 16, 6 (1974), 585–594. https://doi.org/10.1177/001872087401600603 arXiv:https://doi.org/10.1177/001872087401600603PMID: 4442903.
[2]
Nikola Banovic, Tofi Buzali, Fanny Chevalier, Jennifer Mankoff, and Anind K. Dey. 2016. Modeling and Understanding Human Routine Behavior. Association for Computing Machinery, New York, NY, USA, 248–260. https://doi.org/10.1145/2858036.2858557
[3]
Allen Bierbaum, Patrick Hartling, and Carolina Cruz-Neira. 2003. Automated testing of virtual reality application interfaces. In Proceedings of the workshop on Virtual Environments 2003. 107–114.
[4]
Gunnar A Borg. 1982. Psychophysical bases of perceived exertion.Medicine and science in sports and exercise 14, 5 (1982), 377–381.
[5]
Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. 2022. MyoSuite–A contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205.13600 (2022).
[6]
Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Seungmoon Song, Yuval Tassa, Massimo Sartori, and Vikash Kumar. 2022. MyoChallenge: Learning contact-rich manipulation using a musculoskeletal hand. https://sites.google.com/view/myochallenge.
[7]
Stuart K. Card, Thomas P. Moran, and Allen Newell. 1983. The psychology of human-computer interaction. Crc Press.
[8]
Noshaba Cheema, Laura A. Frey-Law, Kourosh Naderi, Jaakko Lehtinen, Philipp Slusallek, and Perttu Hämäläinen. 2020. Predicting Mid-Air Interaction Movements and Fatigue Using Deep Reinforcement Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376701
[9]
Xiuli Chen, Gilles Bailly, Duncan P Brumby, Antti Oulasvirta, and Andrew Howes. 2015. The Emergence of Interactive Behaviour: A Model of Rational Menu Search. (2015).
[10]
Xiuli Chen, Sandra Dorothee Starke, Chris Baber, and Andrew Howes. 2017. A cognitive model of how people make decisions through interaction with visual displays. In Proceedings of the 2017 CHI conference on human factors in computing systems. 1205–1216.
[11]
Michael Damsgaard, John Rasmussen, Søren Tørholm Christensen, Egidijus Surma, and Mark de Zee. 2006. Analysis of musculoskeletal systems in the AnyBody Modeling System. Simulation Modelling Practice and Theory 14, 8 (2006), 1100–1111. https://doi.org/10.1016/j.simpat.2006.09.001 SIMS 2004.
[12]
Scott Delp, Frank Anderson, Allison Arnold, Peter Loan, A. Habib, Chand John, Eran Guendelman, and Darryl Thelen. 2007. OpenSim: Open-Source Software to Create and Analyze Dynamic Simulations of Movement. Biomedical Engineering, IEEE Transactions on 54 (12 2007), 1940 – 1950. https://doi.org/10.1109/TBME.2007.901024
[13]
Christopher L. Dembia, Nicholas A. Bianco, Antoine Falisse, Jennifer L. Hicks, and Scott L. Delp. 2021. OpenSim Moco: Musculoskeletal optimal control. PLOS Computational Biology 16, 12 (12 2021), 1–21. https://doi.org/10.1371/journal.pcbi.1008493
[14]
Konstantinos Dimitropoulos, Ioannis Hatzilygeroudis, and Konstantinos Chatzilygeroudis. 2022. A Brief Survey of Sim2Real Methods for Robot Learning. In Advances in Service and Industrial Robotics, Andreas Müller and Mathias Brandstötter (Eds.). Springer International Publishing, Cham, 133–140.
[15]
Florian Fischer, Miroslav Bachinski, Markus Klar, Arthur Fleig, and Jörg Müller. 2021. Reinforcement Learning Control of a Biomechanical Model of the Upper Extremity. Scientific Reports 11, 1 (2021), 1–15.
[16]
Florian Fischer, Arthur Fleig, Markus Klar, and Jörg Müller. 2022. Optimal Feedback Control for Modeling Human-Computer Interaction. ACM Trans. Comput.-Hum. Interact. 29, 6, Article 51 (april 2022), 70 pages. https://doi.org/10.1145/3524122
[17]
Paul M Fitts. 1954. The information capacity of the human motor system in controlling the amplitude of movement.Journal of experimental psychology 47, 6 (1954), 381.
[18]
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing. Ieee, 6645–6649.
[19]
Patrick Harms. 2019. Automated usability evaluation of virtual reality applications. ACM Transactions on Computer-Human Interaction (TOCHI) 26, 3 (2019), 1–36.
[20]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
[21]
Lorenz Hetzel, John Dudley, Anna Maria Feit, and Per Ola Kristensson. 2021. Complex Interaction as Emergent Behaviour: Simulating Mid-Air Virtual Keyboard Typing using Reinforcement Learning. IEEE Transactions on Visualization and Computer Graphics (2021), 1–1. https://doi.org/10.1109/TVCG.2021.3106494
[22]
Chris J Hunt, Guy Brown, and Gordon Fraser. 2014. Automatic testing of natural user interfaces. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. IEEE, 123–132.
[23]
Aleksi Ikkala, Florian Fischer, Markus Klar, Miroslav Bachinski, Arthur Fleig, Andrew Howes, Perttu Hämäläinen, Jörg Müller, Roderick Murray-Smith, and Antti Oulasvirta. 2022. Breathing Life Into Biomechanical User Models. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST ’22). Association for Computing Machinery, New York, NY, USA, Article 90, 14 pages. https://doi.org/10.1145/3526113.3545689
[24]
Aleksi Ikkala and Perttu Hämäläinen. 2020. Converting Biomechanical Models from OpenSim to MuJoCo., 277–281 pages. arxiv:2006.10618 [q-bio.QM] https://arxiv.org/abs/2006.10618
[25]
Sujin Jang, Wolfgang Stuerzlinger, Satyajit Ambike, and Karthik Ramani. 2017. Modeling Cumulative Arm Fatigue in Mid-Air Interaction Based on Perceived Exertion and Kinetics of Arm Motion. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 3328–3339. https://doi.org/10.1145/3025453.3025523
[26]
Jussi Jokinen, Aditya Acharya, Mohammad Uzair, Xinhui Jiang, and Antti Oulasvirta. 2021. Touchscreen typing as optimal supervisory control. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.
[27]
Antti Kangasrääsiö, Kumaripaba Athukorala, Andrew Howes, Jukka Corander, Samuel Kaski, and Antti Oulasvirta. 2017. Inferring Cognitive Models from Data Using Approximate Bayesian Computation. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 1295–1306. https://doi.org/10.1145/3025453.3025576
[28]
Kadiray Karakaya, Enes Yigitbas, and Gregor Engels. 2022. Automated UX Evaluation for User-Centered Design of VR Interfaces. In International Conference on Human-Centred Software Engineering. Springer, 140–149.
[29]
Markus Klar, Florian Fischer, Arthur Fleig, Miroslav Bachinski, and Jörg Müller. 2023. Simulating Interaction Movements via Model Predictive Control. ACM Trans. Comput.-Hum. Interact. 30, 3, Article 44 (jun 2023), 50 pages. https://doi.org/10.1145/3577016
[30]
Eileen Kowler. 2011. Eye movements: The past 25 years. Vision Research 51, 13 (2011), 1457–1483. https://doi.org/10.1016/j.visres.2010.12.014 Vision Research 50th Anniversary Issue: Part 2.
[31]
P. O. Kristensson and Th. Müllners. 2021. Design and Analysis of Intelligent Text Entry Systems with Function Structure Models and Envelope Analysis. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–12.
[32]
Francesco Lacquaniti, Carlo Terzuolo, and Paolo Viviani. 1983. The law relating the kinematic and figural aspects of drawing movements. Acta Psychologica 54, 1 (1983), 115 – 130.
[33]
Alexander Lavin, Hector Zenil, Brooks Paige, David Krakauer, Justin Gottschlich, Tim Mattson, Anima Anandkumar, Sanjay Choudry, Kamil Rocki, Atılım Güneş Baydin, 2021. Simulation Intelligence: Towards a New Generation of Scientific Methods. arXiv preprint arXiv:2112.03235 (2021).
[34]
Leng-Feng Lee and Brian R Umberger. 2016. Generating optimal control simulations of musculoskeletal movement using OpenSim and MATLAB. PeerJ 4 (2016), e1638.
[35]
Alexander C Li, Carlos Florensa, Ignasi Clavera, and Pieter Abbeel. 2019. Sub-policy adaptation for hierarchical reinforcement learning. arXiv preprint arXiv:1906.05862 (2019).
[36]
Qing Liu, Gustavo Alves, and Jian Zhao. 2023. Challenges and Opportunities for Software Testing in Virtual Reality Application Development. Graphics Interface 2023 -second deadline (2023).
[37]
John M Looft, Nicole Herkert, and Laura Frey-Law. 2018. Modification of a three-compartment muscle fatigue model to predict peak torque decline during intermittent tasks. Journal of biomechanics 77 (2018), 16–25.
[38]
J. Alberto Álvarez Martín, Henrik Gollee, Jörg Müller, and Roderick Murray-Smith. 2021. Intermittent control as a model of mouse movements. ACM Transactions on Computer-Human Interaction (TOCHI) 28, 5 (2021), 1–46.
[39]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, 2015. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529–533.
[40]
Jörg Müller, Antti Oulasvirta, and Roderick Murray-Smith. 2017. Control theoretic models of pointing. ACM Transactions on Computer-Human Interaction (TOCHI) 24, 4 (2017), 1–36.
[41]
Roderick Murray-Smith, Antti Oulasvirta, Andrew Howes, Jörg Müller, Aleksi Ikkala, Miroslav Bachinski, Arthur Fleig, Florian Fischer, and Markus Klar. 2022. What simulation can do for HCI research. Interactions 29, 6 (2022), 48–53.
[42]
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. 2018. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems 31 (2018).
[43]
Masaki Nakada, Tao Zhou, Honglin Chen, Tomer Weiss, and Demetri Terzopoulos. 2018. Deep Learning of Biomimetic Sensorimotor Control for Biomechanical Human Animation. ACM Trans. Graph. 37, 4, Article 56 (jul 2018), 15 pages. https://doi.org/10.1145/3197517.3201305
[44]
Antti Oulasvirta, Jussi P. P. Jokinen, and Andrew Howes. 2022. Computational Rationality as a Theory of Interaction. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems(CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 359, 14 pages. https://doi.org/10.1145/3491102.3517739
[45]
Melissa Quek. 2013. The role of simulation in developing and designing applications for 2-class motor imagery brain-computer interfaces. Ph. D. Dissertation. University of Glasgow.
[46]
M. Quek, D. Boland, J. Williamson, R. Murray-Smith, M. Tavella, S. Perdikis, Martijn Schreuder, and Michael Tangermann. 2011. Simulating the feel of brain-computer interfaces for design, development and social interaction. Proceedings of the 2011 Annual conference on Human factors in computing systems – CHI ’11 (2011), 25.
[47]
Ehsan Rashedi and Maury A Nussbaum. 2015. Mathematical models of localized muscle fatigue: sensitivity analysis and assessment of two occupationally-relevant models. PloS one 10, 12 (2015), e0143872.
[48]
Dhia Elhaq Rzig, Nafees Iqbal, Isabella Attisano, Xue Qin, and Foyzul Hassan. 2023. Virtual Reality (VR) Automated Testing in the Wild: A Case Study on Unity-Based VR Applications. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (Seattle, WA, USA) (ISSTA 2023). Association for Computing Machinery, New York, NY, USA, 1269–1281. https://doi.org/10.1145/3597926.3598134
[49]
Katherine R. Saul, Xiao Hu, Craig M. Goehler, Meghan E. Vidt, Melissa Daly, Anca Velisar, and Wendy M. Murray. 2014. Benchmarking of dynamic simulation predictions in two software platforms using an upper limb musculoskeletal model.Computer methods in biomechanics and biomedical engineering 5842, May 2016 (2014), 1–14. https://doi.org/10.1080/10255842.2014.916698
[50]
Pierre Schumacher, Daniel Haeufle, Dieter Büchler, Syn Schmitt, and Georg Martius. 2023. DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=C-xa_D3oTj6
[51]
Sofia Seinfeld, Tiare Feuchtner, Antonella Maselli, and Jörg Müller. 2020. User Representations in Human-Computer Interaction. Human–Computer Interaction 0, 0 (2020), 1–39. https://doi.org/10.1080/07370024.2020.1724790 arXiv:https://doi.org/10.1080/07370024.2020.1724790
[52]
Ajay Seth, Michael Sherman, Jeffrey A. Reinbolt, and Scott L. Delp. 2011. OpenSim: a musculoskeletal modeling and simulation framework for in silico investigations and exchange. Procedia IUTAM 2 (2011), 212–232. https://doi.org/10.1016/j.piutam.2011.04.021 IUTAM Symposium on Human Body Dynamics.
[53]
Hans Strasburger, Ingo Rentschler, and Martin Jüttner. 2011. Peripheral vision and pattern recognition: A review. Journal of Vision 11, 5 (12 2011), 13–13. https://doi.org/10.1167/11.5.13 arXiv:https://arvojournals.org/arvo/content_public/journal/jov/933487/jov-11-5-13.pdf
[54]
Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. 5026–5033. https://doi.org/10.1109/IROS.2012.6386109
[55]
Emanuel Todorov and Michael I. Jordan. 2002. Optimal feedback control as a theory of motor coordination. Nature Neuroscience 5, 11 (2002), 1226–1235. https://doi.org/10.1038/nn963
[56]
Brian R. Umberger, Karin G.M. Gerritsen, and Philip E. Martin. 2003. A Model of Human Muscle Energy Expenditure. Computer Methods in Biomechanics and Biomedical Engineering 6, 2 (2003), 99–111. https://doi.org/10.1080/1025584031000091678 arXiv:https://doi.org/10.1080/1025584031000091678PMID: 12745424.
[57]
Huawei Wang, Vittorio Caggiano, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. 2022. MyoSim: Fast and physiologically realistic MuJoCo models for musculoskeletal and exoskeletal studies. In 2022 International Conference on Robotics and Automation (ICRA). IEEE, 8104–8111.
[58]
Xiaoyin Wang. 2022. VRTest: an extensible framework for automatic testing of virtual reality scenes. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. 232–236.
[59]
DA Winter. 1984. Biomechanics of human movement with applications to the study of human locomotion. Critical reviews in biomedical engineering 9, 4 (1984), 287—314. http://europepmc.org/abstract/MED/6368126
[60]
Ting Xia and Laura A Frey Law. 2008. A theoretical approach for modeling peripheral muscle fatigue and recovery. Journal of biomechanics 41, 14 (2008), 3046–3052.
[61]
Zhuangdi Zhu, Kaixiang Lin, Anil K Jain, and Jiayu Zhou. 2020. Transfer learning in deep reinforcement learning: A survey. arXiv preprint arXiv:2009.07888 (2020).
Index Terms
- SIM2VR: Towards Automated Biomechanical Testing in VR
Recommendations
VR Grabbers: Ungrounded Haptic Retargeting for Precision Grabbing Tools
UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and TechnologyHaptic feedback in VR is important for realistic simulation in virtual reality. However, recreating the haptic experience for hand tools in VR traditionally requires hardware with precise actuators, adding complexity to the system. We propose Ungrounded ...
Effect of VR technology matureness on VR sickness
AbstractIn this paper relationship of perceived virtual reality (VR) sickness phenomenon with different generations of virtual reality head mounted displays (VR HMD) is presented. Action content type omnidirectional video clip was watched by means of four ...
Walk or Teleport? The Impact of VR Users' Locomotion Technique in Asymmetric VR-MR Setup
NordiCHI '24 Adjunct: Adjunct Proceedings of the 2024 Nordic Conference on Human-Computer InteractionAlthough locomotion within virtual reality (VR) is an integral environment navigation aspect, its impact is under-explored in asymmetric VR setups that support VR features along with mixed reality (MR) ones. In order to understand users’ needs in ...
Comments
Information & Contributors
Information
Published In
October 2024
2334 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution-ShareAlike International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
UIST '24
UIST '24: The 37th Annual ACM Symposium on User Interface Software and Technology
October 13 - 16, 2024
PA, Pittsburgh, USA
Acceptance Rates
Overall Acceptance Rate 561 of 2,567 submissions, 22%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 267Total Downloads
- Downloads (Last 12 months)267
- Downloads (Last 6 weeks)116
Reflects downloads up to 01 Jan 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in