Motor Skill Learning by Virtual Co-embodiment with an AI Teacher Trained in Human Teaching Behavior
Authors: 
Haruto Takita, Daiki Kodama, Yuji Hatada, Takuji Narumi, Michitaka HiroseAuthors Info & Claims
Haruto Takita, Daiki Kodama, Yuji Hatada, Takuji Narumi, Michitaka HiroseAuthors Info & ClaimsArticle No.: 5, Pages 1 - 9
Abstract
Virtual reality is gaining attention as a tool to facilitate motor skill learning. Numerous studies have been conducted on virtual co-embodiment, in which the movements of the teacher and student are weighted and averaged into a single avatar for motor skill learning. Previous studies have shown that virtual co-embodiment with a human teacher enhances motor skill learning efficiency, and the behavior of the human teacher is important for effective learning. However, this system has some challenges, such as the human playing the teacher’s role must be skilled in teaching and using virtual co-embodiment, and the teacher can be adversely affected by the learner. To solve these problems, we created an AI teacher using long short-term memory, which outputs the behavior of the teacher based on the input of the learner’s behavior data and the state of the experimental environment and trained the AI teacher by supervised learning using behavior training data of a human virtual co-embodiment. We confirmed that this AI teacher can generate behaviors similar to those of the human teacher and investigated the efficiency of motor skill learning using virtual co-embodiment with the AI teacher. Co-embodiment with the AI teacher reduced performance during the learning phase but improved performance during subsequent independent task execution. We further analyzed the assist proportion of the teacher and observed that when the co-embodied with an AI teacher, the assist proportion is lower than when the co-embodied with a human, suggesting that learning efficiency may be enhanced when the co-embodied partner is a mixture of supportive and obstructive.
References
[1]
Christian Bjørndal, Rob Gray, and Thomas Schack. 2024. Learning and Skill Acquisition in Sports: Theoretical Perspectives. https://doi.org/10.3389/978-2-8325-4370-2
[2]
Matthew Botvinick and Jonathan Cohen. 1998. Rubber hands ‘feel’ touch that eyes see. Nature 391, 6669 (1998), 756–756. https://doi.org/10.1038/35784
[3]
Jack Brookes, Faisal Mushtaq, Earle Jamieson, Aaron J. Fath, Geoffrey Bingham, Peter Culmer, Richard M. Wilkie, and Mark Mon-Williams. 2020. Exploring disturbance as a force for good in motor learning. PLOS ONE 15, 5 (05 2020), 1–21. https://doi.org/10.1371/journal.pone.0224055
[4]
Kuo Chang, Jia Zhang, Yang-Sheng Huang, Tzu-Chien Liu, and Yao-Ting Sung. 2019. Applying augmented reality in physical education on motor skills learning. Interactive Learning Environments 28 (06 2019), 1–13. https://doi.org/10.1080/10494820.2019.1636073
[5]
Rebecca Fribourg, Nami Ogawa, Ludovic Hoyet, Ferran Argelaguet, Takuji Narumi, Michitaka Hirose, and Anatole Lécuyer. 2021. Virtual Co-Embodiment: Evaluation of the Sense of Agency While Sharing the Control of a Virtual Body Among Two Individuals. IEEE Transactions on Visualization and Computer Graphics 27, 10 (2021), 4023–4038. https://doi.org/10.1109/TVCG.2020.2999197
[6]
Dr Tim Gabbett, Matthew Wake, and Bruce Abernethy. 2011. Use of dual-task methodology for skill assessment and development: Examples from rugby league. Journal of Sports Sciences 29, 1 (2011), 7–18. https://doi.org/10.1080/02640414.2010.514280
[7]
Tim J Gabbett and Bruce Abernethy. 2012. Dual-task assessment of a sporting skill: influence of task complexity and relationship with competitive performances. Journal of sports sciences 30, 16 (2012), 1735–1745. https://doi.org/10.1080/02640414.2012.713979
[8]
Shaun Gallagher. 2000. Philosophical conceptions of the self: implications for cognitive science. Trends in Cognitive Sciences 4, 1 (2000), 14–21. https://doi.org/10.1016/S1364-6613(99)01417-5
[9]
G. Ganesh, A. Takagi, R. Osu, T. Yoshioka, and Burdet E. Kawato, M.2014. Two is better than one: Physical interactions improve motor performance in humans. Scientific Reports 4, 3824 (2014). https://doi.org/10.1038/srep03824
[10]
Takayoshi Hagiwara, Gowrishankar Ganesh, Maki Sugimoto, Masahiko Inami, and Michiteru Kitazaki. 2020. Individuals Prioritize the Reach Straightness and Hand Jerk of a Shared Avatar over Their Own. Iscience (2020). https://doi.org/10.1016/j.isci.2020.101732
[11]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780.
[12]
Cyrine H’mida, Olivier Degrenne, Nafaa Souissi, Ghazi Rekik, Khaled Trabelsi, Mohamed Jarraya, Nicola Luigi Bragazzi, and Aïmen Khacharem. 2020. Learning a Motor Skill from Video and Static Pictures in Physical Education Students—Effects on Technical Performances, Motivation and Cognitive Load. International Journal of Environmental Research and Public Health 17, 23 (2020), 9067. https://doi.org/10.3390/ijerph17239067
[13]
Elmar Kal, Rens Prosée, Marinus Winters, and John Van Der Kamp. 2018. Does implicit motor learning lead to greater automatization of motor skills compared to explicit motor learning? A systematic review. PloS one 13, 9 (2018), e0203591.
[14]
Konstantina Kilteni, Raphaela Groten, and Mel Slater. 2012. The sense of embodiment in virtual reality. Presence: Teleoperators and Virtual Environments 21, 4 (2012), 373–387. https://doi.org/10.1162/PRES_a_00124
[15]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[16]
Daiki Kodama, Takato Mizuho, Yuji Hatada, Takuji Narumi, and Michitaka Hirose. 2023. Effects of Collaborative Training Using Virtual Co-embodiment on Motor Skill Learning. IEEE Transactions on Visualization and Computer Graphics (2023). https://doi.org/10.1109/TVCG.2023.3247112
[17]
Jaebong Lee and Seungmoon Choi. 2010. Effects of haptic guidance and disturbance on motor learning: Potential advantage of haptic disturbance. In 2010 IEEE Haptics Symposium. 335–342. https://doi.org/10.1109/HAPTIC.2010.5444635
[18]
Laura Marchal–Crespo, Jasmin Schneider, Lukas Jaeger, and Robert Riener. 2014. Learning a locomotor task: with or without errors?Journal of NeuroEngineering and Rehabilitation 11 (2014), 1–10. https://doi.org/10.1186/1743-0003-11-25
[19]
Harold Pashler. 1994. Dual-task interference in simple tasks: data and theory.Psychological bulletin 116, 2 (1994), 220.
[20]
Ali Utku Pehlivan, Dylan P. Losey, and Marcia K. O’Malley. 2016. Minimal Assist-as-Needed Controller for Upper Limb Robotic Rehabilitation. IEEE Transactions on Robotics 32, 1 (2016), 113–124. https://doi.org/10.1109/TRO.2015.2503726
[21]
Sigrist Roland, Rauter Georg, Marchal-Crespo Laura, Riener Robert, and Wolf Peter. 2015. Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning. 233 (2015), 909––925. https://doi.org/10.1007/s00221-014-4167-7
[22]
Daniel Roth and Marc Erich Latoschik. 2020. Construction of the Virtual Embodiment Questionnaire (VEQ). IEEE Transactions on Visualization and Computer Graphics 26, 12 (2020), 3546–3556. https://doi.org/10.1109/TVCG.2020.3023603
[23]
Govindarajan Srimathveeravalli and Kesavadas Thenkurussi. 2005. Motor Skill Training Assistance Using Haptic Attributes(WHC ’05). IEEE Computer Society, USA, 452–457. https://doi.org/10.1109/WHC.2005.96
[24]
Haruto Takita, Yuji Hatada, Takuji Narumi, and Michitaka Hirose. 2023. Effects of Virtual Co-embodiment on Declarative Memory-Based Motor Skill Learning. In ACM Symposium on Applied Perception 2023 (Los Angeles, CA, USA) (SAP ’23). Association for Computing Machinery, New York, NY, USA, Article 11, 10 pages. https://doi.org/10.1145/3605495.3605797
[25]
Gang Tang, Jinman Lei, Chentong Shao, Xiong Hu, Weidong Cao, and Shaoyang Men. 2021. Short-term prediction in vessel heave motion based on improved LSTM model. IEEE Access 9 (2021), 58067–58078.
[26]
Michael Tombu and Pierre Jolicœur. 2003. A central capacity sharing model of dual-task performance.Journal of Experimental Psychology: Human Perception and Performance 29, 1 (2003), 3.
[27]
Lijing Wang and Lijuan Wang. 2024. Relationships between Motor Skills and Academic Achievement in School-Aged Children and Adolescents: A Systematic Review. Children 11, 3 (2024).
[28]
Geoffrey Webb, Claude Sammut, Claudia Perlich, Tamás Horváth, Stefan Wrobel, Kevin Korb, William Noble, Christina Leslie, Michail Lagoudakis, Novi Quadrianto, Wray Buntine, Lise Getoor, Galileo Namata, Jiawei Jin, Jo-Anne Ting, Sethu Vijayakumar, Stefan Schaal, and Luc De Raedt. 2010. Leave-One-Out Cross-Validation. https://doi.org/10.1007/978-0-387-30164-8_469
[29]
Ungyeon Yang. 1999. Just follow me: An immersive vr-based motion training system. In Proceedings of the International Conference on Virtual Systems and Multimedia, 1999. 435–444. https://doi.org/10.3390/ijerph17239067
[30]
Ungyeon Yang and Gerard Jounghyun Kim. 2002. Implementation and evaluation of “just follow me”: An immersive, VR-based, motion-training system. Presence: Teleoperators & Virtual Environments 11, 3 (2002), 304–323. https://doi.org/10.1162/105474602317473240
[31]
Yufeng Zhang, Shuai Li, Karen J. Nolan, and Damiano Zanotto. 2019. Adaptive Assist-as-needed Control Based on Actor-Critic Reinforcement Learning. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 4066–4071. https://doi.org/10.1109/IROS40897.2019.8968464
Index Terms
- Motor Skill Learning by Virtual Co-embodiment with an AI Teacher Trained in Human Teaching Behavior
Recommendations
Effects of Virtual Co-embodiment on Declarative Memory-Based Motor Skill Learning
SAP '23: ACM Symposium on Applied Perception 2023This study investigated the learning effect and skill retention when virtual co-embodiment, in which movements of people were weighted and averaged into a single avatar, was used to learn motor skills requiring declarative memory. Previous studies have ...
Effect of Weight Adjustment in Virtual Co-embodiment During Collaborative Training
AHs '23: Proceedings of the Augmented Humans International Conference 2023Acquisition of motor skills plays an essential role in various contexts, such as sports, factory jobs, and nursing. For effective motor skill learning, “virtual co-embodiment” has been proposed as a novel virtual reality (VR) based method in which a ...
Examining the potential of web-based multimedia to support complex fine motor skill learning: An empirical study
Research on the utilization of the Web for complex fine motor skill learning that involves whole body movements is still scarce. The aim of this study was to evaluate the impact of the introduction of a multimedia web-based learning environment, which ...
Comments
Information & Contributors
Information
Published In
August 2024
172 pages
ISBN:9798400710612
DOI:10.1145/3675231
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 30 August 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- Grant-in-Aid for Scientific Research (S)
- Grant-in-Aid for Scientific Research (A)
- JST Moonshot Research & Development Program
Conference
SAP '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 43 of 94 submissions, 46%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 226Total Downloads
- Downloads (Last 12 months)226
- Downloads (Last 6 weeks)41
Reflects downloads up to 23 Feb 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in