Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3472749.3474753acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article
Open access

So Predictable! Continuous 3D Hand Trajectory Prediction in Virtual Reality

Published: 12 October 2021 Publication History
  • Get Citation Alerts
  • Abstract

    We contribute a novel user- and activity-independent kinematics-based regressive model for continuously predicting ballistic hand movements in virtual reality (VR). Compared to prior work on end-point prediction, continuous hand trajectory prediction in VR enables an early estimation of future events such as collisions between the user’s hand and virtual objects such as UI widgets. We developed and validated our prediction model through a user study with 20 participants. The study collected hand motion data with a 3D pointing task and a gaming task with three popular VR games. Results show that our model can achieve a low Root Mean Square Error (RMSE) of 0.80 cm, 0.85 cm and 3.15 cm from future hand positions ahead of 100 ms, 200 ms and 300 ms respectively across all the users and activities. In pointing tasks, our predictive model achieves an average angular error of 4.0° and 1.5° from the true landing position when 50% and 70% of the way through the movement. A follow-up study showed that the model can be applied to new users and new activities without further training.

    Supplementary Material

    VTT File (p332-talk.vtt)
    VTT File (p332-video_figure.vtt)
    VTT File (p332-video_preview.vtt)
    MP4 File (p332-video_figure.mp4)
    Video figure and captions
    MP4 File (p332-talk.mp4)
    Talk video and captions
    MP4 File (p332-video_preview.mp4)
    Video preview and captions

    References

    [1]
    A Deniz Aladagli, Erhan Ekmekcioglu, Dmitri Jarnikov, and Ahmet Kondoz. 2017. Predicting head trajectories in 360 virtual reality videos. In 2017 International Conference on 3D Immersion (IC3D). IEEE, 1–6.
    [2]
    Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Lars Petersson, and Stephen Gould. 2020. A stochastic conditioning scheme for diverse human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5223–5232.
    [3]
    Elena Arabadzhiyska, Okan Tarhan Tursun, Karol Myszkowski, Hans Peter Seidel, and Piotr Didyk. 2017. Saccade landing position prediction for gaze-contingent rendering. ACM Transactions on Graphics 36, 4 (2017). https://doi.org/10.1145/3072959.3073642
    [4]
    Stanislav Aranovskiy, Rosane Ushirobira, Denis Efimov, and Géry Casiez. 2016. Modeling pointing tasks in mouse-based human-computer interactions. In 2016 IEEE 55th Conference on Decision and Control (CDC). IEEE, 6595–6600.
    [5]
    Mahdi Azmandian, Mark Hancock, Hrvoje Benko, Eyal Ofek, and Andrew D Wilson. 2016. Haptic retargeting: Dynamic repurposing of passive haptics for enhanced virtual reality experiences. In Proceedings of the 2016 chi conference on human factors in computing systems. 1968–1979.
    [6]
    Ronald Azuma and Gary Bishop. 1995. A frequency-domain analysis of head-motion prediction. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. 401–408.
    [7]
    Myroslav Bachynskyi and Jörg Müller. 2020. Dynamics of Aimed Mid-air Movements. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–12.
    [8]
    Daniel Bullock and Stephen Grossberg. 1988. Neural dynamics of planned arm movements: emergent invariants and speed-accuracy properties during trajectory formation.Psychological review 95, 1 (1988), 49.
    [9]
    Elie Cattan, Amélie Rochet-Capellan, Pascal Perrier, and François Bérard. 2015. Reducing latency with a continuous prediction: Effects on users’ performance in direct-touch target acquisitions. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces. 205–214.
    [10]
    Yeonjoo Cha and Rohae Myung. 2013. Extended Fitts’ law for 3D pointing tasks using 3D target arrangements. International Journal of Industrial Ergonomics 43, 4(2013), 350–355.
    [11]
    Yang Chen, Xingang Zhao, and Jianda Han. 2013. Hierarchical projection regression for online estimation of elbow joint angle using EMG signals. Neural Computing and Applications 23, 3 (2013), 1129–1138.
    [12]
    Chris Christou. 2010. Virtual reality in education. In Affective, interactive and cognitive methods for e-learning design: creating an optimal education experience. IGI Global, 228–243.
    [13]
    Aldrich Clarence, Jarrod Knibbe, Maxime Cordeil, and Michael Wybrow. 2021. Unscripted Retargeting: Reach Prediction for Haptic Retargeting in Virtual Reality. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). IEEE, 150–159.
    [14]
    Nachiket Deo and Mohan M Trivedi. 2018. Convolutional social pooling for vehicle trajectory prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1468–1476.
    [15]
    Massimiliano Di Luca and Arash Mahnan. 2019. Perceptual limits of visual-haptic simultaneity in virtual reality interactions. In 2019 IEEE World Haptics Conference (WHC). IEEE, 67–72.
    [16]
    Mohammed S Elbamby, Cristina Perfecto, Mehdi Bennis, and Klaus Doppler. 2018. Toward low-latency and ultra-reliable virtual reality. IEEE Network 32, 2 (2018), 78–84.
    [17]
    Hesham Elsayed, Mayra Donaji Barrera Machuca, Christian Schaarschmidt, Karola Marky, Florian Müller, Jan Riemann, Andrii Matviienko, Martin Schmitz, Martin Weigel, and Max Mühlhäuser. 2020. VRSketchPen: Unconstrained Haptic Assistance for Sketching in Virtual 3D Environments. In 26th ACM Symposium on Virtual Reality Software and Technology (Virtual Event, Canada) (VRST ’20). Association for Computing Machinery, New York, NY, USA, Article 3, 11 pages. https://doi.org/10.1145/3385956.3418953
    [18]
    Behnam Esfahbod. 2011. SFU Motion Capture Database. https://mocap.cs.sfu.ca/
    [19]
    Tamar Flash and Neville Hogan. 1985. The coordination of arm movements: an experimentally confirmed mathematical model. Journal of neuroscience 5, 7 (1985), 1688–1703.
    [20]
    Laura Freina and Michela Ott. 2015. A literature review on immersive virtual reality in education: state of the art and perspectives. In The international scientific conference elearning and software for education, Vol. 1. 10–1007.
    [21]
    Nirit Gavish, Teresa Gutiérrez, Sabine Webel, Jorge Rodríguez, Matteo Peveri, Uli Bockholt, and Franco Tecchia. 2015. Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interactive Learning Environments 23, 6 (2015), 778–798.
    [22]
    Henry Griffith, Subir Biswas, and Oleg Komogortsev. 2018. Towards Reduced Latency in Saccade Landing Position Prediction Using Velocity Profile Methods. In Proceedings of the Future Technologies Conference. Springer, 79–91.
    [23]
    Liang-Yan Gui, Yu-Xiong Wang, Xiaodan Liang, and José MF Moura. 2018. Adversarial geometry-aware human motion prediction. In Proceedings of the European Conference on Computer Vision (ECCV). 786–803.
    [24]
    Serhan Gül, Sebastian Bosse, Dimitri Podborski, Thomas Schierl, and Cornelius Hellge. 2020. Kalman Filter-based Head Motion Prediction for Cloud-based Mixed Reality. In Proceedings of the 28th ACM International Conference on Multimedia. 3632–3641.
    [25]
    Serhan Gül, Dimitri Podborski, Thomas Buchholz, Thomas Schierl, and Cornelius Hellge. 2020. Low-latency cloud-based volumetric video streaming using head motion prediction. In Proceedings of the 30th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video. 27–33.
    [26]
    Markus Hahn, Lars Krüger, and Christian Wöhler. 2008. 3d action recognition and long-term prediction of human motion. In International Conference on Computer Vision Systems. Springer, 23–32.
    [27]
    Rorik Henrikson, Tovi Grossman, Sean Trowbridge, Daniel Wigdor, and Hrvoje Benko. 2020. Head-Coupled Kinematic Template Matching: A Prediction Model for Ray Pointing in VR. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.
    [28]
    Alejandro Hernandez, Jurgen Gall, and Francesc Moreno-Noguer. 2019. Human motion prediction via spatio-temporal inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 7134–7143.
    [29]
    Neville Hogan. 1984. An organizing principle for a class of voluntary movements. Journal of Neuroscience 4, 11 (1984), 2745–2754.
    [30]
    Xueshi Hou, Jianzhong Zhang, Madhukar Budagavi, and Sujit Dey. 2019. Head and body motion prediction to enable mobile VR experiences with low latency. In 2019 IEEE Global Communications Conference (GLOBECOM). IEEE, 1–7.
    [31]
    ByeoungDo Kim, Chang Mook Kang, Jaekyum Kim, Seung Hi Lee, Chung Choo Chung, and Jun Won Choi. 2017. Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 399–404.
    [32]
    Edward Lank, Yi-Chun Nikko Cheng, and Jaime Ruiz. 2007. Endpoint prediction using motion kinematics. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 637–646.
    [33]
    Rynson WH Lau and Addison Chan. 2008. Motion prediction for online gaming. In International Workshop on Motion in Games. Springer, 104–114.
    [34]
    Steve LaValle. 2013. The Latent Power of Prediction. https://developer.oculus.com/blog/the-latent-power-of-prediction/. Accessed: 2021-04-01.
    [35]
    Chen Li, Zhen Zhang, Wee Sun Lee, and Gim Hee Lee. 2018. Convolutional sequence to sequence model for human dynamics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5226–5234.
    [36]
    Hongyi Liu and Lihui Wang. 2017. Human motion prediction for human-robot collaboration. Journal of Manufacturing Systems 44 (2017), 287–294.
    [37]
    Lei Liu, Robert van Liere, Catharina Nieuwenhuizen, and Jean-Bernard Martens. 2009. Comparing aimed movements in the real world and in virtual reality. In 2009 IEEE Virtual Reality Conference. IEEE, 219–222.
    [38]
    Wei Mao, Miaomiao Liu, Mathieu Salzmann, and Hongdong Li. 2019. Learning trajectory dependencies for human motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9489–9497.
    [39]
    Julieta Martinez, Michael J Black, and Javier Romero. 2017. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2891–2900.
    [40]
    Aythami Morales, Francisco M Costela, Ruben Tolosana, and Russell L Woods. 2018. Saccade Landing Point Prediction: A Novel Approach based on Recurrent Neural Networks. In Proceedings of the 2018 International Conference on Machine Learning Technologies. 1–5.
    [41]
    Mathieu Nancel, Stanislav Aranovskiy, Rosane Ushirobira, Denis Efimov, Sebastien Poulmane, Nicolas Roussel, and Géry Casiez. 2018. Next-point prediction for direct touch using finite-time derivative estimation. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 793–807.
    [42]
    Mathieu Nancel, Daniel Vogel, Bruno De Araujo, Ricardo Jota, and Géry Casiez. 2016. Next-point prediction metrics for perceived spatial errors. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 271–285.
    [43]
    Diederick C Niehorster, Li Li, and Markus Lappe. 2017. The accuracy and precision of position and orientation tracking in the HTC vive virtual reality system for scientific research. i-Perception 8, 3 (2017), 2041669517708205.
    [44]
    Daniel Eger Passos and Bernhard Jung. 2020. Measuring the accuracy of inside-out tracking in XR devices using a high-precision robotic arm. In International Conference on Human-Computer Interaction. Springer, 19–26.
    [45]
    Dario Pavllo, Christoph Feichtenhofer, Michael Auli, and David Grangier. 2019. Modeling human motion with quaternion-based neural networks. International Journal of Computer Vision(2019), 1–18.
    [46]
    Dario Pavllo, David Grangier, and Michael Auli. 2018. Quaternet: A quaternion-based recurrent model for human motion. arXiv preprint arXiv:1805.06485(2018).
    [47]
    Roshan Lalintha Peiris, Wei Peng, Zikun Chen, Liwei Chan, and Kouta Minamizawa. 2017. Thermovr: Exploring integrated thermal haptic feedback with head mounted displays. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 5452–5456.
    [48]
    Kjetil Raaen, Ragnhild Eg, and Ivar Kjellmo. 2019. Playing with delay: An interactive VR demonstration. In Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems. 19–21.
    [49]
    Nimesha Ranasinghe, Pravar Jain, Nguyen Thi Ngoc Tram, Koon Chuan Raymond Koh, David Tolley, Shienny Karwita, Lin Lien-Ya, Yan Liangkun, Kala Shamaiah, Chow Eason Wai Tung, 2018. Season traveller: Multisensory narration for enhancing the virtual reality experience. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.
    [50]
    Nimesha Ranasinghe, Thi Ngoc Tram Nguyen, Yan Liangkun, Lien-Ya Lin, David Tolley, and Ellen Yi-Luen Do. 2017. Vocktail: A virtual cocktail for pairing digital taste, smell, and color sensations. In Proceedings of the 25th ACM international conference on Multimedia. 1139–1147.
    [51]
    Joël Randrianandrasana, Arnaud Chanonier, Hervé Deleau, Thomas Muller, Philippe Porral, Michaël Krajecki, and Laurent Lucas. 2018. Multi-user predictive rendering on remote multi-GPU clusters. In 2018 IEEE Fourth VR International Workshop on Collaborative Virtual Environments (3DCVE). IEEE, 1–4.
    [52]
    Magnus JE Richardson and Tamar Flash. 2002. Comparing smooth arm movements with the two-thirds power law and the related segmented-control hypothesis. Journal of neuroscience 22, 18 (2002), 8201–8211.
    [53]
    Miguel Fabián Romero Rondón, Lucile Sassatelli, Ramón Aparicio Pardo, and Frédéric Precioso. 2020. Track: a Multi-Modal Deep Architecture for Head Motion Prediction in 360° Videos. In 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2586–2590.
    [54]
    Maria T Schultheis and Albert A Rizzo. 2001. The application of virtual reality technology in rehabilitation.Rehabilitation psychology 46, 3 (2001), 296.
    [55]
    David Tolley, Thi Ngoc Tram Nguyen, Anthony Tang, Nimesha Ranasinghe, Kensaku Kawauchi, and Ching Chiuan Yen. 2019. Windywall: exploring creative wind simulations. In Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction. 635–644.
    [56]
    Etienne Van Wyk and Ruth De Villiers. 2009. Virtual reality training applications for the mining industry. In Proceedings of the 6th international conference on computer graphics, virtual reality, visualisation and interaction in Africa. 53–63.
    [57]
    Tran Huy Vu, Archan Misra, Quentin Roy, Kenny Choo Tsu Wei, and Youngki Lee. 2018. Smartwatch-based Early Gesture Detection 8 Trajectory Tracking for Interactive Gesture-Driven Applications. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 1–27. https://doi.org/10.1145/3191771
    [58]
    Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. 2017. The pose knows: Video forecasting by generating pose futures. In Proceedings of the IEEE international conference on computer vision. 3332–3341.
    [59]
    Min Wang, Xinyu Wu, Duxin Liu, Can Wang, Ting Zhang, and Pingan Wang. 2015. A human motion prediction algorithm for non-binding lower extremity exoskeleton. In 2015 IEEE International Conference on Information and Automation. IEEE, 369–374.
    [60]
    Zhen Wang, Hengsuo Xu, and Hongliang Yuan. 2020. Research on Design and Experience of Immersive Virtual Reality Psychological Relaxation Game Based on Image. In IOP Conference Series: Materials Science and Engineering, Vol. 740. IOP Publishing, 012118.
    [61]
    Ning Xie, Gabrielle Ras, Marcel van Gerven, and Derek Doran. 2020. Explainable deep learning: A field guide for the uninitiated. arXiv preprint arXiv:2004.14545(2020).
    [62]
    Michael Yates, Arpad Kelemen, and Cecilia Sik Lanyi. 2016. Virtual reality gaming in the rehabilitation of the upper extremities post-stroke. Brain injury 30, 7 (2016), 855–863.
    [63]
    Shulin Zhao, Haibo Zhang, Sandeepa Bhuyan, Cyan Subhra Mishra, Ziyu Ying, Mahmut T Kandemir, Anand Sivasubramaniam, and Chita R Das. 2020. Déjà View: Spatio-Temporal Compute Reuse for ‘Energy-Efficient 360° VR Video Streaming. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, 241–253.

    Cited By

    View all
    • (2024)CLOVR: Collecting and Logging OpenVR Data from SteamVR Applications2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00095(485-492)Online publication date: 16-Mar-2024
    • (2024)Enhanced Human-Robot Collaboration with Intent Prediction using Deep Inverse Reinforcement Learning2024 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA57147.2024.10610595(7880-7887)Online publication date: 13-May-2024
    • (2024)Learning Human-Arm Reaching Motion Using a Wearable Device in Human–Robot CollaborationIEEE Access10.1109/ACCESS.2024.336566112(24855-24865)Online publication date: 2024
    • Show More Cited By

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology
    October 2021
    1357 pages
    ISBN:9781450386357
    DOI:10.1145/3472749
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 12 October 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Activity-independent
    2. Hand motion prediction
    3. Kinematics-based Model
    4. User-independent
    5. Virtual Reality

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    UIST '21

    Acceptance Rates

    Overall Acceptance Rate 842 of 3,967 submissions, 21%

    Upcoming Conference

    UIST '24

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)1,001
    • Downloads (Last 6 weeks)112
    Reflects downloads up to 12 Aug 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)CLOVR: Collecting and Logging OpenVR Data from SteamVR Applications2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00095(485-492)Online publication date: 16-Mar-2024
    • (2024)Enhanced Human-Robot Collaboration with Intent Prediction using Deep Inverse Reinforcement Learning2024 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA57147.2024.10610595(7880-7887)Online publication date: 13-May-2024
    • (2024)Learning Human-Arm Reaching Motion Using a Wearable Device in Human–Robot CollaborationIEEE Access10.1109/ACCESS.2024.336566112(24855-24865)Online publication date: 2024
    • (2024)Continuous Prediction of Pointing Targets With Motion and Eye-Tracking in Virtual RealityIEEE Access10.1109/ACCESS.2024.335078812(5933-5946)Online publication date: 2024
    • (2024)Long short-term memory prediction of user’s locomotion in virtual realityVirtual Reality10.1007/s10055-024-00962-928:1Online publication date: 5-Mar-2024
    • (2023)N-euro PredictorProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108847:3(1-25)Online publication date: 27-Sep-2023
    • (2023)Examining the Fine Motor Control Ability of Linear Hand Movement in Virtual Reality2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR55154.2023.00058(427-437)Online publication date: Mar-2023
    • (2023)A Deep Learning-Based Approach for Foot Placement PredictionIEEE Robotics and Automation Letters10.1109/LRA.2023.32905218:8(4959-4966)Online publication date: Aug-2023
    • (2023)Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecasting2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01260(13656-13665)Online publication date: 1-Oct-2023
    • (2022)Predicting Touch Accuracy for Rectangular Targets by Using One-Dimensional Task ResultsProceedings of the ACM on Human-Computer Interaction10.1145/35677326:ISS(525-537)Online publication date: 14-Nov-2022
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Get Access

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media