short-paper

Open access

XRMan: Towards Real-time Hand-Object Pose Tracking in eXtended Reality

Authors:

Dongho Han,

Jingyu Lee,

Minjae Kim,

Youngki LeeAuthors Info & Claims

ACM MobiCom '24: Proceedings of the 30th Annual International Conference on Mobile Computing and Networking

Pages 1575 - 1577

https://doi.org/10.1145/3636534.3697422

Published: 04 December 2024 Publication History

PDF eReader

Abstract

Accurate tracking of hand and object poses is essential for immersive XR applications. However, existing methods often struggle with the stringent requirements of XR environments. We present XRMan, a real-time hand-object pose tracking system designed for in-the-wild scenarios. XRMan uses a drift monitoring module for consistent accuracy, while farthest-point sampling and collider-based optimization streamline iterative optimization and reduce latency. Our system reduces end-to-end latency by 38.9% compared to the baseline, with further improvements expected from the collider-based post-optimization module.

References

[1]

Jiayi Chen, Mi Yan, Jiazhao Zhang, Yinzhen Xu, Xiaolong Li, Yijia Weng, Li Yi, Shuran Song, and He Wang. 2022. Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wild. arXiv preprint arXiv:2209.12009 (2022).

Google Scholar

[2]

Elmer G Gilbert, Daniel W Johnson, and S Sathiya Keerthi. 1988. A fast procedure for computing the distance between complex objects in three-dimensional space. IEEE Journal on Robotics and Automation 4, 2 (1988), 193--203.

Crossref

Google Scholar

[3]

Yana Hasson, Gül Varol, Ivan Laptev, and Cordelia Schmid. 2021. Towards unconstrained joint hand-object reconstruction from RGB videos. In ArXiv.

Google Scholar

[4]

Roy Miles, Mehmet Kerim Yucel, Bruno Manganelli, and Albert Saà-Garriga. 2023. Mobilevos: Real-time video object segmentation contrastive learning meets knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10480--10490.

Crossref

Google Scholar

[5]

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Google Scholar

[6]

Javier Romero, Dimitrios Tzionas, and Michael J. Black. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) 36, 6 (Nov. 2017).

Google Scholar

Index Terms

XRMan: Towards Real-time Hand-Object Pose Tracking in eXtended Reality
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Object detection
        Tracking
      2. Image and video acquisition
        Camera calibration
        Motion capture
  2. Computer graphics
2. Social and professional topics
  1. Professional topics
    1. History of computing
      1. History of computing theory

Index terms have been assigned to the content through auto-classification.

Recommendations

Real-time camera pose estimation via line tracking

Real-time camera calibration has been intensively studied in augmented reality. However, for texture-less and texture-repeated scenes as well as poorly illuminated scenes, obtaining a stable calibration is still an open problem. In the paper, we propose ...
Robust camera pose estimation using 2d fiducials tracking for real-time augmented reality systems
VRCAI '04: Proceedings of the 2004 ACM SIGGRAPH international conference on Virtual Reality continuum and its applications in industry

Augmented reality (AR) deals with the problem of dynamically and accurately align virtual objects with the real world. Among the used methods, vision-based techniques have advantages for AR applications, their registration can be very accurate, and ...
Real-Time Markerless Tracking for Augmented Reality: The Virtual Visual Servoing Framework

Tracking is a very important research subject in a real-time augmented reality context. The main requirements for trackers are high accuracy and little latency at a reasonable cost. In order to address these issues, a real-time, robust, and efficient 3D ...

Comments

Information & Contributors

Information

Published In

ACM MobiCom '24: Proceedings of the 30th Annual International Conference on Mobile Computing and Networking

December 2024

2476 pages

ISBN:9798400704895

DOI:10.1145/3636534

Chair:
Weisong Shi,
Program Chairs:
Deepak Ganesan,
Nicholas Lane

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 December 2024

Check for updates

Qualifiers

Short-paper

Funding Sources

National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT)

Conference

ACM MobiCom '24

Sponsor:

SIGMOBILE

ACM MobiCom '24: 30th Annual International Conference on Mobile Computing and Networking

November 18 - 22, 2024

DC, Washington D.C., USA

Acceptance Rates

Overall Acceptance Rate 440 of 2,972 submissions, 15%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
83
Total Downloads

Downloads (Last 12 months)83
Downloads (Last 6 weeks)37

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Index Terms

Recommendations

Real-time camera pose estimation via line tracking

Robust camera pose estimation using 2d fiducials tracking for real-time augmented reality systems

Real-Time Markerless Tracking for Augmented Reality: The Virtual Visual Servoing Framework

Comments

Published In

Sponsors

Publisher

Publication History

Check for updates

Qualifiers

Funding Sources

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

PDF

eReader

Login options

Full Access

Abstract

References

Index Terms

Recommendations

Real-time camera pose estimation via line tracking

Robust camera pose estimation using 2d fiducials tracking for real-time augmented reality systems

Real-Time Markerless Tracking for Augmented Reality: The Virtual Visual Servoing Framework

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

View options

PDF

eReader

Login options

Full Access

Share

Share this Publication link

Share on social media

Affiliations