research-article

RemoteFusion: real time depth camera fusion for remote collaboration on physical tasks

Authors:

Stuart Anderson,

Bruce ThomasAuthors Info & Claims

VRCAI '13: Proceedings of the 12th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry

Pages 235 - 242

https://doi.org/10.1145/2534329.2534331

Published: 17 November 2013 Publication History

Abstract

Remote guidance systems allow humans to collaborate on physical tasks across large distances and have applications in fields such as medicine, maintenance and working with hazardous substances. Existing systems typically provide two dimensional video streams to remote participants, and these are restricted to viewpoint locations based on the placement of physical cameras. Recent systems have incorporated the ability of a remote expert to annotate their 2D view and for these annotations to be displayed in the physical workspace to the local worker. We present a prototype remote guidance system, called RemoteFusion, which is based on the volumetric fusion of commodity depth cameras. The system incorporates real-time 3D fusion with color, the ability to distinguish and render dynamic elements of a scene whether human or non-human, a multi-touch driven free 3D viewpoint, and a Spatial Augmented Reality (SAR) light annotation mechanism. We provide a physical overview of the system, including hardware and software configuration, and detail the implementation of each of the key features.

References

[1]

Adcock, M., Feng, D., and Thomas, B. 2013. Visualization of Off-Surface 3D Viewpoint Locations in Spatial Augmented Reality. Proceedings of the ACM symposium on Spatial User Interaction - SUI '13.

Digital Library

[2]

Alem, L., Tecchia, F., and Huang, W. 2011. HandsOnVideo: Towards a Gesture based Mobile AR System for Remote Collaboration. In: L. Alem and W. Huang, eds., Recent Trends of Mobile Collaborative Augmented Reality Systems. Springer New York, 135--148.

[3]

Butler, D. A., Izadi, S., Hilliges, O., Molyneaux, D., Hodges, S., and Kim, D. 2012. Shake'n'sense: reducing interference for overlapping structured light depth cameras. Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems - CHI '12, ACM Press, 1933.

Digital Library

[4]

Fussell, S., Setlock, L., Yang, J., Ou, J., Mauer, E., and Kramer, A. 2004. Gestures Over Video Streams to Support Remote Collaboration on Physical Tasks. Human-Computer Interaction 19, 3, 273--309.

Digital Library

[5]

Izadi, S., Davison, A., Fitzgibbon, A., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., and Freeman, D. 2011. KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera. Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11, ACM Press, 559.

Digital Library

[6]

Jones, B. R., Benko, H., Ofek, E., and Wilson, A. D. 2013. IllumiRoom. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI '13, ACM Press, 869.

Digital Library

[7]

Kainz, B., Schmalstieg, D., Hauswiesner, S., Reitmayr, G., Steinberger, M., Grasset, R., Gruber, L., Veas, E., Kalkofen, D., and Seichter, H. 2012. OmniKinect. Proceedings of the 18th ACM symposium on Virtual reality software and technology - VRST '12, ACM Press, 25.

Digital Library

[8]

Kirk, D., Crabtree, A., and Rodden, T. 2005. Ways of the Hands. In: H. Gellersen, K. Schmidt, M. Beaudouin-Lafon and W. Mackay, eds., ECSCW 2005. Springer-Verlag, Berlin/Heidelberg, 1--21.

Digital Library

[9]

Kuzuoka, H. 1992. Spatial workspace collaboration: a SharedView video support system for remote collaboration capability. Proceedings of the ACM Conference on Human Factors in Computing Systems, ACM, 533--540.

Digital Library

[10]

Maimone, A. and Fuchs, H. 2011. Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras. 2011 10th IEEE International Symposium on Mixed and Augmented Reality, IEEE, 137--146.

Digital Library

[11]

Maimone, A. and Fuchs, H. 2012. Reducing interference between multiple structured light depth sensors using motion. 2012 IEEE Virtual Reality (VR), IEEE, 51--54.

Digital Library

[12]

Newcombe, R. A., Davison, A. J., Izadi, S., Kohli, P., Hilliges, O., Shotton, J., Molyneaux, D., Hodges, S., Kim, D., and Fitzgibbon, A. 2011. KinectFusion: Real-time dense surface mapping and tracking. 2011 10th IEEE International Symposium on Mixed and Augmented Reality, IEEE, 127--136.

Digital Library

[13]

Palmer, D., Adcock, M., Smith, J., Hutchins, M., Gunn, C., Stevenson, D., and Taylor, K. 2007. Annotating with light for remote guidance. Proceedings of the 2007 conference of the computer-human interaction special interest group (CHISIG) of Australia on Computer-human interaction: design: activities, artifacts and environments - OZCHI '07, ACM Press, 103.

Digital Library

[14]

Tecchia, F., Alem, L., and Huang, W. 2012. 3D helping hands. Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry - VRCAI '12, ACM Press, 323.

Digital Library

[15]

Welch, G., Sonnenwald, D. H., Mayer-Patel, K., State, A., Towles, H., Cairns, B., and Fuchs, H. 2005. Remote 3D medical consultation. 2nd International Conference on Broadband Networks, 2005., IEEE, 103--110.

[16]

Wilson, A., Benko, H., Izadi, S., and Hilliges, O. 2012. Steerable augmented reality with the beamatron. Proceedings of the 25th annual ACM symposium on User interface software and technology - UIST '12, ACM Press, 413.

Digital Library

Cited By

Liao JVan KXia ZSuzuki R(2024)RealityEffects: Augmenting 3D Volumetric Videos with Object-Centric Annotation and Dynamic Visual EffectsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661631(1248-1261)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661631
Huang XXiao R(2024)SurfShareProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314187:4(1-24)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631418
Borhani ZSharma POrtega F(2024)Survey of Annotations in Extended Reality SystemsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328886930:8(5074-5096)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3288869
Show More Cited By

Index Terms

RemoteFusion: real time depth camera fusion for remote collaboration on physical tasks
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Graphical user interfaces
2. Information systems
  1. Information systems applications
    1. Multimedia information systems

Recommendations

Visualization of off-surface 3D viewpoint locations in spatial augmented reality
SUI '13: Proceedings of the 1st symposium on Spatial user interaction

Spatial Augmented Reality (SAR) systems can be used to convey guidance in a physical task from a remote expert. Sometimes that remote expert is provided with a single camera view of the workspace but, if they are given a live captured 3D model and can ...
MirageTable: freehand interaction on a projected augmented reality tabletop
CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Instrumented with a single depth camera, a stereoscopic projector, and a curved screen, MirageTable is an interactive system designed to merge real and virtual worlds into a single spatially registered experience on top of a table. Our depth camera ...
Sublimate: state-changing virtual and physical rendering to augment interaction with shape displays
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Recent research in 3D user interfaces pushes towards immersive graphics and actuated shape displays. Our work explores the hybrid of these directions, and we introduce sublimation and deposition, as metaphors for the transitions between physical and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRCAI '13: Proceedings of the 12th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry

November 2013

325 pages

ISBN:9781450325905

DOI:10.1145/2534329

General Chairs:
Charlie C. L. Wang
The Chinese University of Hong Kong, China
,
Nadia Magnenat-Thalmann
Nanyang Technological University, Singapore and MIRALab, University of Geneva, Switzerland
,
Zhigeng Pan
Hangzhou Normal University, China

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 November 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

VRCAI 2013

Sponsor:

SIGGRAPH

VRCAI 2013: 12th ACM International Conference on Virtual Reality Continuum and Its Applications in Industry

November 17 - 19, 2013

Hong Kong, Hong Kong

Acceptance Rates

VRCAI '13 Paper Acceptance Rate 35 of 75 submissions, 47%;

Overall Acceptance Rate 51 of 107 submissions, 48%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

50
Total Citations
View Citations
799
Total Downloads

Downloads (Last 12 months)54
Downloads (Last 6 weeks)3

Reflects downloads up to 02 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liao JVan KXia ZSuzuki R(2024)RealityEffects: Augmenting 3D Volumetric Videos with Object-Centric Annotation and Dynamic Visual EffectsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661631(1248-1261)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661631
Huang XXiao R(2024)SurfShareProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314187:4(1-24)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631418
Borhani ZSharma POrtega F(2024)Survey of Annotations in Extended Reality SystemsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328886930:8(5074-5096)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3288869
Praveena PWang YSenft EGleicher MMutlu B(2023)Periscope: A Robotic Camera System to Support Remote Physical CollaborationProceedings of the ACM on Human-Computer Interaction10.1145/36101997:CSCW2(1-39)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610199
Johnson JSharkey TButarbutar IXiong DHuang RSy LWeibel N(2023)UnMapped: Leveraging Experts’ Situated Experiences to Ease Remote Guidance in Collaborative Mixed RealityProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581444(1-20)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581444
Faridan MKumari BSuzuki R(2023)ChameleonControl: Teleoperating Real Human Surrogates through Mixed Reality Gestural Guidance for Remote Hands-on ClassroomsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581381(1-13)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581381
Tian HLee GBai HBillinghurst M(2023)Using Virtual Replicas to Improve Mixed Reality Remote CollaborationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324711329:5(2785-2795)Online publication date: May-2023
https://doi.org/10.1109/TVCG.2023.3247113
Fidalgo CYan YCho HSousa MLindlbauer DJorge J(2023)A Survey on Remote Assistance and Training in Mixed Reality EnvironmentsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324708129:5(2291-2303)Online publication date: May-2023
https://doi.org/10.1109/TVCG.2023.3247081
Zaman FAnslow CChalmers ARhee T(2023)MRMAC: Mixed Reality Multi-user Asymmetric Collaboration2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00074(591-600)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR59233.2023.00074
Chang ELee YYoo B(2023)A User Study on the Comparison of View Interfaces for VR-AR Communication in XR Remote CollaborationInternational Journal of Human–Computer Interaction10.1080/10447318.2023.2241294(1-16)Online publication date: 17-Aug-2023
https://doi.org/10.1080/10447318.2023.2241294
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents