research-article

Free access

Procedurally generated virtual reality from 3D reconstructed physical space

Authors:

Sergio Garrido-Jurado,

Chris Schmandt,

Pattie MaesAuthors Info & Claims

VRST '16: Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology

Pages 191 - 200

https://doi.org/10.1145/2993369.2993372

Published: 02 November 2016 Publication History

Abstract

We present a novel system for automatically generating immersive and interactive virtual reality (VR) environments using the real world as a template. The system captures indoor scenes in 3D, detects obstacles like furniture and walls, and maps walkable areas (WA) to enable real-walking in the generated virtual environment (VE). Depth data is additionally used for recognizing and tracking objects during the VR experience. The detected objects are paired with virtual counterparts to leverage the physicality of the real world for a tactile experience. Our approach is new, in that it allows a casual user to easily create virtual reality worlds in any indoor space of arbitrary size and shape without requiring specialized equipment or training. We demonstrate our approach through a fully working system implemented on the Google Project Tango tablet device.

References

[1]

Alexandre, L. A. 2012. 3d descriptors for object and category recognition: a comparative evaluation. In Workshop on Color-Depth Camera Fusion in Robotics at the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vilamoura, Portugal, vol. 1, Citeseer, 7.

[2]

Besl, P. J., and McKay, N. D. 1992. Method for registration of 3-d shapes. In Robotics-DL tentative, International Society for Optics and Photonics, 586--606.

[3]

Bowman, D. A., Koller, D., and Hodges, L. F. 1997. Travel in immersive virtual environments: An evaluation of viewpoint motion control techniques. In Virtual Reality Annual International Symposium, 1997., IEEE 1997, IEEE, 45--52.

Digital Library

[4]

Burdea, G., and Coiffet, P. 2003. Virtual reality technology. Presence: Teleoperators and virtual environments 12, 6, 663--664.

Digital Library

[5]

Cheng, L.-P., Roumen, T., Rantzsch, H., Köhler, S., Schmidt, P., Kovacs, R., Jasper, J., Kemper, J., and Baudisch, P. 2015. Turkdeck: Physical virtual reality based on people. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, ACM, 417--426.

Digital Library

[6]

Cruz-Neira, C., Sandin, D. J., DeFanti, T. A., Kenyon, R. V., and Hart, J. C. 1992. The cave: audio visual experience automatic virtual environment. Communications of the ACM 35, 6, 64--73.

Digital Library

[7]

Gibson, J. J. 1979. The Ecological Approach to Visual Perception. Houghton Mifflin.

[8]

Hettiarachchi, A., and Wigdor, D. 2016. Annexing reality: Enabling opportunistic use of everyday objects as tangible proxies in augmented reality. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, CHI '16, 1957--1967.

Digital Library

[9]

Hinckley, K., Pausch, R., Goble, J. C., and Kassell, N. F. 1994. Passive real-world interface props for neurosurgical visualization. In Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, 452--458.

Digital Library

[10]

Hoffman, H. G. 1998. Physically touching virtual objects using tactile augmentation enhances the realism of virtual environments. In Virtual Reality Annual International Symposium, 1998. Proceedings., IEEE 1998, IEEE, 59--63.

Digital Library

[11]

Insko, B. E. 2001. Passive haptics significantly enhances virtual environments. PhD thesis, University of North Carolina at Chapel Hill.

Digital Library

[12]

Lai, K., Bo, L., Ren, X., and Fox, D. 2013. Rgb-d object recognition: Features, algorithms, and a large scale benchmark. In Consumer Depth Cameras for Computer Vision. Springer, 167--192.

[13]

Nassani, A., Bai, H., Lee, G., and Billinghurst, M. 2015. Tag it!: Ar annotation using wearable sensors. In SIGGRAPH Asia 2015 Mobile Graphics and Interactive Applications, ACM, New York, NY, USA, SA '15, 12:1--12:4.

Digital Library

[14]

Nescher, T., Zank, M., and Kunz, A. 2016. Simultaneous mapping and redirected walking for ad hoc free walking in virtual environments. In IEEE Virtual Reality Conference 2016, IEEE, 239--240.

[15]

Newcombe, R. A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohli, P., Shotton, J., Hodges, S., and Fitzgibbon, A. 2011. Kinectfusion: Real-time dense surface mapping and tracking. In IEEE ISMAR, IEEE.

Digital Library

[16]

Nuernberger, B., Ofek, E., Benko, H., and Wilson, A. D. 2016. Snaptoreality: Aligning augmented reality to the real world. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, CHI '16, 1233--1244.

Digital Library

[17]

Peasley, B., and Birchfield, S. 2013. Replacing projective data association with lucas-kanade for kinectfusion. In Robotics and Automation (ICRA), 2013 IEEE International Conference on, IEEE, 638--645.

[18]

Qayyum, U., Kim, J., et al. 2013. Inertial-kinect fusion for outdoor 3d navigation. In Australasian Conference on Robotics and Automation (ACRA).

[19]

Razzaque, S., Kohn, Z., and Whitton, M. C. 2001. Redirected walking. In Proceedings of EUROGRAPHICS, vol. 9, Citeseer, 105--106.

[20]

Shapira, L., Gal, R., Ofek, E., and Kohli, P. 2014. FLARE: fast layout for augmented reality applications. IEEE Institute of Electrical and Electronics Engineers.

[21]

Simeone, A. L., Velloso, E., and Gellersen, H. 2015. Substitutional reality: Using the physical environment to design virtual reality experiences. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, ACM, 3307--3316.

Digital Library

[22]

Song, S., and Xiao, J. 2014. Sliding shapes for 3d object detection in depth images. In Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6--12, 2014, Proceedings, Part VI, 634--651.

[23]

Sra, M., and Schmandt, C. 2015. Metaspace: Full-body tracking for immersive multiperson virtual reality. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, ACM, 47--48.

Digital Library

[24]

Suma, E. A., Clark, S., Krum, D., Finkelstein, S., Bolas, M., and Warte, Z. 2011. Leveraging change blindness for redirection in virtual environments. In 2011 IEEE Virtual Reality Conference, IEEE, 159--166.

Digital Library

[25]

Sutherland, I. E. 1968. A head-mounted three dimensional display. In Proceedings of the December 9--11, 1968, fall joint computer conference, part I, ACM, 757--764.

Digital Library

[26]

Syswerda, G. 1989. Uniform crossover in genetic algorithms. In Proceedings of the 3rd International Conference on Genetic Algorithms, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2--9.

Digital Library

[27]

Usoh, M., Arthur, K., Whitton, M. C., Bastos, R., Steed, A., Slater, M., and Brooks Jr, F. P. 1999. Walking > walking-in-place > flying, in virtual environments. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing Co., 359--364.

Digital Library

[28]

Warren, D. H., and Cleaves, W. T. 1971. Visual-proprioceptive interaction under large amounts of conflict. Journal of experimental psychology 90, 2, 206.

[29]

Whitley, D. 1994. A genetic algorithm tutorial. Statistics and computing 4, 2, 65--85.

Cited By

Fukaya KDaylamani-Zad DAgius H(2025)Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the ArtACM Computing Surveys10.1145/370849957:5(1-38)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3708499
Cao JYuan ZMao TWang ZLi Z(2025)NeRF-based Polarimetric Multi-view StereoPattern Recognition10.1016/j.patcog.2024.111036158(111036)Online publication date: Feb-2025
https://doi.org/10.1016/j.patcog.2024.111036
Jiang XZhou WSun JChen SFung A(2025)Empathy enhancement through VRInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103397194:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.ijhcs.2024.103397
Show More Cited By

Index Terms

Procedurally generated virtual reality from 3D reconstructed physical space
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Virtual reality
    2. Interaction techniques

Recommendations

Interacting with virtual reality scenes on mobile devices
MobileHCI '05: Proceedings of the 7th international conference on Human computer interaction with mobile devices & services

This paper discusses alternative approaches for interacting with virtual reality scenes on mobile devices, based upon work conducted as part of the locus project [4]. Three prototypes are introduced that adopt different interaction paradigms for mobile ...
Shared Reconstructed Environments in Extended Reality
SVR '24: Proceedings of the 26th Symposium on Virtual and Augmented Reality

Modern extended reality headsets can identify the geometry of the physical surrounding space, which can be shared and synchronized with remote users, allowing them to view the space through their own extended reality headsets from different points of ...
Asymmetric Design Approach and Collision Avoidance Techniques For Room-scale Multiplayer Virtual Reality
UIST '16 Adjunct: Adjunct Proceedings of the 29th Annual ACM Symposium on User Interface Software and Technology

Recent advances in consumer virtual reality (VR) technology have made it easy to accurately capture users' motions over room-sized areas allowing natural locomotion for navigation in VR. While this helps create a stronger match between proprioceptive ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '16: Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology

November 2016

363 pages

ISBN:9781450344913

DOI:10.1145/2993369

Conference Chairs:
Dieter Kranzlmüller
Leibniz Supercomputing Centre (LRZ) and Ludwig-Maximilians-Universität (LMU)
,
Gudrun Klinker
Technical University of Munich (TUM)

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Best Paper

Author Tags

Qualifiers

Research-article

Conference

VRST '16

Sponsor:

VRST '16: 22th ACM Symposium on Virtual Reality Software and Technology

November 2 - 4, 2016

Munich, Germany

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

106
Total Citations
View Citations
4,274
Total Downloads

Downloads (Last 12 months)582
Downloads (Last 6 weeks)111

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Fukaya KDaylamani-Zad DAgius H(2025)Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the ArtACM Computing Surveys10.1145/370849957:5(1-38)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3708499
Cao JYuan ZMao TWang ZLi Z(2025)NeRF-based Polarimetric Multi-view StereoPattern Recognition10.1016/j.patcog.2024.111036158(111036)Online publication date: Feb-2025
https://doi.org/10.1016/j.patcog.2024.111036
Jiang XZhou WSun JChen SFung A(2025)Empathy enhancement through VRInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103397194:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.ijhcs.2024.103397
Shoukat MSargano AYou LHabib Z(2024)3DRecNet: A 3D Reconstruction Network with Dual Attention and Human-Inspired MemoryElectronics10.3390/electronics1317339113:17(3391)Online publication date: 26-Aug-2024
https://doi.org/10.3390/electronics13173391
Shoukat MSargano AMalyshev AYou LHabib Z(2024)SS3DNet-AF: A Single-Stage, Single-View 3D Reconstruction Network with Attention-Based FusionApplied Sciences10.3390/app14231142414:23(11424)Online publication date: 8-Dec-2024
https://doi.org/10.3390/app142311424
Jiang HSong LWeng DSun ZLi HDongye XZhang ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)In Situ 3D Scene Synthesis for Ubiquitous Embodied InterfacesProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681616(3666-3675)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681616
Ze MYu D(2024)Space Renovation in Virtual RealityCompanion Publication of the 2024 ACM Designing Interactive Systems Conference10.1145/3656156.3663711(130-134)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3656156.3663711
Liao JVan KXia ZSuzuki R(2024)RealityEffects: Augmenting 3D Volumetric Videos with Object-Centric Annotation and Dynamic Visual EffectsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661631(1248-1261)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661631
Rebelo AFerreira PNóbrega R(2024)Adaptive Virtual Environments in Small Physical Spaces: Navigation Design for Customized User Experiences in Virtual RealityExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650942(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650942
Pointecker FFriedl-Knirsch JJetter HAnthes C(2024)From Real to Virtual: Exploring Replica-Enhanced Environment Transitions along the Reality-Virtuality ContinuumProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642844(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642844
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten