research-article

Public Access

PanoTrace: interactive 3D modeling of surround-view panoramic images in virtual reality

Authors:

Tobias HöllererAuthors Info & Claims

VRST '17: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology

Article No.: 32, Pages 1 - 10

https://doi.org/10.1145/3139131.3139158

Published: 08 November 2017 Publication History

Abstract

Full-surround panoramic imagery can provide a viewer with a high-resolution visual impression of a pictured real or realistically rendered environment, but it does not provide as high a level of immersion as modeled 3D geometry can, when viewed with virtual reality (VR) headsets or projection-based setups. In this paper, we demonstrate that augmenting panorama images with geometrical models can be done simply in VR itself and can significantly increase the feeling of immersion a viewer experiences. We propose a novel interactive modeling tool that allows users to model geometry depicted in a surround-panoramic scene directly in VR, utilizing projection mapping of the panorama on top of the evolving geometry. The user interface is intuitive and allows novice users to produce geometry that approximates ground truth models sufficiently to enhance a user's VR viewing experience. We designed a user study that compares users' self-reported levels of immersion, scene realism, and discomfort on a set of created models and comparison cases. Our results indicate that our modeled scenes produce a significantly higher sense of immersion than a basic dome geometry for the panorama when viewed in VR with head orientation and position tracking.

Supplementary Material

MP4 File (a32-sayyad.mp4)

Download
42.82 MB

References

[1]

Robert Anderson, David Gallup, Jonathan T Barron, Janne Kontkanen, Noah Snavely, Carlos Hernández, Sameer Agarwal, and Steven M Seitz. 2016. Jump: virtual reality video. ACM Transactions on Graphics (TOG) 35, 6 (2016), 198.

Digital Library

[2]

Nick Barber. 2017. Start Caring About VR And 360-Degree Video. (2017). https://www.forbes.com/sites/forrester/2017/01/10/start-caring-about-vr-and-360-degree-video/ accessed on July 1, 2017.

[3]

Jeff Butterworth, Andrew Davidson, Stephen Hench, and Marc. T. Olano. 1992. 3DM: A Three Dimensional Modeler Using a Head-mounted Display. In Proceedings of the 1992 Symposium on Interactive 3D Graphics (I3D '92). ACM, New York, NY, USA, 135--138.

Digital Library

[4]

Cesar Cadena, Yasir Latif, and Ian D. Reid. 2016. Measuring the performance of single image depth estimation methods. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 4150--4157.

Digital Library

[5]

Shenchang Eric Chen. 1995. QuickTime VR: An Image-based Approach to Virtual Environment Navigation. In Proceedings of the 22Nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '95). ACM, New York, NY, USA, 29--38.

Digital Library

[6]

Anoop Cherian, Vassilios Morellas, and Nikolaos Papanikolopoulos. 2009. Accurate 3D ground plane estimation from a single image. In Robotics and Automation, 2009. ICRA'09. IEEE International Conference on. IEEE, 2243--2249.

[7]

Antonio Criminisi, Ian Reid, and Andrew Zisserman. 1999. Single view metrology. In Computer Vision, 1999. The Proceedings of the Seventh IEEE International Conference on, Vol. 1. IEEE, 434--441.

[8]

Paul E Debevec, Camillo J Taylor, and Jitendra Malik. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. ACM, 11--20.

Digital Library

[9]

Michael F. Deering. 1995. HoloSketch: A Virtual Reality Sketching/Animation Tool. ACM Trans. Comput.-Hum. Interact. 2, 3 (Sept. 1995), 220--238.

Digital Library

[10]

Stephen DiVerdi, Jason Wither, and Tobias Hollerer. 2008. Envisor: Online environment map construction for mixed reality. In Virtual Reality Conference, 2008. VR'08. IEEE. IEEE, 19--26.

[11]

Facebook 2017. 360 Video. (2017). https://facebook360.fb.com/

[12]

Duke Gledhill, Gui Yun Tian, Dave Taylor, and David Clarke. 2003. Panoramic imagingfi!?a review. Computers & Graphics 27, 3 (jun 2003), 435--445.

[13]

Michael Gleicher. 1995. Image Snapping. In Proceedings of the 22Nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '95). ACM, New York, NY, USA, 183--190.

Digital Library

[14]

Google Blocks 2017. Create 3D models in VR - Google VR. (2017). https://vr.google.com/blocks/

[15]

Google Cardboard VR 2017. Cardboard Camera VR Photo Format. (2017). https://developers.google.com/vr/concepts/cardboard-camera-vr-photo-format

[16]

Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The lumigraph. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques - SIGGRAPH '96. ACM Press, New York, New York, USA, 43--54.

Digital Library

[17]

Luis E. Gurrieri and Eric Dubois. 2013. Stereoscopic cameras for the real-time acquisition of panoramic 3D images and videos, Andrew J. Woods, Nicolas S. Holliman, and Gregg E. Favalora (Eds.). International Society for Optics and Photonics, 86481W.

[18]

D. Hoiem, A. A. Efros, and M. Hebert. 2005. Geometric context from a single image. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, Vol. 1. 654--661 Vol. 1.

Digital Library

[19]

Youichi Horry, Ken-Ichi Anjyo, and Kiyoshi Arai. 1997. Tour into the picture: using a spidery mesh interface to make animation from a single image. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 225--232.

Digital Library

[20]

Fay Huang and Reinhard Klette. 2010. Stereo panorama acquisition and automatic image disparity adjustment for stereoscopic visualization. Multimedia Tools and Applications 47, 3 (2010), 353--377.

Digital Library

[21]

Bret Jackson and Daniel F. Keefe. 2016. Lift-Off: Using Reference Imagery and Freehand Sketching to Create 3D Models in VR. IEEE Transactions on Visualization and Computer Graphics 22, 4 (April 2016), 1442--1451.

Digital Library

[22]

Karol Kwiatek and Regina Tokarczyk. 2015. Immersive photogrammetry in 3D modelling. Geomatics and Environmental Engineering 9, 2 (2015).

[23]

J.J. LaViola, E. Kruijff, D.A. Bowman, R.P. McMahan, and I.P. Poupyrev. 2017. 3D User Interfaces: Theory and Practice. Pearson Education, Limited. https://books.google.com/books?id=ilUyjwEACAAJ

[24]

Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques - SIGGRAPH '96. ACM Press, New York, New York, USA, 31--42.

Digital Library

[25]

Congcong Li, Adarsh Kowdle, Ashutosh Saxena, and Tsuhan Chen. 2010. Towards holistic scene understanding: feedback enabled cascaded classification models. In Proceedings of the 23rd International Conference on Neural Information Processing Systems. Curran Associates Inc., 1351--1359. http://dl.acm.org/citation.cfm?id=2997047

[26]

Jiandong Liang and Mark Green. 1994. JDCAD: A highly interactive 3D modeling system. Computers & graphics 18, 4 (1994), 499--506.

[27]

Daniel P. Mapes and J. Michael Moshell. 1995. A Two-handed Interface for Object Manipulation in Virtual Environments. Presence: Teleoper. Virtual Environ. 4, 4 (Jan. 1995), 403--416.

Digital Library

[28]

Matterport 2017. Immersive 3D for the Real World. (2017). https://matterport.com/

[29]

Nokia Ozo 2017. 360 virtual reality camera. (2017). https://ozo.nokia.com/

[30]

Oculus Medium 2017. v. 1.1.2. (2017). https://www.oculus.com/experiences/rift/1336762299669605/

[31]

Byong Mok Oh, Max Chen, Julie Dorsey, and Frédo Durand. 2001. Image-based modeling and photo editing. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 433--442.

Digital Library

[32]

Shmuel Peleg, Moshe Ben-Ezra, and Yael Pritch. 2001. Omnistereo: Panoramic stereo imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 3 (2001), 279--290.

Digital Library

[33]

Ashutosh Saxena, Sung H Chung, and Andrew Y Ng. 2005. Learning depth from single monocular images. In NIPS, Vol. 18. 1--8.

[34]

A. Saxena, Min Sun, and A.Y. Ng. 2009. Make3D: Learning 3D Scene Structure from a Single Still Image. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 5 (may 2009), 824--840.

Digital Library

[35]

Richard Stoakley, Matthew J Conway, and Randy Pausch. 1995. Virtual reality on a WIM: interactive worlds in miniature. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM Press/Addison-Wesley Publishing Co., 265--272.

Digital Library

[36]

Richard Szeliski and Heung-Yeung Shum. 1997. Creating full view panoramic image mosaics and environment maps. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 251--258.

Digital Library

[37]

Unity - Game Engine 2017. v. 2017.1. (2017). https://unity3d.com/

[38]

Anton van den Hengel, Anthony Dick, Thorsten Thormählen, Ben Ward, and Philip H. S. Torr. 2007. VideoTrace: Rapid Interactive Scene Modelling from Video. ACM Trans. Graph. 26, 3, Article 86 (July 2007).

Digital Library

[39]

Christian Weissig, Oliver Schreer, Peter Eisert, and Peter Kauff. 2012. The Ultimate Immersive Experience: Panoramic 3D Video Acquisition. Springer, Berlin, Heidelberg, 671--681.

Digital Library

[40]

Jennifer Whyte,NBouchlaghem, A Thorpe, and R McCaffer. 2000. From CAD to virtual reality: modelling approaches, data exchange and interactive 3D building design tools. Automation in construction 10, 1 (2000), 43--55.

[41]

Yingen Xiong and Kari Pulli. 2010. Fast panorama stitching for high-quality panoramic images on mobile phones. IEEE Transactions on Consumer Electronics 56, 2 (2010).

Digital Library

[42]

YouTube VR 2017. Virtual Reality Videos. (2017). https://support.google.com/youtube/answer/6316263?hl=en

[43]

Li Zhang, Guillaume Dugas-Phocion, Jean-Sebastien Samson, and Steven M Seitz. 2002. Single-view modelling of free-form scenes. Computer Animation and Virtual Worlds 13, 4 (2002), 225--235.

[44]

Yinda Zhang, Shuran Song, Ping Tan, and Jianxiong Xiao. 2014. PanoContext: A Whole-Room 3D Context Model for Panoramic Scene Understanding. In ECCV: European Conference on Computer Vision. 668--686.

Cited By

Chang QZhang TLiu WCui Y(2024)FEDet: Feature Enhancement Object Detection with Panoramic Images2024 IEEE 4th International Conference on Software Engineering and Artificial Intelligence (SEAI)10.1109/SEAI62072.2024.10674235(92-98)Online publication date: 21-Jun-2024
https://doi.org/10.1109/SEAI62072.2024.10674235
Khan AHossain MCovaci ASirlantzis KXu C(2024)Light field imaging technology for virtual reality content creation: A reviewIET Image Processing10.1049/ipr2.1314418:11(2817-2837)Online publication date: 14-Jun-2024
https://doi.org/10.1049/ipr2.13144
Cha SMa JSeo JKim JHan S(2022)Empirical comparison of spatial experience between photo-based IVE and real spaceArchitectural Science Review10.1080/00038628.2022.213408966:1(1-16)Online publication date: 17-Oct-2022
https://doi.org/10.1080/00038628.2022.2134089
Show More Cited By

Index Terms

PanoTrace: interactive 3D modeling of surround-view panoramic images in virtual reality
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies
    2. Interaction paradigms
      1. Virtual reality

Recommendations

Design and Implementation of Virtual Reality Geometric Modeling in Apla^+VR
Structured Object-Oriented Formal Language and Method
Abstract
The geometric modeling languages for virtual reality could be classified into graphical modeling languages and script modeling languages. The graphical modeling languages are easier to use, but difficult to ensure the precision of 3D models. While ...
Haptic around: multiple tactile sensations for immersive environment and interaction in virtual reality
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

In this paper, we present Haptic Around, a hybrid-haptic feedback system, which utilizes fan, hot air blower, mist creator and heat light to recreate multiple tactile sensations in virtual reality for enhancing the immersive environment and interaction. ...
Viewing Geometric Protein Structures From Inside a CAVE

We compute and visualize four geometric protein models: the space filling diagram, the solvent accessible surface, the molecular surface, and the alpha complex. Relations between the models are illustrated via continuous deformations. A supercomputer ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '17: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology

November 2017

437 pages

ISBN:9781450355483

DOI:10.1145/3139131

General Chairs:
Morten Fjeld
Chalmers University of Technology
,
Marco Fratarcangeli
Chalmers University of Technology
,
Daniel Sjölie
University of Gothenburg
,
Program Chairs:
Oliver Staadt
University of Rostock
,
Jonas Unger
Linköping University

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 November 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Office of Naval Research

Conference

VRST '17

Sponsor:

VRST '17: 23rd ACM Symposium on Virtual Reality Software and Technology

November 8 - 10, 2017

Gothenburg, Sweden

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
1,015
Total Downloads

Downloads (Last 12 months)110
Downloads (Last 6 weeks)22

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chang QZhang TLiu WCui Y(2024)FEDet: Feature Enhancement Object Detection with Panoramic Images2024 IEEE 4th International Conference on Software Engineering and Artificial Intelligence (SEAI)10.1109/SEAI62072.2024.10674235(92-98)Online publication date: 21-Jun-2024
https://doi.org/10.1109/SEAI62072.2024.10674235
Khan AHossain MCovaci ASirlantzis KXu C(2024)Light field imaging technology for virtual reality content creation: A reviewIET Image Processing10.1049/ipr2.1314418:11(2817-2837)Online publication date: 14-Jun-2024
https://doi.org/10.1049/ipr2.13144
Cha SMa JSeo JKim JHan S(2022)Empirical comparison of spatial experience between photo-based IVE and real spaceArchitectural Science Review10.1080/00038628.2022.213408966:1(1-16)Online publication date: 17-Oct-2022
https://doi.org/10.1080/00038628.2022.2134089
Hsu WChen PChen C(2021)An Exploration of the Impact of Virtual Reality Interfaces on Online ShoppingResearch Anthology on E-Commerce Adoption, Models, and Applications for Modern Business10.4018/978-1-7998-8957-1.ch039(742-763)Online publication date: 2021
https://doi.org/10.4018/978-1-7998-8957-1.ch039
de Dinechin GPaljic ATanant J(2021)Impact of View-Dependent Image-Based Effects on Perception of Visual Realism and Presence in Virtual Reality Environments Created Using Multi-Camera SystemsApplied Sciences10.3390/app1113617311:13(6173)Online publication date: 2-Jul-2021
https://doi.org/10.3390/app11136173
Li JHe YJiao JHu YHan YWen JShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)Extending 6-DoF VR Experience Via Multi-Sphere Images InterpolationProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475339(4632-4640)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475339
Schäfer AReis GStricker D(2021)Investigating the Sense of Presence Between Handcrafted and Panorama Based Virtual EnvironmentsProceedings of Mensch und Computer 202110.1145/3473856.3474024(402-405)Online publication date: 5-Sep-2021
https://dl.acm.org/doi/10.1145/3473856.3474024
Hsu WChen PChen C(2020)An Exploration of the Impact of Virtual Reality Interfaces on Online ShoppingInformation Resources Management Journal10.4018/IRMJ.202004010233:2(19-39)Online publication date: 1-Apr-2020
https://doi.org/10.4018/IRMJ.2020040102
Bertel TYuan MLindroos RRichardt C(2020)OmniPhotosACM Transactions on Graphics10.1145/3414685.341777039:6(1-12)Online publication date: 27-Nov-2020
https://dl.acm.org/doi/10.1145/3414685.3417770
Pratapa SManocha D(2020)RANDM: Random Access Depth Map Compression Using Range-Partitioning and Global DictionarySymposium on Interactive 3D Graphics and Games10.1145/3384382.3384524(1-11)Online publication date: 5-May-2020
https://dl.acm.org/doi/10.1145/3384382.3384524
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten