Generative Terrain Authoring with Mid-air Hand Sketching in Virtual Reality
Authors: 
Yushen Hu, Keru Wang, Yuli Shao, Jan Plass, Zhu Wang, Ken PerlinAuthors Info & Claims
Yushen Hu, Keru Wang, Yuli Shao, Jan Plass, Zhu Wang, Ken PerlinAuthors Info & ClaimsArticle No.: 26, Pages 1 - 10
Abstract
Terrain generation and authoring in Virtual Reality (VR) offers unique benefits, including 360-degree views, improved spatial perception, immersive and intuitive design experience and natural input modalities. Yet even in VR it can be challenging to integrate natural input modalities, preserve artistic controls and lower the effort of landscape prototyping. To tackle these challenges, we present our VR-based terrain generation and authoring system, which utilizes hand tracking and a generative model to allow users to quickly prototype natural landscapes, such as mountains, mesas, canyons and volcanoes. Via positional hand tracking and hand gesture detection, users can use their hands to draw mid-air strokes to indicate desired shapes for the landscapes. A Conditional Generative Adversarial Network trained by using real-world terrains and their height maps then helps to generate a realistic landscape which combines features of training data and the mid-air strokes. In addition, users can use their hands to further manipulate their mid-air strokes to edit the landscapes. In this paper, we explore this design space and present various scenarios of terrain generation. Additionally, we evaluate our system across a diverse user base that varies in VR experience and professional background. The study results indicate that our system is feasible, user-friendly and capable of fast prototyping.
Supplemental Material
MP4 File
Supplemental video
- Download
- 239.00 MB
References
[1]
Rahul Arora and Karan Singh. 2021. Mid-air drawing of curves on 3d surfaces in virtual reality. ACM Transactions on Graphics (TOG) 40, 3 (2021), 1–17.
[2]
Christopher Beckham and Christopher Pal. 2017. A step towards procedural terrain generation with gans. arXiv preprint arXiv:1707.03383 (2017).
[3]
Jonathan M Cohen, John F Hughes, and Robert C Zeleznik. 2000. Harold: A world made of drawings. In Proceedings of the 1st international symposium on Non-photorealistic animation and rendering. 83–90.
[4]
Karin Coninx, Frank Van Reeth, and Eddy Flerackers. 1997. A hybrid 2D/3D user interface for immersive object modeling. In Proceedings Computer Graphics International. IEEE, 47–55.
[5]
John J Dudley, Hendrik Schuff, and Per Ola Kristensson. 2018. Bare-handed 3D drawing in augmented reality. In Proceedings of the 2018 Designing Interactive Systems Conference. 241–252.
[6]
Alain Fournier, Don Fussell, and Loren Carpenter. 1982. Computer rendering of stochastic models. Commun. ACM 25, 6 (1982), 371–384.
[7]
James Gain, Patrick Marais, and Wolfgang Straßer. 2009. Terrain sketching. In Proceedings of the 2009 symposium on Interactive 3D graphics and games. 31–38.
[8]
James Gain, Bruce Merry, and Patrick Marais. 2015. Parallel, realistic and controllable terrain synthesis. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 105–116.
[9]
Eric Galin, Eric Guérin, Adrien Peytavie, Guillaume Cordonnier, Marie-Paule Cani, Bedrich Benes, and James Gain. 2019. A review of digital terrain modeling. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 553–577.
[10]
Éric Guérin, Julie Digne, Eric Galin, Adrien Peytavie, Christian Wolf, Bedrich Benes, and Benoît Martinez. 2017. Interactive example-based terrain authoring with conditional generative adversarial networks.ACM Trans. Graph. 36, 6 (2017), 228–1.
[11]
Eric Guérin, Adrien Peytavie, Simon Masnou, Julie Digne, Basile Sauvage, James Gain, and Eric Galin. 2022. Gradient Terrain Authoring. In Computer Graphics Forum, Vol. 41. Wiley Online Library, 85–95.
[12]
Takeo Igarashi and John F Hughes. 2007. A suggestive interface for 3D drawing. In ACM SIGGRAPH 2007 courses. 20–es.
[13]
Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka. 2006. Teddy: a sketching interface for 3D freeform design. In ACM SIGGRAPH 2006 Courses. 11–es.
[14]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1125–1134.
[15]
Hans-Christian Jetter, Roman Rädle, Tiare Feuchtner, Christoph Anthes, Judith Friedl, and Clemens Nylandsted Klokmose. 2020. " in vr, everything is possible!": Sketching and simulating spatially-aware interactive spaces in virtual reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–16.
[16]
David Ledo, Steven Houben, Jo Vermeulen, Nicolai Marquardt, Lora Oehlberg, and Saul Greenberg. 2018. Evaluation Strategies for HCI Toolkit Research. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–17. https://doi.org/10.1145/3173574.3173610
[17]
Joshua Lochner, James Gain, Simon Perche, Adrien Peytavie, Eric Galin, and Eric Guérin. 2023. Interactive Authoring of Terrain using Diffusion Models. In Computer Graphics Forum. Wiley Online Library, e14941.
[18]
Benoit B Mandelbrot and Benoit B Mandelbrot. 1982. The fractal geometry of nature. Vol. 1. WH freeman New York.
[19]
Masatoshi Matsumiya, Haruo Takemura, and Naokazu Yokoya. 2000. An immersive modeling system for 3d free-form design using implicit surfaces. In Proceedings of the ACM symposium on Virtual reality software and technology. 67–74.
[20]
Gavin SP Miller. 1986. The definition and rendering of terrain maps. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques. 39–48.
[21]
F Kenton Musgrave, Craig E Kolb, and Robert S Mace. 1989. The synthesis and rendering of eroded fractal terrains. ACM Siggraph Computer Graphics 23, 3 (1989), 41–50.
[22]
Shanthika Naik, Aryamaan Jain, Avinash Sharma, and KS Rajan. 2022. Deep Generative Framework for Interactive 3D Terrain Authoring and Manipulation. In IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium. IEEE, 6410–6413.
[23]
Emmanouil Panagiotou and Eleni Charou. 2020. Procedural 3D terrain generation using Generative Adversarial Networks. arXiv preprint arXiv:2010.06411 (2020).
[24]
Simon Perche, Adrien Peytavie, Bedrich Benes, Eric Galin, and Eric Guérin. 2023. Authoring terrains with spatialised style. In Computer Graphics Forum, Vol. 42. Wiley Online Library, e14936.
[25]
Helen Perkunder, Johann Habakuk Israel, and Marc Alexa. 2010. Shape Modeling with Sketched Feature Lines in Immersive 3D Environments. In SBIM. 127–134.
[26]
Ken Perlin. 1985. An image synthesizer. ACM Siggraph Computer Graphics 19, 3 (1985), 287–296.
[27]
Ken Perlin and Eric M Hoffert. 1989. Hypertexture. In Proceedings of the 16th annual conference on Computer graphics and interactive techniques. 253–262.
[28]
Adrien Peytavie, Eric Galin, Jérôme Grosjean, and Stéphane Mérillou. 2009. Arches: a framework for modeling complex terrains. In Computer graphics forum, Vol. 28. Wiley Online Library, 457–467.
[29]
Ken Pfeuffer, Benedikt Mayer, Diako Mardanbegi, and Hans Gellersen. 2017. Gaze+ pinch interaction in virtual reality. In Proceedings of the 5th symposium on spatial user interaction. 99–108.
[30]
Pascale Roudier, Bernard Peroche, and Michel Perrin. 1993. Landscapes synthesis achieved through erosion and deposition process simulation. In Computer Graphics Forum, Vol. 12. Wiley Online Library, 375–383.
[31]
Kristofer Schlachter, Benjamin Ahlbrand, Zhu Wang, Ken Perlin, and Valerio Ortenzi. 2022. Zero-shot multi-modal artist-controlled retrieval and exploration of 3d object sets. In SIGGRAPH Asia 2022 Technical Communications. 1–4.
[32]
ryan rs spick and james walker. 2019. Realistic and textured terrain generation using GANs. In Proceedings of the 16th ACM SIGGRAPH European Conference on Visual Media Production. 1–10.
[33]
Payam Tabrizian, Anna Petrasova, Brendan Harmon, Vaclav Petras, Helena Mitasova, and Ross Meentemeyer. 2016. Immersive tangible geospatial modeling. In Proceedings of the 24th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. 1–4.
[34]
François-Xavier Talgorn and Farès Belhadj. 2018. Real-time sketch-based terrain generation. In Proceedings of Computer Graphics International 2018. 13–18.
[35]
Osama Tolba, Julie Dorsey, and Leonard McMillan. 1999. Sketching with projective 2D strokes. In Proceedings of the 12th annual ACM symposium on User interface software and technology. 149–157.
[36]
Richard F Voss. 1985. Random fractal forgeries. In Fundamental Algorithms for Computer Graphics: NATO Advanced Study Institute directed by JE Bresenham, RA Earnshaw, MLV Pitteway. Springer, 805–835.
[37]
Shing Ming Wong, Chien-Wen Chen, Tse-Yu Pan, Hung-Kuo Chu, and Min-Chun Hu. 2022. GetWild: A VR Editing System with AI-Generated 3D Object and Terrain. In Proceedings of the 30th ACM International Conference on Multimedia. 6988–6990.
[38]
Burkhard Wünsche, Daniel Keymer, and Robert Amor. 2010. Sketch, click, plug and play: accelerated design of virtual environments by integrating multimedia and sketch content into game engines. In Proceedings of the 11th International Conference of the NZ Chapter of the ACM Special Interest Group on Human-Computer Interaction. 33–40.
[39]
Robert C Zeleznik, Kenneth P Herndon, and John F Hughes. 2006. SKETCH: An interface for sketching 3D scenes. In ACM SIGGRAPH 2006 Courses. 9–es.
[40]
Jian Zhang, Chen Li, Peichi Zhou, Changbo Wang, Gaoqi He, and Hong Qin. 2022. Authoring multi-style terrain with global-to-local control. Graphical Models 119 (2022), 101122.
[41]
JM Zheng, KW Chan, and Ian Gibson. 1999. A VR based 3D graphics user interface for CAD modeling system. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 19715. American Society of Mechanical Engineers, 1029–1036.
[42]
Howard Zhou, Jie Sun, Greg Turk, and James M Rehg. 2007. Terrain synthesis from digital elevation models. IEEE transactions on visualization and computer graphics 13, 4 (2007), 834–848.
Index Terms
- Generative Terrain Authoring with Mid-air Hand Sketching in Virtual Reality
Recommendations
The Gesture Authoring Space: Authoring Customised Hand Gestures for Grasping Virtual Objects in Immersive Virtual Environments
MuC '22: Proceedings of Mensch und Computer 2022Natural user interfaces are on the rise. Manufacturers for Augmented, Virtual, and Mixed Reality head mounted displays are increasingly integrating new sensors into their consumer grade products, allowing gesture recognition without additional hardware. ...
Enabling Reusable Haptic Props for Virtual Reality by Hand Displacement
MuC '21: Proceedings of Mensch und Computer 2021Virtual Reality (VR) enables compelling visual experiences. However, providing haptic feedback is still challenging. Previous work suggests utilizing haptic props to overcome such limitations and presents evidence that props could function as a single ...
Virtual Grasping Feedback and Virtual Hand Ownership
SAP '19: ACM Symposium on Applied Perception 2019In this work, we investigate the influence of different visualizations on a manipulation task in virtual reality (VR). Without the haptic feedback of the real world, grasping in VR might result in intersections with virtual objects. As people are highly ...
Comments
Information & Contributors
Information
Published In
October 2024
633 pages
ISBN:9798400705359
DOI:10.1145/3641825
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 09 October 2024
Check for updates
Badges
Best Paper
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
VRST '24: 30th ACM Symposium on Virtual Reality Software and Technology
October 9 - 11, 2024
Trier, Germany
Acceptance Rates
Overall Acceptance Rate 66 of 254 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 343Total Downloads
- Downloads (Last 12 months)343
- Downloads (Last 6 weeks)99
Reflects downloads up to 25 Jan 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in