Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3229147.3229166acmconferencesArticle/Chapter ViewAbstractPublication PagessiggraphConference Proceedingsconference-collections
research-article

3D sketching for interactive model retrieval in virtual reality

Published: 17 August 2018 Publication History

Abstract

We describe a novel method for searching 3D model collections using free-form sketches within a virtual environment as queries. As opposed to traditional sketch retrieval, our queries are drawn directly onto an example model. Using immersive virtual reality the user can express their query through a sketch that demonstrates the desired structure, color and texture. Unlike previous sketch-based retrieval methods, users remain immersed within the environment without relying on textual queries or 2D projections which can disconnect the user from the environment. We perform a test using queries over several descriptors, evaluating the precision in order to select the most accurate one. We show how a convolutional neural network (CNN) can create multi-view representations of colored 3D sketches. Using such a descriptor representation, our system is able to rapidly retrieve models and in this way, we provide the user with an interactive method of navigating large object datasets. Through a user study we demonstrate that by using our VR 3D model retrieval system, users can perform search more quickly and intuitively than with a naive linear browsing method. Using our system users can rapidly populate a virtual environment with specific models from a very large database, and thus the technique has the potential to be broadly applicable in immersive editing systems.

Supplementary Material

ZIP File (01-033.zip)
Supplemental files.

References

[1]
Alaa E. Abdel-Hakim and Aly A. Farag. 2006. CSIFT: A SIFT Descriptor with Color Invariant Characteristics. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), 17-22 June 2006, New York, NY, USA. 1978--1983.
[2]
Vandeborre J.P. Ansary T.F., Daoudi M. 2007. A Bayesian 3-D Search Engine Using Adaptive Views Clustering. IEEE Transaction on Multimedia 9, 1 (2007), 78 âĂŞ-88.
[3]
Jonathan Ashley, Myron Flickner, James L. Hafner, Denis Lee, Wayne Niblack, and Dragutin Petkovic. 1995. The Query By Image Content (QBIC) System. In Proceedings of the 1995 ACM SIGMOD International Conference on Management of Data, San Jose, California, May 22-25, 1995. 475.
[4]
Martins N. V. Beatriz S. 2015. Air-Sketching for Object Retrieval. Technical Report. Instituto Superior Tecnico, Lisboa, Portugao.
[5]
Alberto Del Bimbo, Pietro Pala, and Simone Santini. 1994. Visual Image Retrieval by Elastic Deformation of Object Sketches. In Proceedings IEEE Symposium on Visual Languages, St. Louis, Missouri, USA, October 4-7, 1994. 216--223.
[6]
Shinde J.V. Birari D.R. 2015. Survey on Sketch Based Image Retrieval. International Journal of Advanced Research in Computer and Communication Engineering 4, 12 (2015).
[7]
Arthur W. Brody and Chris Hartman. 1999. BLUI: a body language user interface for 3D gestural drawing. In Human Vision and Electronic Imaging (SPIE Proceedings), Vol. 3644. SPIE, 356--363.
[8]
Kimmel R. Bronstein M. A., Bronstein M. M. 2009. Topology-Invariant Similarity of Nonrigid Shapes. Int. J. Comput. Vis. 81 (2009), 281 âĂŞ-- 301.
[9]
T. Bui, L. Ribeiro, M. Ponti, and J. Collomosse. 2017. Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network. Computer Vision and Image Understanding (2017).
[10]
Tu Bui, Leonardo Ribeiro, Moacir Ponti, and John P. Collomosse. 2016. Generalisation and Sharing in Triplet Convnets for Sketch based Visual Search. CoRR abs/1611.05301 (2016).
[11]
Jeff Butterworth, Andrew Davidson, Stephen Hench, and Marc. T. Olano. 1992. 3DM: A Three Dimensional Modeler Using a Head-mounted Display. In Proceedings of the 1992 Symposium on Interactive 3D Graphics (I3D '92). ACM, New York, NY, USA, 135--138.
[12]
Abdolah Chalechale, Golshah Naghdy, and Alfred Mertins. 2005. Sketch-based image matching Using Angular partitioning. IEEE Trans. Systems, Man, and Cybernetics, Part A 35, 1 (2005), 28--41.
[13]
K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. 2014. Return of the Devil in the Details: Delving Deep into Convolutional Nets. In British Machine Vision Conference. arXiv:cs/1405.3531
[14]
Han-wool Choi, Hee-joon Kim, Jeong-in Lee, and Young-Ho Chai. 2005. Free Hand Stroke Based Virtual Sketching, Deformation and Sculpting of NURBS Surface. In Proceedings of the 2005 International Conference on Augmented Tele-existence (ICAT '05). ACM, New York, NY, USA, 3--9.
[15]
James H. Clark. 1976. Designing Surfaces in 3-D. Commun. ACM 19, 8 (Aug. 1976), 454--460.
[16]
John P. Collomosse, Graham McNeill, and Leon Adam Watts. 2008. Free-hand sketch grouping for video retrieval. In ICPR. IEEE Computer Society, 1--4.
[17]
Navneet Dalal and Bill Triggs. 2005. Histograms of Oriented Gradients for Human Detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), 20-26 June 2005, San Diego, CA, USA. 886--893.
[18]
Michael F. Deering. 1996. The HoloSketch VR Sketching System. Commun. ACM 39, 5 (May 1996), 54--61.
[19]
Mathias Eitz, Kristian Hildebrand, Tamy Boubekeur, and Marc Alexa. 2011. Sketch-Based Image Retrieval: Benchmark and Bag-of-Features Descriptors. IEEE Transactions on Visualization and Computer Graphics 17, 11 (2011), 1624--1636.
[20]
Mathias Eitz, Ronald Richter, Tamy Boubekeur, Kristian Hildebrand, and Marc Alexa. 2012. Sketch-based shape retrieval. ACM Trans. Graph. 31, 4 (2012), 31:1--31:10.
[21]
Alexa M. Eitz M., Hays J. 2012. How Do Humans Sketch Objects?. In ACM Trans. Graphics, 31(4).
[22]
Michele Fiorentino, Giuseppe Monno, Pietro Renzulli, and Antonio Uva. 2003. 3D sketch stroke segmentation and fitting in virtual reality. (01 2003).
[23]
Manuel J. Fonseca, Alfredo Ferreira, and Joaquim A. Jorge. 2004. Towards 3D Modeling Using Sketches and Retrieval. In Proceedings of the First Eurographics Conference on Sketch-Based Interfaces and Modeling (SBM'04). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 127--136.
[24]
M. J. Fonseca, S. James, and J. Collomosse. 2012. Skeletons from sketches of dancing poses. In 2012 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 247--248.
[25]
Thomas Funkhouser, Patrick Min, Michael Kazhdan, Joyce Chen, Alex Halderman, David Dobkin, and David Jacobs. 2003. A Search Engine for 3D Models. ACM Trans. Graph. 22, 1 (Jan. 2003), 83--105.
[26]
Rui Hu and John Collomosse. 2013. A Performance Evaluation of Gradient Field HOG Descriptor for Sketch Based Image Retrieval. Comput. Vis. Image Underst. 117, 7 (July 2013), 790--806.
[27]
Stuart James and John Collomosse. 2014. Interactive Video Asset Retrieval Using Sketched Queries. In Proceedings of the 11th European Conference on Visual Media Production (CVMP '14). ACM, New York, NY, USA, Article 11, 8 pages.
[28]
Ricardo Jota, Alfredo Ferreira, Mariana Cerejo, José Santos, Manuel J. Fonseca, and Joaquim A. Jorge. 2006. Recognizing Hand Gestures with CALI. In SIACG. Eurographics Association, 187--193.
[29]
Daniel F. Keefe, Daniel Acevedo Feliz, Tomer Moscovich, David H. Laidlaw, and Joseph J. LaViola, Jr. 2001. CavePainting: A Fully Immersive 3D Artistic Medium and Interactive Experience. In Proceedings of the 2001 Symposium on Interactive 3D Graphics (I3D '01). ACM, New York, NY, USA, 85--93.
[30]
Daniel F. Keefe, Robert C. Zeleznik, and David H. Laidlaw. 2007. Drawing on Air: Input Techniques for Controlled 3D Line Illustration. 13, 5 (2007), 1067--1081.
[31]
Yann Lecun, LÃl'on Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. In Proceedings of the IEEE. 2278--2324.
[32]
Yong Jae Lee, C. Lawrence Zitnick, and Michael F. Cohen. 2011. ShadowDraw: real-time user guidance for freehand drawing. ACM Trans. Graph. 30, 4 (2011), 27:1--27:10.
[33]
Biao Leng, Yu Liu, Kai Yu, Xiangyang Zhang, and Zhang Xiong. 2016. 3D Object Understanding with 3D Convolutional Neural Networks. Inf. Sci. 366, C (Oct. 2016), 188--201.
[34]
Bo Li, Yijuan Lu, Fuqing Duan, Shuilong Dong, Yachun Fan, Lu Qian, Hamid Laga, Haisheng Li, Yuxiang Li, Peng Liu, Maks Ovsjanikov, Hedi Tabia, Yuxiang Ye, Huanpu Yin, and Ziyu Xue. 2016a. 3D Sketch-based 3D Shape Retrieval. In Proceedings of the Eurographics 2016 Workshop on 3D Object Retrieval (3DOR '16). Eurographics Association, Goslar Germany, Germany, 47--54.
[35]
Bo Li, Yijuan Lu, Fuqing Duan, Shuilong Dong, Yachun Fan, Lu Qian, Hamid Laga, Haisheng Li, Yuxiang Li, Peng Liu, Maks Ovsjanikov, Hedi Tabia, Yuxiang Ye, Huanpu Yin, and Ziyu Xue. 2016b. 3D Sketch-Based 3D Shape Retrieval. In Eurographics Workshop on 3D Object Retrieval, A. Ferreira, A. Giachetti, and D. Giorgi (Eds.). The Eurographics Association.
[36]
Bo Li, Yijuan Lu, Henry Johan, and Ribel Fares. 2017. Sketch-based 3D Model Retrieval Utilizing Adaptive View Clustering and Semantic Information. Multimedia Tools Appl. 76, 24 (Dec. 2017), 26603--26631.
[37]
David G. Lowe. 2004. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 60, 2 (01 Nov 2004), 91--110.
[38]
Gen Nishida, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Adrien Bousseau. 2016. Interactive Sketching of Urban Procedural Models. ACM Trans. Graph. 35, 4, Article 130 (July 2016), 11 pages.
[39]
Robert Osada, Thomas Funkhouser, Bernard Chazelle, and David Dobkin. 2002. Shape Distributions. ACM Transactions on Graphics 21 (2002), 807--832.
[40]
Yonggang Qi, Yi-Zhe Song, Honggang Zhang, and Jun Liu. 2016. Sketch-based image retrieval via Siamese convolutional neural network. In 2016 IEEE International Conference on Image Processing, ICIP 2016, Phoenix, AZ, USA, September 25-28, 2016. 2460--2464.
[41]
Dominik Rausch, Ingo Assenmacher, and Torsten W. Kuhlen. 2010. 3D Sketch Recognition for Interaction in Virtual Environments. In Proceedings of the Seventh Workshop on Virtual Reality Interactions and Physical Simulations, VRIPHYS 2010, Copenhagen, Denmark, 2010. 115--124.
[42]
David Gavilan Ruiz, Suguru Saito, and Masayuki Nakajima. 2007. Sketch-to-collage. In SIGGRAPH Posters. ACM, 35.
[43]
Raif M. Rustamov. 2010. Robust Volumetric Shape Descriptor. In 3DOR. Eurographics Association, 1--5.
[44]
Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The sketchy database: learning to retrieve badly drawn bunnies. ACM Trans. Graph. 35, 4 (2016), 119:1--119:12.
[45]
Tiago Santos, Alfredo Ferreira, Filipe Dias, and Manuel J. Fonseca. 2008. Using Sketches and Retrieval to Create LEGO Models. In Proceedings of the Fifth Eurographics Conference on Sketch-Based Interfaces and Modeling (SBM'08). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 89--96.
[46]
Steven Schkolne, Michael Pruett, and Peter Schröder. 2001. Surface Drawing: Creating Organic 3D Shapes with the Hand and Tangible Tools. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '01). ACM, New York, NY, USA, 261--268.
[47]
Eugenio Di Sciascio, G. Mingolla, and Marina Mongiello. 1999. Content-Based Image Retrieval over the Web Using Query by Sketch and Relevance Feedback. In VISUAL (Lecture Notes in Computer Science), Vol. 1614. Springer, 123--130.
[48]
HyoJong Shin and Takeo Igarashi. 2007. Magic Canvas: Interactive Design of a 3-D Scene Prototype from Freehand Sketches. In Proceedings of Graphics Interface 2007 (GI '07). ACM, New York, NY, USA, 63--70.
[49]
Pedro Manuel Antunes Sousa and Manuel J. Fonseca. 2010. Sketch-based retrieval of drawings using spatial proximity. J. Vis. Lang. Comput. 21, 2 (2010), 69--80.
[50]
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. 2015a. Multi-view Convolutional Neural Networks for 3D Shape Recognition. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV) (ICCV '15). IEEE Computer Society, Washington, DC, USA, 945--953.
[51]
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learned-Miller. 2015b. Multi-view convolutional neural networks for 3d shape recognition. In Proc. ICCV.
[52]
Fang Wang, Le Kang, and Yi Li. 2015. Sketch-based 3D Shape Retrieval using Convolutional Neural Networks. CoRR abs/1504.03504 (2015). arXiv:1504.03504 http://arxiv.org/abs/1504.03504
[53]
Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu. 2014. Learning Fine-grained Image Similarity with Deep Ranking. CoRR abs/1404.4661 (2014).
[54]
Gerold Wesche and Hans-Peter Seidel. 2001. FreeDrawer: A Free-form Sketching System on the Responsive Workbench. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '01). ACM, New York, NY, USA, 167--174.
[55]
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D ShapeNets: A deep representation for volumetric shapes. In CVPR. IEEE Computer Society, 1912--1920.
[56]
Kun Xu, Kang Chen, Hongbo Fu, Wei-Lun Sun, and Shi-Min Hu. 2013. Sketch2Scene: Sketch-based Co-retrieval and Co-placement of 3D Models. ACM Transactions on Graphics 32, 4 (2013), 123:1--123:12.
[57]
Yongxin Yang and Timothy M. Hospedales. 2015. Deep Neural Networks for Sketch Recognition. CoRR abs/1501.07873 (2015). http://arxiv.org/abs/1501.07873
[58]
Yuxiang Ye, Bo Li, and Yijuan Lu. 2016. 3D sketch-based 3D model retrieval with convolutional neural network., 2936-2941 pages.
[59]
Q. Yu, F. Liu, Y. Z. Song, T. Xiang, T. M. Hospedales, and C. C. Loy. 2016. Sketch Me That Shoe. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 799--807.

Cited By

View all
  • (2025)Real Estate Information Exploration in VR with LoD Control by Physical DistanceDatabase Systems for Advanced Applications. DASFAA 2024 International Workshops10.1007/978-981-96-0914-7_21(269-276)Online publication date: 23-Jan-2025
  • (2024)Reality3DSketch: Rapid 3D Modeling of Objects From Single Freehand SketchesIEEE Transactions on Multimedia10.1109/TMM.2023.332753326(4859-4870)Online publication date: 1-Jan-2024
  • (2024)From sketch to reality: precision-friendly 3D generation technologyThe Visual Computer10.1007/s00371-024-03425-z41:2(1367-1378)Online publication date: 16-May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
Expressive '18: Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering
August 2018
200 pages
ISBN:9781450358927
DOI:10.1145/3229147
  • General Chairs:
  • Brian Wyvill,
  • Hongbo Fu
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

  • EUROGRAPHICS: The European Association for Computer Graphics

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. CNN
  2. HCI
  3. sketch
  4. virtual reality

Qualifiers

  • Research-article

Funding Sources

  • European Union

Conference

Expressive '18
Sponsor:

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)93
  • Downloads (Last 6 weeks)4
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Real Estate Information Exploration in VR with LoD Control by Physical DistanceDatabase Systems for Advanced Applications. DASFAA 2024 International Workshops10.1007/978-981-96-0914-7_21(269-276)Online publication date: 23-Jan-2025
  • (2024)Reality3DSketch: Rapid 3D Modeling of Objects From Single Freehand SketchesIEEE Transactions on Multimedia10.1109/TMM.2023.332753326(4859-4870)Online publication date: 1-Jan-2024
  • (2024)From sketch to reality: precision-friendly 3D generation technologyThe Visual Computer10.1007/s00371-024-03425-z41:2(1367-1378)Online publication date: 16-May-2024
  • (2023)Towards Sustainable Virtual Reality: Gathering Design Guidelines for Intuitive Authoring ToolsSustainability10.3390/su1504292415:4(2924)Online publication date: 6-Feb-2023
  • (2023)Eyes on Immersive Search: Eye-Tracking Study of Search Engine Result Pages in Immersive Virtual EnvironmentsProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578306(43-54)Online publication date: 19-Mar-2023
  • (2023)When XR and AI Meet - A Scoping Review on Extended Reality and Artificial IntelligenceProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581072(1-45)Online publication date: 19-Apr-2023
  • (2023)Tesseract: Querying Spatial Design Recordings by Manipulating Worlds in MiniatureProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580876(1-16)Online publication date: 19-Apr-2023
  • (2023)Integrating AEC Domain-Specific Multidisciplinary Knowledge for Informed and Interactive Feedback in Early Design StagesAdvances in Information Technology in Civil and Building Engineering10.1007/978-3-031-32515-1_12(153-171)Online publication date: 30-Aug-2023
  • (2022)Learning Transferable and Discriminative Representations for 2D Image-Based 3D Model RetrievalIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2022.316896732:10(7147-7159)Online publication date: Oct-2022
  • (2022)Evaluation of design guidelines for the development of intuitive virtual reality authoring tools: a case study with NVIDIA Omniverse2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct57072.2022.00078(357-362)Online publication date: Oct-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media