research-article

How do humans sketch objects?

Authors:

Marc AlexaAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 31, Issue 4

Article No.: 44, Pages 1 - 10

https://doi.org/10.1145/2185520.2185540

Published: 01 July 2012 Publication History

Abstract

Humans have used sketching to depict our visual world since prehistoric times. Even today, sketching is possibly the only rendering technique readily available to all humans. This paper is the first large scale exploration of human sketches. We analyze the distribution of non-expert sketches of everyday objects such as 'teapot' or 'car'. We ask humans to sketch objects of a given category and gather 20,000 unique sketches evenly distributed over 250 object categories. With this dataset we perform a perceptual study and find that humans can correctly identify the object category of a sketch 73% of the time. We compare human performance against computational recognition methods. We develop a bag-of-features sketch representation and use multi-class support vector machines, trained on our sketch dataset, to classify sketches. The resulting recognition method is able to identify unknown sketches with 56% accuracy (chance is 0.4%). Based on the computational model, we demonstrate an interactive sketch recognition system. We release the complete crowd-sourced dataset of sketches to the community.

Supplementary Material

JPG File (tp117_12.jpg)

Download
16.88 KB

ZIP File (a44-eitz.zip)

Supplemental material.

Download
90.46 MB

MP4 File (tp117_12.mp4)

Download
37.85 MB

References

[1]

Chalechale, A., Naghdy, G., and Mertins, A. 2005. Sketch-based image matching using angular partitioning. IEEE Trans. Systems, Man and Cybernetics, Part A 35, 1, 28--41.

Digital Library

[2]

Chen, T., Cheng, M., Tan, P., Shamir, A., and Hu, S. 2009. Sketch2Photo: internet image montage. ACM Trans. Graph. (Proc. SIGGRAPH ASIA) 28, 5, 124:1--124:10.

Digital Library

[3]

Datta, R., Joshi, D., Li, J., and Wang, J. 2008. Image retrieval: ideas, influences, and trends of the new age. ACM Computing Surveys 40, 2, 1--60.

Digital Library

[4]

Dixon, D., Prasad, M., and Hammond, T. 2010. iCanDraw?: using sketch recognition and corrective feedback to assist a user in drawing human faces. In Proc. Int'l. Conf. on Human Factors in Computing Systems, 897--906.

Digital Library

[5]

Eitz, M., Hildebrand, K., Boubekeur, T., and Alexa, M. 2011. Sketch-based image retrieval: benchmark and bag-of-features descriptors. IEEE Trans. Visualization and Computer Graphics 17, 11, 1624--1636.

Digital Library

[6]

Eitz, M., Richter, R., Hildebrand, K., Boubekeur, T., and Alexa, M. 2011. Photosketcher: interactive sketch-based image synthesis. IEEE Computer Graphics and Applications 31, 6, 56--66.

Digital Library

[7]

Eitz, M., Richter, R., Boubekeur, T., Hildebrand, K., and Alexa, M. 2012. Sketch-based shape retrieval. ACM Trans. Graph. (Proc. SIGGRAPH) 31, 4. to appear.

Digital Library

[8]

Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., and Zisserman, A. 2010. The PASCAL visual object classes (VOC) challenge. Int'l. Journal of Computer Vision 88, 2, 303--338.

Digital Library

[9]

Fu, H., Zhou, S., Liu, L., and Mitra, N. 2011. Animated construction of line drawings. ACM Trans. Graph. (Proc. SIGGRAPH ASIA) 30, 6, 133:1--133:10.

Digital Library

[10]

Garland, M., and Heckbert, P. 1997. Surface simplification using quadric error metrics. in Proc. SIGGRAPH, 209--216.

Digital Library

[11]

Georgescu, B., Shimshoni, I., and Meer, P. 2003. Mean shift based clustering in high dimensions: a texture classification example. in IEEE Int'l. Conf. Computer Vision, 456--463.

Digital Library

[12]

Griffin, G., Holub, A., and Perona, P. 2007. Caltech-256 object category dataset. Tech. rep., California institute of Technology.

[13]

Hammond, T., and Davis, R. 2005. LADDER, a sketching language for user interface developers. Computers & Graphics 29, 4, 518--532.

Digital Library

[14]

Herot, C. F. 1976. Graphical input through machine recognition of sketches. Computer Graphics (Proc. SIGGRAPH) 10, 2, 97--102.

Digital Library

[15]

LaViola Jr., J. J., and Zeleznik, R. 2007. MathPad: a system for the creation and exploration of mathematical sketches. ACM Trans. Graph. (Proc. SIGGRAPH) 23, 3, 432--440.

Digital Library

[16]

Lazebnik, S., Schmid, C., and Ponce, J. 2006. Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In IEEE Conf. Computer Vision and Pattern Recognition, 2169--2178.

Digital Library

[17]

Lee, Y., Zitnick, C., and Cohen, M. 2011. ShadowDraw: real-time user guidance for freehand drawing. ACM Trans. Graph. (Proc. SIGGRAPH) 30, 4, 27:1--27:10.

Digital Library

[18]

Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. Int'l. Journal of Computer Vision 60, 2, 91--110.

Digital Library

[19]

Ouyang, T., and Davis, R. 2011. ChemInk: a natural real-time recognition system for chemical drawings. In Proc. Int'l. Conf. Intelligent User Interfaces, 267--276.

Digital Library

[20]

Paulson, B., and Hammond, T. 2008. PaleoSketch: accurate primitive sketch recognition and beautification. In Proc. Int'l. Conf. Intelligent User Interfaces, 1--10.

Digital Library

[21]

Philbin, J., Chum, O., Isard, M., Sivic, J., and Zisserman, A. 2008. Lost in quantization: improving particular object retrieval in large scale image databases. In IEEE Conf. Computer Vision and Pattern Recognition, 1--8.

[22]

Russell, B., Torralba, A., Murphy, K., and Freeman, W. 2008. LabelMe: a database and web-based tool for image annotation. Int'l Journal of Computer Vision 77, 1, 157--173.

Digital Library

[23]

Samet, H. 2006. Foundations of multidimensional and metric data structures. Morgan Kaufmann.

Digital Library

[24]

Schölkopf, B., and Smola, A. 2002. Learning with kernels. MIT Press.

[25]

Sezgin, T. M., Stahovich, T., and Davis, R. 2001. Sketch based interfaces: early processing for sketch understanding. In Workshop on Perceptive User Interfaces, 1--8.

Digital Library

[26]

Shilane, P., Min, P., Kazhdan, M., and Funkhouser, T. 2004. The Princeton Shape Benchmark. In Shape Modeling International, 167--178.

Digital Library

[27]

Shrivastava, A., Malisiewicz, T., Gupta, A., and Efros, A. A. 2011. Data-driven visual similarity for cross-domain image matching. ACM Trans. Graph. (Proc. SIGGRAPH ASIA) 30, 6, 154:1--154:10.

Digital Library

[28]

Sivic, J., and Zisserman, A. 2003. Video Google: a textretrieval approach to object matching in videos. In IEEE Int'l. Conf. Computer Vision, 1470--1477.

Digital Library

[29]

Sutherland, I. 1964. SketchPad: a man-machine graphical communication system. In Proc. AFIPS, 323--328.

Digital Library

[30]

van der Maaten, L., and Hinton, G. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, 2579--2605.

[31]

Walther, D., Chai, B., Caddigan, E., Beck, D., and FeiFei, L. 2011. Simple line drawings suffice for functional MRI decoding of natural scene categories. Proc. National Academy of Sciences 108, 23, 9661--9666.

[32]

Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. 2010. SUN database: large-scale scene recognition from abbey to zoo. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, 3485--3492.

Cited By

Yang RLi DHu CZhang H(2025)SKVOS: Sketch-Based Video Object Segmentation with a Large-Scale BenchmarkApplied Sciences10.3390/app1504175115:4(1751)Online publication date: 9-Feb-2025
https://doi.org/10.3390/app15041751
Liang SDai WCheng CWei YCai Y(2025)Rethinking sketch-based 3D shape retrieval: A simple baseline and benchmark reconstructionNeurocomputing10.1016/j.neucom.2024.128990618(128990)Online publication date: Feb-2025
https://doi.org/10.1016/j.neucom.2024.128990
Chen YLiu YKayar-Ceylan G(2025)CSG-based ML-supported 3D translation of sketches into game assets for game designersThe Visual Computer10.1007/s00371-024-03758-9Online publication date: 5-Jan-2025
https://doi.org/10.1007/s00371-024-03758-9
Show More Cited By

Index Terms

How do humans sketch objects?
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Graphics input devices

Recommendations

Hand-drawn face sketch recognition by humans and a PCA-based algorithm for forensic applications
Special issue on recent advances in biometrics

Because face sketches represent the original faces in a very concise yet recognizable form, they play an important role in criminal investigations, human visual perception, and face biometrics. In this paper, we compared the performances of humans and a ...
Sketch based modeling system
ICVR'07: Proceedings of the 2nd international conference on Virtual reality

In the design process of the external form of a product, sketching is applied for creating and embodying idea. Sketches are able to reflect the aesthetic sense of designers the most easily, making them the most efficient design tool. Sketches can be ...
Sketch Learning Environment for Human Body Figure by Imitative Drawing
KES '09: Proceedings of the 13th International Conference on Knowledge-Based and Intelligent Information and Engineering Systems: Part II

We developed an interactive learning environment for imitative figure sketching. Figure sketching is more difficult for novice than other sketching. People can easily find errors of figure sketch, since human is sensitive to human body figure. There are ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 31, Issue 4

July 2012

935 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2185520

Issue’s Table of Contents

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2012

Published in TOG Volume 31, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

587
Total Citations
View Citations
2,512
Total Downloads

Downloads (Last 12 months)299
Downloads (Last 6 weeks)25

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang RLi DHu CZhang H(2025)SKVOS: Sketch-Based Video Object Segmentation with a Large-Scale BenchmarkApplied Sciences10.3390/app1504175115:4(1751)Online publication date: 9-Feb-2025
https://doi.org/10.3390/app15041751
Liang SDai WCheng CWei YCai Y(2025)Rethinking sketch-based 3D shape retrieval: A simple baseline and benchmark reconstructionNeurocomputing10.1016/j.neucom.2024.128990618(128990)Online publication date: Feb-2025
https://doi.org/10.1016/j.neucom.2024.128990
Chen YLiu YKayar-Ceylan G(2025)CSG-based ML-supported 3D translation of sketches into game assets for game designersThe Visual Computer10.1007/s00371-024-03758-9Online publication date: 5-Jan-2025
https://doi.org/10.1007/s00371-024-03758-9
Li DJabbireddy SZhang YMetzler CVarshney A(2024)Instant-SFH: Non-Iterative Sparse Fourier Holograms Using Perlin NoiseSensors10.3390/s2422735824:22(7358)Online publication date: 18-Nov-2024
https://doi.org/10.3390/s24227358
Rishwinder SRadhika Rakesh K(2024)Artificial intelligence application in education: A systematic studyi-manager's Journal of Educational Technology10.26634/jet.21.1.2053221:1(56)Online publication date: 2024
https://doi.org/10.26634/jet.21.1.20532
Chou TWang Y(2024)A Study on Color Theme Generation Using Convolutional Neural NetworksProceedings of the 2024 The 6th World Symposium on Software Engineering (WSSE)10.1145/3698062.3698103(273-278)Online publication date: 13-Sep-2024
https://dl.acm.org/doi/10.1145/3698062.3698103
Liu PRoot AXu ALi YKjolstad FBik A(2024)Compiler Support for Sparse Tensor ConvolutionsProceedings of the ACM on Programming Languages10.1145/36897218:OOPSLA2(275-303)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3689721
Li XYu JLi ZLu HYuan RCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Dr. CLIP: CLIP-Driven Universal Framework for Zero-Shot Sketch Image RetrievalProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680702(9554-9562)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680702
Choi CLee JPark JKim Y(2024)3Doodle: Compact Abstraction of Objects with 3D StrokesACM Transactions on Graphics10.1145/365815643:4(1-13)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658156
Mo HGao CWang R(2024)Joint Stroke Tracing and Correspondence for 2D AnimationACM Transactions on Graphics10.1145/364989043:3(1-17)Online publication date: 9-Apr-2024
https://dl.acm.org/doi/10.1145/3649890
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents