Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Pen-Based Interaction

  • Living reference work entry
  • First Online:
Handbook of Human Computer Interaction

Abstract

The use of pens in human-computer interaction has been investigated since Ivan Sutherland’s Sketchpad graphical communication system in the early 1960s. We provide an overview of the major developments in pen-based interaction over the last six decades and compare different hardware solutions and pen-tracking techniques. In addition to pen-based interaction with digital devices, we discuss more recent digital pen and paper solutions where pen- and paper-based interaction is augmented with digital information and services. We outline different interface and interaction styles and present various academic as well as commercial application domains where pen-based interaction has been successfully applied. Furthermore, we discuss several issues to be considered when designing pen-based interactions and conclude with an outlook of potential future challenges and directions for pen-based human-computer interaction.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

References

  • Aliakseyeu D, Nacenta MA, Subramanian S, Gutwin C (2006) Bubble Radar: efficient pen-based interaction. In: Proceedings of the working conference on advanced visual interfaces (AVI 2006), Venezia, pp 19–26. https://doi.org/10.1145/1133265.1133271

  • Annett MK (2014) The fundamental issues of pen-based interaction with tablet devices. PhD thesis, University of Alberta

    Google Scholar 

  • Berque D, Bonebright T, Whitesell M (2004) Using pen-based computers across the computer science curriculum. In: Proceedings of the 35th SIGCSE technical symposium on computer science education (SIGCSE 2004), Norfolk, pp 61–65. https://doi.org/10.1145/1028174.971324

  • Bi X, Moscovich T, Ramos G, Balakrishnan R, Hinckley K (2008) An exploration of pen rolling for pen-based interaction. In: Proceedings of the 21st annual ACM symposium on user interface software and technology (UIST 2008), pp 191–200. https://doi.org/10.1145/1449715.1449745

  • Bush V (1945) As we may think. Atlantic Mon 176(1):101–108. https://doi.org/10.1145/227181.227186

    MATH  Google Scholar 

  • Buxton W (1990) A three-state model of graphical input. In: Proceedings of the 3rd international conference on human-computer interaction (INTERACT 1990), Cambridge, pp 449–456

    Google Scholar 

  • Carr R, Shafer D (1991) The power of PenPoint. Addison Wesley, Boston, MA, USA

    MATH  Google Scholar 

  • Chee YM et al (2011) Ink Markup Language (InkML). W3C Recommendation. https://www.w3.org/TR/InkML

  • Cohen PR, Oviatt S (2017) Multimodal speech and pen interfaces. Association for Computing Machinery and Morgan & Claypool, pp 403–447. https://doi.org/10.1145/3015783.3015795

  • Davis MR, Ellis TO (1964) The RAND tablet: a man-machine graphical communication device. In: Proceedings of the fall joint computer conference (AFIPS 1964), San Francisco, pp 325–331. https://doi.org/10.1145/1464052.1464080

  • Dimond TL (1957) Devices for reading handwritten characters. In: Proceedings of the eastern joint computer conference: computers with deadlines to meet (RE-ACM-AIEE 1957), Washington, DC, pp 232–237. https://doi.org/10.1145/1457720.1457765

    Google Scholar 

  • Drey T, Gugenheimer J, Karlbauer J, Milo M, Rukzio E (2020) VRSketchIn: exploring the design space of pen and tablet interaction for 3D sketching in virtual reality. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2020). https://doi.org/10.1145/3313831.3376628

  • Everitt KM, Klemmer SR, Lee R, Landay JA (2003) Two worlds apart: bridging the gap between physical and virtual media for distributed design collaboration. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2003), Fort Lauderdale, pp 553–560. https://doi.org/10.1145/642611.642707

  • Fellion N, Pietrzak T, Girouard A (2017) FlexStylus: leveraging bend input for pen interaction. In: Proceedings of the ACM symposium on user interface software and technology (UIST 2017), Quebec City, pp 375–385. https://doi.org/10.1145/3126594.3126597

  • Forster B (2001) Writing to the future. Computerworld, Needham, MA, USA

    MATH  Google Scholar 

  • Goldberg HE (1914) Controller. US Patent 1,117,184

    Google Scholar 

  • Gray E (1888) Telautograph. US Patent 386,815

    Google Scholar 

  • Grossman T, Hinckley K, Baudisch P, Agrawala M, Balakrishnan R (2006) Hover widgets: using the tracking state to extend the capabilities of pen-operated devices. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2006), MontrĂ©al, pp 861–870. https://doi.org/10.1145/1124772.1124898

  • Hahne U, Schild J, Elstner S, Alexa M (2009) Multi-touch Focus+Context sketch-based interaction. In: Proceedings of the 6th Eurographics symposium on sketch-based interfaces and modeling (SBIM 2009), New Orleans, pp 77–83. https://doi.org/10.1145/1572741.1572755

  • Haller M, Brandl P, Leithinger D, Leitner J, Seifried T, Billinghurst M (2006) Shared design space: sketching ideas using digital pens and a large augmented tabletop setup. In: Proceedings of the 16th international conference on artificial reality and telexistence (ICAT 2006), Hangzhou, pp 185–196. https://doi.org/10.1007/11941354_20

  • Hammond T, Davis R (2005) LADDER, a sketching language for user interface developers. Comput Graph 29(4):518–532. https://doi.org/10.1145/1281500.1281546

    Article  MATH  Google Scholar 

  • Heinrichs FHF (2015) Mobile pen-and-paper interaction: infrastructure design, conceptual frameworks of interaction and interaction theory. PhD thesis, Technical University of Darmstadt

    Google Scholar 

  • Hinckley K, Zhao S, Sarin R, Baudisch P, Cutrell E, Shilman M, Tan D (2007) InkSeine: in situ search for active note taking. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2007), San Jose, pp 251–260. https://doi.org/10.1145/1240624.1240666

  • Hinckley K, Pahud M, Benko H, Irani P, Guimbretière F, Gavriliu M, Chen XA, Matulic F, Buxton W, Wilson A (2014) Sensing techniques for Tablet+Stylus interaction. In: Proceedings of the 27th annual ACM symposium on user interface software and technology (UIST 2014), Honolulu, pp 605–614. https://doi.org/10.1145/2642918.2647379

  • Hofer R, Kunz A (2010) Digisketch: taming anoto technology on LCDs. In: Proceedings of the ACM symposium on engineering interactive computing systems (EICS 2010), Berlin, pp 103–108. https://doi.org/10.1145/1822018.1822034

  • Hwang S, Bianchi A, Ahn M, Wohn K (2013) MagPen: magnetically driven pen interactions on and around conventional smartphones. In: Proceedings of the 15th international conference on human-computer interaction with mobile devices and services (MobileHCI 2013), Munich, pp 412–415. https://doi.org/10.1145/2493190.2493194

  • Ispas A (2011) Beyond the digital capture of paper notes: investigations of enhanced natural notetaking based on digital pen and paper technology. PhD thesis, ETH Zurich. https://doi.org/10.3929/ethz-a-007139493, dissertation ETH No. 20157

  • Kam M, Wang J, Iles A, Tse E, Chiu J, Glaser D, Tarshish O, Canny J (2005) Livenotes: a system for cooperative and augmented note-taking in lectures. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2005), Portland, pp 531–540. https://doi.org/10.1145/1054972.1055046

  • Kay A, Goldberg A (1977) Personal dynamic media. Computer 10(3):31–41

    Article  Google Scholar 

  • Klamka K, Dachselt R (2017) IllumiPaper: illuminated interactive paper. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2017), Denver, pp 5605–5618. https://doi.org/10.1145/3025453.3025525

  • Koike H, Sato Y, Kobayashi Y (2001) Integrating paper and digital information on EnhancedDesk: a method for realtime finger tracking on an augmented desk system. ACM Trans Comput-Hum Interact 8(4):307–322. https://doi.org/10.1145/504704.504706

    Article  Google Scholar 

  • Kurtenbach G, Buxton W (1994) User learning and performance with marking menus. In: Proceedings of the ACM conference on human factors in computing systems (CHI 1994), Boston, pp 258–264. https://doi.org/10.1145/191666.191759

  • Levine SR, Ehrlich SF (1991) The freestyle system. Springer, Boston, pp 3–21. https://doi.org/10.1007/978-1-4684-5883-1_1

    MATH  Google Scholar 

  • Li Z, Annett M, Hinckley K, Singh K, Wigdor D (2019) HoloDoc: enabling mixed reality workspaces that harness physical and digital content. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2019), Glasgow, pp 1–14. https://doi.org/10.1145/3290605.3300917

  • Liao C (2009) PapierCraft: a paper-based interface to support interaction with digital documents. PhD thesis, University of Maryland

    Google Scholar 

  • Luff P, Adams G, Bock W, Drazin A, Frohlich D, Heath C, Herdman P, King H, Linketscher N, Murphy R, Norrie MC, Sellen A, Signer B, Tallyn E, Zeller E (2007) Augmented paper: developing relationships between digital content and paper. The disappearing computer: interaction design, system infrastructures and applications for smart environments. LNCS 4500:275–297. https://doi.org/10.1007/978-3-540-72727-9

    Google Scholar 

  • Mackay WE, Pothier G, Letondal C, Bøegh K, Sørensen HE (2002) The missing link: augmenting biology laboratory notebooks. In: Proceedings of the 15th annual ACM symposium on user interface software and technology (UIST 2002), Paris, pp 41–50. https://doi.org/10.1145/571985.571992

  • Magrofuoco N, Roselli P, Vanderdonckt J (2021) Two-dimensional stroke gesture recognition: a survey. ACM Comput Surv 54(7). https://doi.org/10.1145/3465400

  • Maierhöfer V, Schmid A, Wimmer R (2024) TipTrack: precise, low-latency, robust optical pen tracking on arbitrary surfaces using an IR-emitting pen tip. In: Proceedings of the 18th international conference on tangible, embedded, and embodied interaction (TEI 2024), Cork. https://doi.org/10.1145/3623509.3633366

  • Matulic F, Norrie MC (2013) Pen and touch gestural environment for document editing on interactive tabletops. In: Proceedings of the ACM international conference on interactive tabletops and surfaces (ITS 2013), St. Andrews, pp 41–50. https://doi.org/10.1145/2512349.2512802

    MATH  Google Scholar 

  • Matulic F, Arakawa R, Vogel B, Vogel D (2020) PenSight: enhanced interaction with a pen-top camera. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2020), pp 1–14. https://doi.org/10.1145/3313831.3376147

  • Microsoft (2021) Application gestures and semantic behavior. https://learn.microsoft.com/en-us/windows/win32/tablet/application-gestures-and-semantic-behavior

  • Moran TP, Chiu P, van Melle W (1997) Pen-based interaction techniques for organizing material on an electronic whiteboard. In: Proceedings of the 10th annual ACM symposium on user interface software and technology (UIST 1997), Banff, pp 45–54. https://doi.org/10.1145/263407.263508

  • Mueller PA, Oppenheimer DM (2014) The pen is mightier than the keyboard: advantages of longhand over laptop note taking. Psychol Sci 25(6):1159–1168. https://doi.org/10.1177/0956797614524581

    Article  Google Scholar 

  • Norrie MC, Signer B, Grossniklaus M, Belotti R, Decurtins C, Weibel N (2007) Context-aware platform for mobile data management. Wirel Netw (WINET) 13(6):855–870. https://doi.org/10.1007/s11276-006-9858-y

    Article  Google Scholar 

  • Pfeuffer K, Hinckley K, Pahud M, Buxton B (2017) Thumb + Pen interaction on tablets. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2017), Denver, pp 3254–3266. https://doi.org/10.1145/3025453.3025567

  • Pham DM, Stuerzlinger W (2019) Is the pen mightier than the controller? A comparison of input devices for selection in virtual and augmented reality. In: Proceedings of the 25th ACM symposium on virtual reality software and technology (VRST 2019), Parramatta, pp 1–11. https://doi.org/10.1145/3359996.3364264

  • Ramos GA, Balakrishnan R (2007) Pressure marks. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2007), San Jose, pp 1375–1384. https://doi.org/10.1145/1240624.1240834

  • Rekimoto J (1998) A multiple device approach for supporting whiteboard-based interactions. In: Proceedings of the ACM conference on human factors in computing systems (CHI 1998), Los Angeles, pp 344–351. https://doi.org/10.1145/274644.274692

  • Riche Y, Henry Riche N, Hinckley K, Panabaker S, Fuelling S, Williams S (2017) As we may ink? Learning from everyday analog pen use to improve digital ink experiences. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2017), Denver, pp 3241–3253. https://doi.org/10.1145/3025453.3025716

  • Romat H, Riche NH, Hinckley K, Lee B, Appert C, Pietriga E, Collins C (2019) ActiveInk: (Th)Inking with data. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2019), Glasgow, pp 1–13. https://doi.org/10.1145/3290605.3300272

  • Schilit BN, Golovchinsky G, Price MN (1998) Beyond paper: supporting active reading with free form digital ink annotations. In: Proceedings of the ACM conference on human factors in computing systems (CHI 1998), Los Angeles, pp 249–256. https://doi.org/10.1145/274644.274680

  • Sellen AJ, Harper R (2001) The myth of the paperless office. MIT Press, Cambridge, MA, USA

    Book  MATH  Google Scholar 

  • Signer B (2005) Fundamental concepts for interactive paper and cross-media information spaces. PhD thesis, ETH Zurich. https://doi.org/10.3929/ethz-a-005174378, dissertation ETH No. 16218

  • Signer B (2019) Towards cross-media information spaces and architectures. In: Proceedings of the 13th international conference on research challenges in information science (RCIS 2019), Brussels, pp 1–7. https://doi.org/10.1109/RCIS.2019.8877105

  • Signer B, Norrie MC (2007) PaperPoint: a paper-based presentation and interactive paper prototyping tool. In: Proceedings of the 1st international conference on tangible and embedded interaction (TEI 2007), Baton Rouge, pp 57–64. https://doi.org/10.1145/1226969.1226981

  • Signer B, Norrie MC (2010) Interactive paper: past, present and future. In: Proceedings of the 1st international workshop on paper computing (PaperComp 2010), Copenhagen

    Google Scholar 

  • Signer B, Kurmann U, Norrie MC (2007) iGesture: a general gesture recognition framework. In: Proceedings of the 9th international conference on document analysis and recognition (ICDAR 2007), Curitiba, pp 954–958. https://doi.org/10.1109/ICDAR.2007.4377056

  • Song H, Grossman T, Fitzmaurice G, Guimbretière F, Khan A, Attar R, Kurtenbach G (2009) PenLight: combining a mobile projector and a digital pen for dynamic visual overlay. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2009), Boston, pp 143–152. https://doi.org/10.1145/1518701.1518726

  • Steimle J (2012) Pen-and-paper user interfaces: integrating printed and digital documents. Springer, Heidelberg, Germany

    Book  MATH  Google Scholar 

  • Stifelman LJ, Arons B, Schmandt C (2001) The audio notebook: paper and pen interaction with structured speech. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2001), Seattle, pp 182–189. https://doi.org/10.1145/365024.365096

  • Subramonyam H, Seifert C, Shah P, Adar E (2020) texSketch: active diagramming through pen-and-ink annotations. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2020), Honolulu, pp 1–13. https://doi.org/10.1145/3313831.3376155

  • Sutherland IE (1963) Sketchpad: a man-machine graphical communication system. PhD thesis, Massachusetts Institute of Technology

    Google Scholar 

  • Tabard A (2009) Supporting lightweight reflection on familiar information. PhD thesis, UniversitĂ© Paris-Sud

    Google Scholar 

  • Vogelsang A, Signer B (2005) The lost cosmonaut: an interactive narrative environment on basis of digitally enhanced paper. In: Proceedings of the international conference on virtual storytelling 2005, Strasbourg, pp 270–279. https://doi.org/10.1007/11590361_31

  • Walny J, Lee B, Johns P, Riche N, Carpendale S (2012) Understanding pen and touch interaction for data exploration on interactive whiteboards. IEEE Trans Vis Comput Graph 18(12):2779–2788. https://doi.org/10.1145/2642918.2647379

    Article  Google Scholar 

  • Van der Weel F, Van der Meer A (2024) Handwriting but not typewriting leads to widespread brain connectivity: a high-density EEG study with implications for the classroom. Front Psychol 14. https://doi.org/10.3389/fpsyg.2023.1219945

  • Weibel N (2009) A publishing infrastructure for interactive paper documents: supporting interactions across the paper-digital divide. PhD thesis, ETH Zurich. https://doi.org/10.3929/ethz-a-005886877, dissertation ETH No. 18514

  • Weibel N, Norrie MC, Signer B (2007) A model for mapping between printed and digital document instances. In: Proceedings of the ACM symposium on document engineering (DocEng 2007), Winnipeg, pp 19–28. https://doi.org/10.1145/1284420.1284428

  • Weibel N, Signer B, Norrie MC, Hofstetter H, Jetter HC, Reiterer H (2011) PaperSketch: a paper-digital collaborative remote sketching tool. In: Proceedings of the international conference on intelligent user interfaces (IUI 2011), Palo Alto, pp 155–164. https://doi.org/10.1145/1943403.1943428

  • Weiser M (1991) The computer for the 21st century. Sci Am 265(3):3–11

    Article  MATH  Google Scholar 

  • Wellner P (1993) Interacting with paper on the DigitalDesk. Commun ACM 36(7):87–96. https://doi.org/10.1145/159544.159630

    Article  MATH  Google Scholar 

  • Xin Y, Bi X, Ren X (2011) Acquiring and pointing: an empirical study of pen-tilt-based interaction. In: Proceedings of the ACM conference on human factors in computing systems (CHI 2011), Vancouver, pp 849–858. https://doi.org/10.1145/1978942.1979066

  • Yeh RB (2007) Designing interactions that combine pen, paper, and computer. PhD thesis, Stanford University

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Web & Information Systems Engineering (WISE) Lab, Vrije Universiteit Brussel, Brussels, Belgium

    Beat Signer

Authors
  1. Beat Signer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Beat Signer .

Editor information

Editors and Affiliations

  1. Universite catholique de Louvain, Louvain-la-Neuve, Belgium

    Jean Vanderdonckt

  2. IRIT - Interactive Critical Sys Group, Paul Sabatier University, Toulouse, France

    Philippe Palanque

  3. I3S, INRIA wimmics/SPARKS team, Université Nice Sophia Antipolis, Sophia Antipolis Cedex, France

    Marco Winckler

Rights and permissions

Reprints and permissions

Copyright information

© 2025 Springer Nature Switzerland AG

About this entry

Check for updates. Verify currency and authenticity via CrossMark

Cite this entry

Signer, B. (2025). Pen-Based Interaction. In: Vanderdonckt, J., Palanque, P., Winckler, M. (eds) Handbook of Human Computer Interaction. Springer, Cham. https://doi.org/10.1007/978-3-319-27648-9_102-1

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-27648-9_102-1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-27648-9

  • Online ISBN: 978-3-319-27648-9

  • eBook Packages: Living Reference Computer SciencesReference Module Computer Science and Engineering

Publish with us

Policies and ethics

Search

Navigation