Abstract
We introduce Blended Interaction, a new conceptual framework that helps to explain when users perceive user interfaces as “natural” or not. Based on recent findings from embodied cognition and cognitive linguistics, Blended Interaction provides a novel and more accurate description of the nature of human–computer interaction (HCI). In particular, it introduces the notion of conceptual blends to explain how users rely on familiar and real-world concepts whenever they learn to use new digital technologies. We apply Blended Interaction in the context of post-“Windows Icons Menu Pointer” interactive spaces. These spaces are ubiquitous computing environments for computer-supported collaboration of multiple users in a physical space or room, e.g., meeting rooms, design studios, or libraries, augmented with novel interactive technologies and digital computation, e.g., multi-touch walls, tabletops, and tablets. Ideally, in these spaces, the virtues of the familiar physical and social world are combined with that of the digital realm in a considered manner so that desired properties of each are preserved and a seemingly “natural” HCI is achieved. To support designers in this goal, we explain how the users’ conceptual systems use blends to tie together familiar concepts with the novel powers of digital computation. Furthermore, we introduce four domains of design to structure the underlying problem and design space: individual and social interaction, workflow, and physical environment. We introduce our framework by discussing related work, e.g., metaphors, mental models, direct manipulation, image schemas, reality-based interaction, and illustrate Blended Interaction using design decisions we made in recent projects.
Similar content being viewed by others
Notes
Post-WIMP = Post-“Windows Icons Menus Pointer”.
Blackwell provides an extensive review on the role and history of user interface metaphors as a design tool in HCI in [13].
Please note that, unlike HCI literature, Fauconnier and Turner do not refer to the desktop UI as the “desktop metaphor”. In their terminology, the desktop UI is a blend and not a metaphor, as is also discussed by Imaz and Benyon: “When we speak of the desktop metaphor now, we are really referring to a large blend […]. It is usually considered to be a metaphor because most of the traditional functionalities of ordinary work have been maintained as expressions in interface tasks […]. But when observed in detail, it is evident that we are dealing with a blend rather than a metaphor—the blend being based on the metaphor” [5: p. 52].
Blended Interactions user experience design studio at Rochester Institute of Technology. http://blendedinteractions.com/about/ (Last accessed Jul 19, 2013).
References
Ju W, Leifer L (2008) The design of implicit interactions: making interactive systems less obnoxious. Des Issues 24(3):72–84. doi:10.1162/desi.2008.24.3.72
Oulasvirta A (2008) When users “do” the Ubicomp. Interactions 15:6. doi:10.1145/1340961.1340963
Weiser M (1991) The computer for the 21st century. Sci Am 3:94–104. doi:10.1145/329124.329126
Whittaker S, Terveen L, Nardi BA (2000) A reference task agenda for HCI. In: Carroll JM (ed) Human–computer interaction in the new Millenium. Addison-Wesley, New York, pp 167–190
Byrne MD (2003) Cognitive architecture. In: Julie AJ, Andrew S (eds) The human–computer interaction handbook. L. Erlbaum Associates Inc., Hillsdale, NJ, pp 97–117
Kieras D (2003) Model-based evaluation. In: Julie AJ, Andrew S (eds) The human–computer interaction handbook. L. Erlbaum Associates Inc., Hillsdale, NJ, pp 1139–1151
Blackwell AF (1998) Metaphor in Diagrams. PhD Thesis, University of Cambridge, Cambridge
Jetter H-C, Leifert S, Gerken J, Schubert S, Reiterer H (2012) Does (multi-)touch aid users’ spatial memory and navigation in ‘panning’ and in ‘zooming and panning’ UIs? In: Proceedings of International Working Conference on Advanced Visual Interfaces (AVI’12). ACM, New York, NY, pp 83–90
Hutchins EL, Hollan JD, Norman DA (1985) Direct manipulation interfaces. Hum Comput Interact 1:311–338
Ravasio P, Tscherter V (2007) User’s theories of the desktop metaphor or why we should seek metaphor-free interfaces. In: Kaptelinin V, Czerwinski M (eds) Beyond the desktop metaphor: designing integrated digital work environments. MIT Press, Cambridge, MA, pp 265–294
Constantine L, Lockwood L (1999) Software for use: a practical guide to the models and methods of usage-centered design. Addison Wesley, Reading, MA
Imaz M, Benyon D (2007) Designing with Blends: conceptual Foundations of Human-Computer Interaction and Software Engineering. MIT Press. doi:http://mitpress.mit.edu/books/designing-blends
Gibbs RW (2006) Embodiment and cognitive science. Cambridge University Press, Cambridge. doi:10.2277/0521811740
Fauconnier G, Turner M (2002) The way we think: conceptual blending and the mind’s hidden complexities. EUA Basic Books, New York
Dourish P (2004) Where the action is: the foundations of embodied interaction. First MIT Press Paperback Edition edn. MIT Press
Lakoff G, Johnson M (1999) Philosophy in the flesh: the embodied mind and its challenge to Western thought. Basic Books, New York
Lakoff G, Johnson M (1980) Metaphors we live by. University of Chicago Press, Chicago
Jacob RJK, Girouard A, Hirshfield LM, Horn MS, Shaer O, Solovey ET, Zigelbaum J (2008) Reality-based interaction: a framework for post-WIMP interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ‘08). ACM, New York, NY, pp 201–210
Hurtienne J, Israel JH (2007) Image schemas and their metaphorical extensions: intuitive patterns for tangible interaction. In: Proceedings of the 1st international conference on tangible and embedded interaction (TEI ‘07). ACM, New York, NY, pp 127–134
Hurtienne J, Israel JH, Weber K (2008) Cooking up real world business applications combining physicality, digitality, and image schemas. In: Proceedings of the 2nd international conference on Tangible and embedded interaction (TEI ‘08). ACM, New York, NY pp 239–246
Blackwell AF (2006) The reification of metaphor as a design tool. ACM Trans Comput–Hum Interact 13(4):490–530. doi:10.1145/1188816.1188820
Ba Nardi, Zarmer CL (1993) Beyond models and metaphors: visual formalisms in user interface design. J Vis Lang Comput 4:5–33. doi:10.1109/HICSS.1991.184010
Bederson BB, Hollan JD, Perlin K, Meyer J, Bacon D, Furnas GW (1996) Pad++: a zoomable graphical sketchpad for exploring alternate interface physics. J Vis Lang Comput 7:3–32
Jacob RJK, Girouard A, Hirshfield LM, Horn MS, Shaer O, Solovey ET, Zigelbaum J (2007) Reality-based interaction: unifying the new generation of interaction styles. In: CHI ‘07 extended abstracts on human factors in computing systems (CHI EA ‘07). ACM, New York, NY pp 2465–2470
Boden MA (1994) Précis of the creative mind: myths and mechanisms. Behav Brain Sci 17(03):519–531. doi:10.1017/S0140525X0003569X
Constantine L (1998) Use and misuse of metaphor. http://www.foruse.com/articles/metaphor.pdf. Accessed Feb 5, 2013
Norman DA (2002) The design of everyday things. Basic Books, New York
Andrews C, Endert A, North C (2010) Space to think: large high-resolution displays for sensemaking. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ‘10). ACM, New York, NY, pp 55–64
Kirsh D (1995) The intelligent use of space. Artif Intell 73(1–2):31–68. doi:10.1016/0004-3702(94)00017-u
Scott SD, Carpendale S, Inkpen KM (2004) Territoriality in collaborative tabletop workspaces. In: Proceedings of the 2004 ACM conference on computer-supported cooperative work (CSCW ‘04). ACM, New York, NY, pp 294–303
Jetter H-C, Gerken J, Zöllner M, Reiterer H, Milic-Frayling N (2011) Materializing the query with facet-streams: a hybrid surface for collaborative search on tabletops. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ‘11). ACM, New York, NY, pp 3013–3022
Gerken J, Jetter H-C, Schmidt T (2010) Can “touch” get annoying? In ACM international conference on interactive tabletops and surfaces (ITS ‘10). ACM, New York, NY, pp 257–258
Rosson MB, Carroll JM (2002) Usability engineering: scenario-based development of human–computer interaction. Morgan Kaufmann series in interactive technologies, 1st edn. Academic Press, San Fancisco
Geyer F, Budzinski J, Reiterer H (2012) IdeaVis: a hybrid workspace and interactive visualization for paper-based collaborative sketching sessions. In: Proceedings of the 7th Nordic conference on human–computer interaction: Making Sense Through Design (NordiCHI ‘12). ACM, New York, NY, pp 331–340
Collins D (1995) Designing object-oriented user interfaces. Benjamin Cummings, Redwood City, CA
Robinson M (1993) Design for unanticipated use. In: Proceedings of the third conference on European conference on computer-supported cooperative work (ECSCW’93). Kluwer Academic Publishers, Norwell, MA, pp 187–202
O’Keefe B, Slutsky B, Iuliucci N, Nalbandian A, Thanedar A, Mokey S, Mival O (2013) Mobile experiences for tourism: brick city tours. In: CHI ‘13 Extended abstracts on human factors in computing systems (CHI EA ‘13). ACM, New York, NY, pp 1413–1418
Streitz N, Tandler P, Müller-Tomfelde C, Konomi S (2001) Roomware: towards the next generation of human–computer interaction based on an integrated design of real and virtual worlds. In: Carroll JM (ed) Human–Computer Interaction in the New Millenium. Addison Wesley, Reading, MA, pp 551–576. doi:https://www.zotero.org/mikekuniavsky/items/itemKey/DC2Z7CWH
Geyer F, Pfeil U, Höchtl A, Budzinski J, Reiterer H (2011) Designing reality-based interfaces for creative group work. In: Proceedings of the 8th ACM conference on creativity and cognition (C&C ‘11). ACM, New York, NY, pp 165–174
Beyer H, Holtzblatt K (1998) Contextual design: defining customer-centered systems. Morgan Kaufman, San Francisco
Rogers Y (2012) HCI theory: classical, modern, and contemporary, vol Lecture #14. Synthesis Lectures on Human-Centered Informatics. Morgan and Claypool
Acknowledgments
The authors would like to thank Johannes Zagermann and Daniel Klinkhammer for providing some of the illustrations and figures for this article. Furthermore, the authors would like to thank all the organizers and participants of our DCIS 2012 workshop “Designing Collaborative Interactive Spaces” at AVI 2012Footnote 5 and our workshop “Blended Interaction: Envisioning Future Collaborative Interactive Spaces” at CHI 2013Footnote 6 for the valuable feedback on and productive discussion of Blended Interaction that has helped to further develop our framework.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Jetter, HC., Reiterer, H. & Geyer, F. Blended Interaction: understanding natural human–computer interaction in post-WIMP interactive spaces. Pers Ubiquit Comput 18, 1139–1158 (2014). https://doi.org/10.1007/s00779-013-0725-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00779-013-0725-4