Abstract
The experience of interacting with a robot has been shown to be very different in comparison to people’s interaction experience with other technologies and artifacts, and often has a strong social or emotional component—a difference that poses potential challenges related to the design and evaluation of HRI. In this paper we explore this difference, and its implications on evaluating HRI. We outline how this difference is due in part to the general complexity of robots’ overall context of interaction, related to their dynamic presence in the real world and their tendency to invoke a sense of agency.
We suggest that due to these differences HCI evaluation methods should be applied to HRI with care, and we present a survey of select HCI evaluation techniques from the perspective of the unique challenges of robots. We propose a view on social interaction with robots that we call the holistic interaction experience, and introduce a set of three perspectives for exploring social interaction with robots: visceral factors of interaction, social mechanics, and social structures. We demonstrate how our three perspectives can be used in practice, both as guidelines to discuss and categorize robot interaction, and as a component in the evaluation process. Further, we propose an original heuristic for brainstorming various possibilities of interaction experiences based on a concept we call the interaction experience map.
Similar content being viewed by others
References
Barrett HC, Todd PM, Miller GF et al (2005) Accurate judgments of intention from motion cues alone: a cross-cultural study. Evol Human Behav 26(4):313–331. doi:10.1016/j.evolhumbehav.2004.08.015
Bartneck C, Forlizzi J (2004) A design-centred framework for social human-robot interaction. In: IEEE international workshop on robot and human interactive communication, 2004. ROMAN’04, Kurashiki, Okayama, Japan, 20–22 September 2004. IEEE Comput Soc, Los Alamitos, pp 581–594. doi:10.1109/ROMAN.2004.1374827
Bartneck C, van der Hoek M, Mubin O et al (2007) “Daisy, Daisy, give me your answer do!”: switching off a robot. In: Proceedings of the 2nd ACM/IEEE conference on human-robot interaction, 2007. HRI’07, Washington, DC, USA, 10–12 March 2007. ACM, New York, pp 217–222. doi:10.1145/1228716.1228746
Bartneck C, Verbunt M, Mubin O et al (2007) To kill a mockingbird robot. In: Proceedings of the 2nd ACM/IEEE conference on human-robot interaction, 2007. HRI’07, Washington, DC, USA, 10–12 March 2007. ACM, New York, pp 81–87. doi:10.1145/1228716.1228728
Bates J (1994) The role of emotion in believable agents. Commun ACM 37(7):122–125. doi:10.1145/176789.176803
Bethel CL, Bringes C, Murphy RR (2009) Non-facial and non-verbal affective expression in appearance-constrained robots for use in victim management: robots to the rescue! In: Proceedings of the 4th ACM/IEEE conference on human-robot interaction, 2009. HRI’09, San Diego, California, USA, 11–13 March 2009. ACM, New York, pp 191–192. doi:10.1145/1514095.1514130
Beyer H, Holtzblatt K (1998) Contextual design: defining customer-centered systems. Morgan Kauffman, San Mateo
Boehner K, DePaula R, Dourish P et al (2007) How emotion is made and measured. Int J Hum-Comput Stud (IJHCS) 65(4):275–291. doi:10.1016/j.ijhcs.2006.11.016
Breazeal CL (2003) Emotion and sociable humanoid robots. Int J Hum-Comput Stud (IJHCS) 59(1–2):119–155. doi:10.1016/S1071-5819(03)00018-1
Breazeal CL (2003) Toward sociable robots. Robot Autonom Syst 42(3–4):167–175. doi:10.1016/S0921-8890(02)00373-1
Breazeal CL, Brooks AG, Gray J et al (2004) Tutelage and collaboration for humanoid robots. Int J Human Robot (IJHR) 1(2):315–348. doi:10.1142/S0219843604000150
Burgard W, Cremers AB, Fox D et al (1999) Experiences with an interactive museum tour-guide robot. Artif Intell 114(1–2):3–55. doi:10.1016/S0004-3702(99)00070-3
Crabtree A, Benford S, Greenhalgh C et al (2006) Supporting ethnographic studies of ubiquitous computing in the wild. In: Proceedings of the 6th conference on designing interactive systems, 2006. DIS’06, University Park, PA, US, 26–28 June 2006. ACM, New York, pp 60–69. doi:10.1145/1142405.1142417
Cramer H, Goddijn J, Wielinga B et al (2010) Effects of (in)accurate empathy and situational valence on attitudes towards robots. In: Proceedings of the 5th ACM/IEEE conference on human-robot interaction, 2010. HRI’10, Osaka, Japan, 2–5 March 2010. ACM, New York, pp 141–142. doi:10.1145/1734454.1734513
Cramer H, Kemper N, Amin A et al (2009) ‘Give me a hug:’ the effects of touch and autonomy on people’s responses to embodied social agents. Comput Animat Virtual Worlds 20(2–3):2–3. doi:10.1002/cav.317
Csikszentmihalyi M (1990) Flow: the psychology of optimal experience. Harper Collins, New York
Dautenhahn K (2002) Design spaces and niche spaces of believable social robots. In: IEEE international workshop on robot and human interactive communication, 2002. ROMAN’02, Berlin, Germany, 25–27 September 2002. IEEE Comput Soc, Los Alamitos, pp 192–197. doi:10.1109/ROMAN.2002.1045621
Dennett D (1987) The intentional stance. MIT Press, Cambridge
Desmet PMA (2005) Measuring emotions: development and application of an instrument to measure emotional responses to products. In: Blythe MA, Overbeeke K, Monk AF et al (eds) Funology: from usability to enjoyment. Kluwer Academic, Norwell
Dewey J (1980) Art as experience. Perigee Books, New York
Dix A, Finlay J, Abowd GD et al (1998) Human-computer interaction, 2nd edn. Prentice Hall, New York
Dourish P (2001) Where the action is: the foundation of embodied interaction. MIT Press, Cambridge
Drury JL, Scholtz J, Yanco HA (2003) Awareness in human-robot interactions. In: Proceedings of the IEEE international conference on systems, man and cybernetics, 2003. SMC’03, Washington, DC, USA, 5–8 October 2003, vol 1. IEEE Comput Soc, Los Alamitos, pp 912–918. doi:10.1109/ICSMC.2003.1243931
Eberts RE (1994) User interface design. Prentice Hall, New York
Fernaeus Y, Ljungblad S, Jacobsson M et al (2009) Where third wave HCI meets HRI: report from a workshop on user-centred design of robots. In: Adjunct proceedings of the ACM/IEEE international conference on human-robot interaction (late-breaking abstracts), 2009. HRI LBA’09, San Diego, California, USA, 11–13 March 2009. ACM, New York, pp 293–294. doi:10.1145/1514095.1514182
Forlizzi J (2007) How robotic products become social products: an ethnographic study of cleaning in the home. In: Proceedings of the 2nd ACM/IEEE conference on human-robot interaction, 2007. HRI’07, Washington, DC, USA, 10–12 March 2007. ACM, New York, pp 129–136. doi:10.1145/1228716.1228734
Forlizzi J, DiSalvo C (2006) Service robots in the domestic environment: a study of the roomba vacuum in the home. In: Proceedings of the 1st ACM SIGCHI/SIGART conference on human-robot interaction, 2006. HRI’06, Salt Lake City, USA, 2–4 March 2006. ACM, New York, pp 258–256. doi:10.1145/1121241.1121286
Friedman B, Kahn PHJr, Hagman J (2003) Hardware companions?—what online AIBO discussion forums reveal about the human-robotic relationship. In: ACM conference on human factors in computing sysems, 2003. CHI’03, Fort Lauderdale, USA, 5–10 April 2003. ACM, New York, pp 273–280. doi:10.1145/642611.642660
Fussell SR, Kiesler S, Setlock LD et al (2008) How people anthropomorphize robots. In: Proceedings of the 3rd ACM/IEEE conference on human-robot interaction, 2008. HRI’08, Amsterdam, The Netherlands, 12–15 March 2008. ACM, New York, pp 145–152. doi:10.1145/1349822.1349842
Garreau J (2007) Bots on the ground. Washington Post, WWW, http://www.washingtonpost.com/wp-dyn/content/article/2007/05/05/AR2007050501009_pf.html. Visited April 9th, 2008
Gaver B, Dunne T, Pacenti E (1999) Design: cultural probes. Interactions 6(1):21–29. doi:10.1145/291224.291235
Gockley R, Forlizzi J, Simmons R (2006) Interactions with a moody robot. In: Proceedings of the 1st ACM SIGCHI/SIGART conference on human-robot interaction, 2006. HRI’06, Salt Lake City, USA, 2–4 March 2006. ACM, New York, pp 186–193. doi:10.1145/1121241.1121274
Gockley R, Forlizzi J, Simmons R (2007) Natural person-following behavior for social robots. In: Proceedings of the 2nd ACM/IEEE conference on human-robot interaction, 2007. HRI’07, Washington, DC, USA, 10–12 March 2007. ACM, New York, pp 17–24. doi:10.1145/1228716.1228720
Greenberg S (2003) Working through task-centered system design. In: Diaper D, Stanton N (eds) The handbook of task analysis for human-computer interaction. Lawrence Erlbaum Associates, Inc., Mahwah
Groom V, Chen J, Johnson T et al (2010) Critic, compatriot, or chump?: Responses to robot blame attribution. In: Proceedings of the 5th ACM/IEEE conference on human-robot interaction, 2010. HRI’10, Osaka, Japan, 2–5 March 2010. ACM, New York, pp 211–218. doi:10.1145/1734454.1734545
Guo C, Sharlin E (2008) Exploring the use of tangible user interfaces for human-robot interaction: a comparative study. In: ACM conference on human factors in computing sysems, 2007. CHI’07, San Jose, California, USA, 28 April–3 May 2007. ACM, New York, pp 121–130. doi:10.1145/1357054.1357076
Harrison S, Dourish P (1996) Re-place-ing space: the roles of place and space in collaborative systems. In: Proceedings of the ACM conference on computer supported cooperative work, 1996. CSCW’96, Boston, US, 16–20 November 1996. ACM, New York. doi:10.1145/240080.240193
Heider F, Simmel M (1944) An experimental study of apparent behavior. Am J Psychol 57:243–259
Ho CC, MacDorman KF, Pramono ZADD (2008) Human emotion and the uncanny valley: a GLM, MDS, and isomap analysis of robot video ratings. In: Proceedings of the 3rd ACM/IEEE conference on human-robot interaction, 2008. HRI’08, Amsterdam, The Netherlands, 12–15 March 2008. ACM, New York, pp 169–176. doi:10.1145/1349822.1349845
Holz T, Dragone M, O’Hare MP (2009) Where robots and virtual agents meet: a survey of social interaction research across milgram’s reality-virtuality continuum. Int J Soc Robot 1(1):83–93. doi:10.1007/s12369-008-0002-2
Höök K (2005) User-centered design and evaluation of affective interfaces. In: From brows to trust. Lecture notes in computer science, vol 7. Springer, Berlin, New York, Heidelberg
Höök K, Sengers P, Andersson G (2003) Sense and sensibility: evaluation and interactive art. In: ACM conference on human factors in computing sysems, 2003. CHI’03, Fort Lauderdale, USA, 5–10 April 2003. ACM, New York, pp 241–248. doi:10.1145/642611.642654
Hornecker E, Buur J (2006) Getting a grip on tangible interaction: a framework on physical space and social interaction. In: ACM conference on human factors in computing sysems, 2006. CHI’06, Montréal, Quèbec, 22–28 April 2006. ACM, New York, pp 437–446. doi:10.1145/1124772.1124838
Hüttenrauch H, Eklundh KS (2002) Fetch-and-carry with CERO: observations from a long-term user study with a service robot. In: IEEE international workshop on robot and human interactive communication, 2002. ROMAN’02, Berlin, Germany, 25–27 September 2002. IEEE Comput Soc, Los Alamitos, pp 158–163. doi:10.1109/ROMAN.2002.1045615
Isbister K, Höök K, Sharp M et al (2006) The sensual evaluation instrument: developing an affective evaluation tool. In: ACM conference on human factors in computing sysems, 2006. CHI’06, Montréal, Quèbec, 22–28 April 2006. ACM, New York, pp 1163–1172. doi:10.1145/1124772.1124946
Kanda T, Sato R, Saiwaki N et al (2007) A two-month field trial in an elementary school for long-term human-robot interaction. IEEE Trans Robot 23(5):962–971. doi:10.1109/TRO.2007.904904
Kiesler S, Hinds P (2004) Introduction to this special issue on human-robot interaction. Human Comput Interact 19(1/2):1–8. doi:10.1109/TSMCA.2005.850577
Landsberger HA (1958) Hawthorne revisited. Ithaca Press, Ithaca
Lee H, Kim HJ, Kim C (2007) Autonomous behavior design for robotic appliances. In: Proceedings of the 2nd ACM/IEEE conference on human-robot interaction, 2007. HRI’07, Washington, DC, USA, 10–12 March 2007. ACM, New York, pp 201–208. doi:10.1145/1228716.1228744
Lee MK, Forlizzi J, Rybski PE et al (2009) The snackbot: documenting the design of a robot for long-term human-robot interaction. In: Proceedings of the 4th ACM/IEEE conference on human-robot interaction, 2009. HRI’09, San Diego, California, USA, 11–13 March 2009. ACM, New York, pp 7–14. doi:10.1145/1514095.1514100
Lockerd A, Breazeal CL (2004) Tutelage and socially guided robot learning. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, 2004. IROS’04, Sendai, Japan, 28 September–2 October 2004, vol 4. IEEE Comput Soc, Los Alamitos, pp 3475–3480. doi:10.1109/IROS.2004.1389954
Marti P, Pollini A, Rullo A et al (2005) Engaging with artificial pets. In: Proceedings of the annual conference of the European association of cognitive ergonomics, 2005. EACE’05, Chania, Greece, 29 September–1 October 2005. ACM, New York, pp 99–106
Michalowski MP, Sabanovic S, Kozima H (2007) A dancing robot for rhythmic social interaction. In: Proceedings of the 2nd ACM/IEEE conference on human-robot interaction, 2007. HRI’07, Washington, DC, USA, 10–12 March 2007. ACM, New York, pp 89–96. doi:10.1145/1228716.1228729
Mori M (1970) Bukimi no tani: the uncanny valley (in Japanese). Energy 7:33–35. English translation provided at CogSci’05 workshop: Toward social mechanisms of android science. Views of the Uncanny Valley. WWW, http://www.androidscience.com/theuncannyvalley/proceedings2005/uncannyvalley.html
Muhl C, Nagai T (2007) Does disturbance discourage people from communicating with a robot? In: IEEE international workshop on robot and human interactive communication, 2009. ROMAN’09, Toyama, Japan, 27 September–2 October 2009. IEEE Comput Soc, Los Alamitos, pp 1137–1142. doi:10.1109/ROMAN.2007.4415251
Mutlu B, Shiwa T, Kanda T et al (2009) Footing in human-robot conversations: how robots might shape participant roles using gaze cues. In: Proceedings of the 4th ACM/IEEE conference on human-robot interaction, 2009. HRI’09, San Diego, California, USA, 11–13 March 2009. ACM, New York, pp 61–68. doi:10.1109/ROMAN.2007.4415251
Nass C, Moon Y (2000) Machines and mindlessness: social responses to computers. J Soc Issues 56(1):81–103. doi:10.1111/0022-4537.00153
Norman DA (1988) The design of everyday things. Doubleday, New York
Norman DA (2004) Emotional design: why we love (or hate) everyday things. Basic Books, New York
Pacchierotti E, Christensen HI, Jensfelt P (2006) Design of an office guide robot for social interaction studies. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, 2006. IROS’06, Beijing, China, 9–15 October 2006. IEEE Comput Soc, Los Alamitos, pp 4965–4970. doi:10.1109/IROS.2006.282519
Picard RW (1999) Affective computing for HCI. In: Proceedings of the HCI international conference on human-computer interaction, 1999. HCI’99, Munich, Germany, 22–26 August 1999. Lawrence Erlbaum Associates, Inc., Mahwah, pp 829–833
Reeves B, Nass C (1996) The media equation: how people treat computers, television, and new media like real people and places. CSLI Publications, Center for the Study of Language and Information Leland Standford Junior University, Cambridge, UK, first paperback edition
Richer J, Drury JL (2003) A video game-based framework for analyzing human-robot interaction: characterizing interface design in real-time interactive multimedia applications. In: Proceedings of the 1st ACM SIGCHI/SIGART conference on human-robot interaction, 2006. HRI’06, Salt Lake City, USA, 2–4 March 2006. ACM, New York, pp 266–273. doi:10.1145/1121241.1121287
Sanders EBN (1992) Converging perspectives: product development research for the 1990s. Des Manage J 3(4):49–54
Sengers P, Gaver B (2006) Staying open to interpretation: engaging multiple meanings in design and evaluation. In: Proceedings of the 6th conference on designing interactive systems, 2006. DIS’06, University Park, PA, US, 26–28 June 2006. ACM, New York, pp 99–108. doi:10.1145/1142405.1142422
Sharp H, Rogers Y, Preece J (2007) Interaction design: beyond human-computer interaction, 2nd edn. Wiley, New York
Short E, Hart J, Vu M et al (2010) No fair!!: an interaction with a cheating robot. In: Proceedings of the 5th ACM/IEEE conference on human-robot interaction, 2010. HRI’10, Osaka, Japan, 2–5 March 2010. ACM, New York, pp 219–226. doi:10.1145/1734454.1734546
Sidner CL, Lee C, Morency LP et al (2006) The effect of head-nod recognition in human-robot conversation. In: Proceedings of the 1st ACM SIGCHI/SIGART conference on human-robot interaction, 2006. HRI’06, Salt Lake City, USA, 2–4 March 2006. ACM, New York, pp 290–296. doi:10.1145/1121241.1121291
Staudte M, Crocker MW (2009) Visual attention in spoken human-robot interaction. In: Proceedings of the 4th ACM/IEEE conference on human-robot interaction, 2009. HRI’09, San Diego, California, USA, 11–13 March 2009. ACM, New York, pp 77–84. doi:10.1145/1514095.1514111
Strauss A, Corbin J (1998) Basics of qualitative research: techniques and procedures for developing grounded theory. Sage, Thousand Oaks
Sung J, Grinter RE, Christensen HI (2009) “Pimp my Roomba”: designing for personalization. In: ACM conference on human factors in computing sysems, 2009. CHI’09, Boston, USA, 4–9 April 2009. ACM, New York, pp 193–196. doi:10.1145/1518701.1518732
Sung J, Grinter RE, Christensen HI et al (2008) Housewives or technophiles?: understanding domestic robot owners. In: Proceedings of the 3rd ACM/IEEE conference on human-robot interaction, 2008. HRI’08, Amsterdam, The Netherlands, 12–15 March 2008. ACM, New York, pp 129–136. doi:10.1145/1349822.1349840
Sung J, Guo L, Grinter RE et al (2007) “My Roomba is Rambo”: intimate home appliances. In: Proceedings of the international conference on ubiquitous computing, 2007. UBICOMP’07, Innbruck, Austria, 16–17 September 2007. Lecture notes in computer science, vol 4717. Springer, Berlin, New York, Heidelberg. doi:10.1007/978-3-540-74853-3_9
Takayama L, Groom V, Nass C (2009) I’m sorry, Dave: I’m afraid I won’t do that: social aspects of human-agent conflict. In: ACM conference on human factors in computing sysems, 2009. CHI’09, Boston, USA, 4–9 April 2009. ACM, New York, pp 2099–2108. doi:10.1145/1518701.1519021
Takayama L, Ju W, Nass C (2008) Beyond dirty, dangerous and dull: what everyday people think robots should do. In: Proceedings of the 3rd ACM/IEEE conference on human-robot interaction, 2008. HRI’08, Amsterdam, The Netherlands, 12–15 March 2008. ACM, New York, pp 25–32. doi:10.1145/1349822.1349827
Tanaka F, Movellan JR, Fortenberry B et al (2006) Daily HRI evaluation at a classroom environment: reports from dance interaction experiments. In: Proceedings of the 1st ACM SIGCHI/SIGART conference on human-robot interaction, 2006. HRI’06, Salt Lake City, USA, 2–4 March 2006. ACM, New York, pp 3–9. doi:10.1145/1121241.1121245
Tolmie P, Pycock J, Diggins T et al (2002) Unremarkable computing. In: ACM conference on human factors in computing sysems, 2002. CHI’02, Minneapolis, MN, USA, 20–25 April 2002. ACM, New York, pp 399–406. doi:10.1145/503376.503448
Voida A, Grinter RE, Ducheneaut N et al (2005) Listening in: practices surrounding iTunes music sharing. In: ACM conference on human factors in computing sysems, 2005. CHI’05, Portland, OR, USA, 2–7 April 2005. ACM, New York, pp 191–200. doi:10.1145/1054972.1054999
Yanco HA, Drury JL (2004) Classifying human-robot interaction: an updated taxonomy. In: Proceedings of the IEEE international conference on systems, man and cybernetics, 2004. SMC’04, The Hague, The Netherlands, 10–13 October 2004, vol 3. IEEE Comput Soc, Los Alamitos, pp 2841–2846. doi:10.1109/ICSMC.2004.1400763
Young JE, Hawkins R, Sharlin E et al (2009) Toward acceptable domestic robots: applying insights from social psychology. Int J Soc Robot 1(1):95–108. doi:10.1007/s12369-008-0006-y
Young JE, Xin M, Sharlin E (2007) Robot expressionism through cartooning. In: Proceedings of the 2nd ACM/IEEE conference on human-robot interaction, 2007. HRI’07, Washington, DC, USA, 10–12 March 2007. ACM, New York, pp 309–316. doi:10.1145/1228716.1228758
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Young, J.E., Sung, J., Voida, A. et al. Evaluating Human-Robot Interaction. Int J of Soc Robotics 3, 53–67 (2011). https://doi.org/10.1007/s12369-010-0081-8
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12369-010-0081-8