research-article

Do You Feel the Same? On the Robustness of Cued-Recall Debriefing for User Experience Evaluation

Authors:

Effie Lai-Chong Law,

Thomas Dyhre Nielsen,

Matthias HeintzAuthors Info & Claims

ACM Transactions on Computer-Human Interaction (TOCHI), Volume 28, Issue 4

Article No.: 25, Pages 1 - 45

https://doi.org/10.1145/3453479

Published: 23 July 2021 Publication History

Abstract

Cued Recall Debriefing (CRD) is a form of retrospective think aloud approach. It involves re-immersing users to a level where emotional responses are comparable to those experienced during actual interaction with a system. To validate whether the robustness of CRD would vary with the time gap between the actual and recalled event and with the affective state preceding the recall, two empirical studies with altogether 100 participants were conducted. Specifically, participants’ emotions were measured in terms of galvanic skin response (GSR), heart rate (HR), and self-assessment manikin (SAM) rating when they were interacting with an email client seeded with usability problems. The same measures were taken when they viewed the videoed interactions. Two between-subject variables were ‘intervening time’ (from 0 minutes up to 24 hours) and ‘intervening affect’ (images with different valence and arousal). Advanced computational models were applied to optimise the shifting of GSR/HR waves generated at the actual interaction and recall phases, which were found to be significantly correlated. The shifting process is necessary for addressing the memory effect and is a methodological innovation. Overall, CRD proved to be a robust method that can be deployed to a broad range of HCI research and practice contexts.

References

[1]

Marco Albanese, Michaelis Neofytou, Taoufik Ouarrak, Steffen Schneider, and Wolfgang Schöls. 2016. Evaluation of heart rate measurements in clinical studies: A prospective cohort study in patients with heart disease. European Journal of Clinical Pharmacology 72, 7 (2016), 789–795.

[2]

Obead Alhadreti and Pam Mayhew. 2018. Rethinking thinking aloud: A comparison of three think-aloud protocols. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI’18). 44:1–44:12.

Digital Library

[3]

John R. Anderson and Gordon H. Bower. 1979. Human associative memory. Lawrence Erlbaum Associates, Hillsdale, NJ.

[4]

Mattias Arvola, Johan Blomkvist, and Fredrik Wahlman. 2017. Lifelogging in user experience research: Supporting recall and improving data richness. The Design Journal 20, sup1 (2017), S3954–S3965.

[5]

Pedro Avero and Manuel G. Calvo. 2006. Affective priming with pictures of emotional scenes: The role of perceptual similarity and category relatedness. Spanish Journal of Psychology 9, 1 (2006), 10–18.

[6]

Javier A. Bargas-Avila and Kasper Hornbæk. 2011. Old wine in new bottles or novel challenges? A critical analysis of empirical studies of user experience. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2689–2698.

Digital Library

[7]

R. F. Baumeister, E. Bratslavsky, C. Finkenauer, and K. D. Vohs. 2001. Bad is stronger than good. Review of General Psychology 5, 4 (2001), 323–370.

[8]

V. Benet-Martinez and O. P. John. 1998. Los cinco grandes across cultures and ethnic groups: Multitrait multimethod analyses of the big five in Spanish and English. Journal of Personality and Social Psychology 75, 3 (1998), 729–750.

[9]

Todd Bentley, Lorraine Johnston, and Karola von Baggo. 2005. Evaluation using cued-recall debrief to elicit information about a user's affective experiences. In Proceedings of the 2005 Australasian Computer-Human Interaction Conference (OZCHI’05). 1–10. Retrieved from http://dl.acm.org/citation.cfm?id=1108368.1108403.

[10]

David. Benyon. 2019. Designing Interactive Systems : A Comprehensive Guide to HCI, UX and Interaction Design. Pearson. Retrieved 2019 from https://catalogue.pearsoned.co.uk/educator/product/Designing-Interactive-Systems-A-comprehensive-guide-to-HCI-UX-and-interaction-design-3E/9781447920113.page.

[11]

Emma Berry, Narinder Kapur, Lyndsay Williams, Steve Hodges, Peter Watson, Gavin Smyth, James Srinivasan, Reg Smith, Barbara Wilson, and Ken Wood. 2007. The use of a wearable camera, SenseCam, as a pictorial diary to improve autobiographical memory in a patient with limbic encephalitis: A preliminary report. Neuropsychological Rehabilitation 17, 4–5 (2007), 582–601.

[12]

Anol Bhattacherjee. 2001. Understanding information systems continuance: An expectation-confirmation model. MIS Quarterly: Management Information Systems 25, 3 (2001), 351–370.

Digital Library

[13]

Kirsten Boehner, Rogério DePaula, Paul Dourish, and Phoebe Sengers. 2007. How emotion is made and measured. International Journal of Human-Computer Studies 65, 4 (2007), 275–291.

Digital Library

[14]

T. Boren and J. Ramey. 2000. Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication 43, 3 (2000), 261–278.

[15]

Gordon H. Bower and Joseph P. Forgas. 2000. Affect, memory, and social cognition. In Cognition and Emotion. Oxford University Press, 87–168.

[16]

V. A. Bowers and H. L. Snyder. 1990. Concurrent versus retrospective verbal protocols for comparing window usability. In Proceedings of the Human Factors Society 34th Meeting. 1270–1274.

[17]

M. Bradley. 2015. Media Core. Retrieved August 1, 2016 from http://csea.phhp.ufl.edu/media.html.

[18]

Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry 25, 1 (1994), 49–59.

[19]

Jason J. Braithwaite and Derrick G. Watson. 2015. Issues Surrounding the Normalization and Standardisation of Skin Conductance Responses (SCRs). Birmingham.

[20]

Jennifer L. Branch. 2000. Investigating the information-seeking processes of adolescents: The value of using think alouds and think afters. Library & Information Science Research 22, 4 (2000), 371–392.

[21]

Casey L. Brown, Natalia Van Doren, Brett Q. Ford, Iris B. Mauss, Jocelyn W. Sze, and Robert W. Levenson. 2020. Coherence between subjective experience and physiology in emotion: Individual differences and implications for well-being. Emotion 20, 5 (2020), 818–829.

[22]

A. Bruun, E. L.-C. Law, M. Heintz, and P. S. Eriksen. 2016. Asserting real-time emotions through cued-recall: Is it valid? In Proceedings of the 9th Nordic Conference on Human-Computer Interaction.

Digital Library

[23]

Anders Bruun. 2018. It's not complicated: A study of non-specialists analyzing GSR sensor data to detect UX related events. In Proceedings of the 10th Nordic Conference on Human-Computer Interaction (NordiCHI’18). 170–183.

Digital Library

[24]

Anders Bruun and Simon Ahm. 2015. Mind the gap! Comparing retrospective and concurrent ratings of emotion in user experience evaluation. In Proceedings of the 15th IFIP TC13 Conference on Human-Computer Interaction (INTERACT’15). 237–254.

Digital Library

[25]

Anders Bruun, E. L.-C. Effie Lai-Chong Law, Matthias Heintz, and Lana H. A. Alkly. 2016. Understanding the relationship between frustration and the severity of usability problems: What can psychophysiological data (not) tell us? In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI’16). 3975–3987.

Digital Library

[26]

Anders Bruun and Martin Lynge Stentoft. 2019. Lifelogging in the wild: Participant experiences of using lifelogging as a research tool. In Proceedings of the 17th IFIP TC13 Conference on Human-Computer Interaction (INTERACT’19). 431–451.

Digital Library

[27]

Tony W. Buchanan. 2007. Retrieval of emotional memories. Psychological Bulletin 133, 5 (2007), 761–779.

[28]

Teah-Marie Bynion and Matthew T. Feldner. 2017. Self-assessment manikin. In Encyclopedia of Personality and Individual Differences. Virgil Zeigler-Hill and Todd K. Shackelford (Eds.), Springer International Publishing, Cham, 1–3.

[29]

John T. Cacioppo, Louis G. Tassinary, and Gary G. Berntson. 2009. Psychophysiological science: Interdisciplinary approaches to classic questions about the mind. In Handbook of Psychophysiology. Cambridge University Press, 1–16.

[30]

Kwang Ho Choi, Junbeom Kim, O. Sang Kwon, Min Ji Kim, Yeon Hee Ryu, and Ji Eun Park. 2017. Is heart rate variability (HRV) an adequate tool for evaluating human emotions?—A focus on the use of the international affective picture system (IAPS). Psychiatry Research 251 (2017), 192–196.

[31]

M. Csikszentmihalyi and R. Larson. 1987. Validity and reliability of the experience-sampling method. The Journal of Nervous and Mental Disease 175, 9 (1987), 526–536.

[32]

Elise S. Dan-Glauser and Klaus R. Scherer. 2011. The Geneva affective picture database (GAPED): A new 730-picture database focusing on valence and normative significance. Behavior Research Methods 43, 2 (2011), 468–477.

[33]

Michael E. Dawson, Anne M. Schell, Diane L. Filion, and Gary G. Berntson. 2009. The electrodermal system. In Handbook of Psychophysiology. Cambridge University Press, 157–181.

[34]

Nicola Eger, Linden J. Ball, Robert Stevens, and Jon Dodd. 2007. Cueing retrospective verbal reports in usability testing through eye-movement replay. In Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI...But Not As We Know It—Volume 1 (BCS-HCI’07). 129–137. Retrieved from http://dl.acm.org/citation.cfm?id=1531294.1531312

Digital Library

[35]

Panteleimon Ekkekakis. 2013. The Measurement of Affect, Mood, and Emotion A Guide for Health-Behavioral Research. Cambridge University Press.

[36]

Paul Ekman. 1992. An argument for basic emotions. Cognition and Emotion 6, 3–4 (1992), 169–200.

[37]

Phoebe C. Ellsworth. 2013. Appraisal theory: Old and new questions. Emotion Review 5, 2 (2013), 125–131.

[38]

Phoebe C. Ellsworth. 1994. Levels of thought and levels of emotion. In The Nature of Emotion: Fundamental Questions. Paul Ekman and R. J. Davidson (Eds.), Oxford University Press, New York, NY, 192–196.

[39]

K. Anders Ericsson and Herbert A. Simon. 1980. Verbal reports as data. Psychological Review 87, 3 (1980), 215–251.

[40]

Karl Anders Ericsson and Herbert Alexander Simon. 1993. Protocol Analysis: Verbal Reports as Data, Revised Edition. MIT Press, Cambridge, MA.

[41]

Johnny R. J. Fontaine, Klaus R. Scherer, Etienne B. Roesch, and Phoebe C. Ellsworth. 2007. The world of emotions is not two-dimensional. Psychological Science 18, 12 (2007), 1050–1057.

[42]

Jodi Forlizzi and Katja Battarbee. 2004. Understanding experience in interactive systems. In Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques (DIS’04). 261–268.

Digital Library

[43]

Barbara L. Fredrickson and Christine Branigan. 2005. Positive emotions broaden the scope of attention and thought-action repertoires. Cognition & Emotion 19, 3 (2005), 313–332.

[44]

Eva Ganglbauer, Stephanie Deutsch, and Manfred Tscheligi. 2009. Applying psychophysiological methods for measuring user experience: Possibilities, challenges and feasibility. In Proceedings of the User Experience Evaluation Methods in Product Development (UXEM’09).

[45]

Tamara van Gog, Fred Paas, Jeroen J. G. van Merriënboer, Puk Witte, Jeroen J. G. van Merrienboer, and Puk Witte. 2005. Uncovering the problem-solving process: Cued retrospective reporting versus concurrent and retrospective reporting. Journal of Experimental Psychology. Applied 11, 4 (2005), 237–244.

[46]

Zhiwei Guan, Shirley Lee, Elisabeth Cuddihy, and Judith Ramey. 2006. The validity of the stimulated retrospective think-aloud method as measured by eye tracking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’06). 1253.

Digital Library

[47]

Maaike J. Van Den Haak, Menno De Jong, and Peter Jan Schellens. 2003. Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology 22, 5 (2003), 339–351.

[48]

John Paulin Hansen. 1991. The use of eye mark recordings to support verbal retrospection in software testing. Acta Psychologica 76, 1 (1991), 31–49.

[49]

Marc Hassenzahl and Daniel Ullrich. 2007. To do or not to do: Differences in user experience and retrospective judgments depending on the presence or absence of instrumental goals. Interacting with Computers 19, 4 (2007), 429–437.

Digital Library

[50]

Dirk Hermans, Adriaan Spruyt, Jan De Houwer, and Paul Eelen. 2003. Affective priming with subliminally presented pictures. Canadian Journal of Experimental Psychology 57, 2 (2003), 97–114.

[51]

Morten Hertzum, Pia Borlund, and Kristina B. Kristoffersen. 2015. What do thinking-aloud participants say? A comparison of moderated and unmoderated usability sessions. International Journal of Human-Computer Interaction 31, 9 (2015), 557–570.

[52]

Morten Hertzum, Kristin D. Hansen, and Hans H. K. Andersen. 2009. Scrutinising usability evaluation: Does thinking aloud affect behaviour and mental workload? Behaviour & Information Technology 28, 2 (2009), 165–181.

Digital Library

[53]

Steve Hodges, Lyndsay Williams, Emma Berry, Shahram Izadi, James Srinivasan, Alex Butler, Gavin Smyth, Narinder Kapur, and Ken Wood. 2006. SenseCam: A retrospective memory aid. In Proceedings of the 8th International Conference of Ubiquitous Computing (UbiComp’06). 177–193.

Digital Library

[54]

Aulikki Hyrskykari, Saila Ovaska, Pävi Majaranta, Kari-Jouko Räihä, and Merja Lehtinen. 2008. Gaze path stimulation in retrospective think-aloud. Journal of Eye Movement Research 2, 4 (2008).

[55]

Caroll E. Izard. 1993. Organizational and motivational functions of discrete emotions. In Handbook of emotions. Guilford Press, New York, NY, 631–641.

[56]

Oliver P. John, Laura P. Naumann, and Christopher J. Soto. 2008. Paradigm shift to the integrative big five trait taxonomy: History, measurement, and conceptual issues. In Handbook of Personality: Theory and Research. Oliver P. John, Richard W. Robbins, and Lawrence A. Pervin (Eds.), Guilford, New York, 114–156. Retrieved from http://www.ocf.berkeley.edu/∼johnlab/bigfive.htm.

[57]

Daniel Kahneman, Barbara L. Fredrickson, Charles A. Schreiber, and Donald A. Redelmeier. 1993. When more pain is preferred to less: Adding a better end. Psychological Science 4, 6 (1993), 401–405.

[58]

Daniel Kahneman, Alan B. Krueger, David A. Schkade, Norbert Schwarz, and Arthur A. Stone. 2004. A survey method for characterizing daily life experience: The day reconstruction method. Science 306 (5702), 1776–1780. Retrieved from http://science.sciencemag.org/content/306/5702/1776.abstract.

[59]

Liadh Kelly and Gareth J. F. Jones. 2010. An exploration of the utility of GSR in locating events from personal lifelogs for reflection. In Proceedings of the 4th Irish Human Computer Interaction Conference (iHCI’10). Retrieved September 21, 2018 from https://www.semanticscholar.org/paper/An-Exploration-of-the-Utility-of-GSR-in-Locating-Kelly/12e73eb02d357ee0085465d0cff6aa322e19e515.

[60]

Jacqueline Kerr, Simon J. Marshall, Suneeta Godbole, Jacqueline Chen, Amanda Legge, Aiden R. Doherty, Paul Kelly, Melody Oliver, Hannah M. Badland, and Charlie Foster. 2013. Using the SenseCam to improve classifications of sedentary behavior in free-living settings. American Journal of Preventive Medicine 44, 3 (2013), 290–296.

[61]

Sander Koelstra, Christian Mühl, Mohammad Soleymani, Jong Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. 2012. DEAP: A database for emotion analysis, Using physiological signals 3, 1 (2012), 18–31.

Digital Library

[62]

Sari Kujala and Talya Miron-Shatz. 2013. Emotions, experiences and usability in real-life mobile phone use. In Proceedings of the Changing Perspectives (CHI’13). 1061–1070.

Digital Library

[63]

Peter J. Lang. 1995. The emotion probe: Studies of motivation and attention. American Psychologist 50, 5 (1995), 372–385.

[64]

Reed Larson and Mihaly Csikszentmihalyi. 1983. The experience sampling method. New Directions for Methodology of Social & Behavioral Science 15 (1983), 41–56.

[65]

Lucian Leahu, Steve Schwenk, and Phoebe Sengers. 2008. Subjective objectivity: Negotiating emotional meaning. In Proceedings of the Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (DIS’08). 425–434.

Digital Library

[66]

Angela Y. Lee and Brian Sternthal. 1999. The effects of positive mood on memory. Journal of Consumer Research 26, 2 (1999), 115–127. Retrieved from http://www.jstor.org/stable/10.1086/209554.

[67]

Clayton Lewis. 1982. Using the “Thinking-Aloud” Method in Cognitive Interface Design. IBM T. J. Watson Research Center, Yorktown Heights, NY. Retrieved June 19, 2019 from https://www.worldcat.org/title/using-the-thinking-aloud-method-in-cognitive-interface-design/oclc/38214134.

[68]

Alexandros Liapis, Christos Katsanos, Dimitris Sotiropoulos, Michalis Xenos, and Nikos Karousos. 2015. Recognizing emotions in human computer interaction: Studying stress using skin conductance. In Proceedings of the IFIP Conference on Human-Computer Interaction. 255–262.

Digital Library

[69]

World Famous Electronics Llc. 2015. Pulse Sensor. Retrieved August 1, 2016 from http://pulsesensor.com.

[70]

Sascha Mahlke. 2008. Visual Aesthetics and the user Experience.

[71]

Sascha Mahlke, Michael Minge, and Manfred Thüring. 2006. Measuring multiple components of emotions in interactive contexts. In Proceedings of the Extended Abstracts on Human Factors in Computing Systems (CHI EA’06). 1061–1066.

Digital Library

[72]

Regan L. Mandryk and M. Stella Atkins. 2007. A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. International Journal of Human-Computer Studies 65, 4 (2007), 329–347.

Digital Library

[73]

Regan L. Mandryk, M. Stella Atkins, and Kori M. Inkpen. 2006. A continuous and objective evaluation of emotional experience with interactive play environments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’06). 1027–1036.

Digital Library

[74]

Regan L. Mandryk, Kori M. Inkpen, and Thomas W. Calvert. 2006. Using psychophysiological techniques to measure user experience with entertainment technologies. Behaviour and Information Technology 25, 2 (2006), 141–158.

[75]

Sharon McDonald, Tingting Zhao, and Helen M. Edwards. 2013. Dual verbal elicitation: The Complementary use of concurrent and retrospective reporting within a usability test. International Journal of Human-Computer Interaction 29, 10 (2013), 647–660.

[76]

MindPlace. 2014. Mindplace Thoughtstream. Retrieved August 1, 2016 from http://www.mindplace.com/.

[77]

Talya Miron-Shatz, Arthur Stone, and Daniel Kahneman. 2009. Memories of yesterday's emotions: Does the valence of experience affect the memory-experience gap? Emotion 9, 6 (2009), 885–891.

[78]

Jon D. Morris. 1995. Observations: SAM: The self-assessment manikin: An efficient cross-cultural measurement of emotional response. Journal of Advertising Research 35, 6 (1995), 63–68.

[79]

Brendan D. Murray, Alisha C. Holland, and Elisabeth A. Kensinger. 2013. Episodic memory and emotion. In Handbook of Cognition and Emotion. M. D. Robinson, E. R. Watkins, and E. Harmon-Jones (Eds.), Guilford Press, 156–175.

[80]

Jakob Nielsen. 1993. Usability Engineering. Academic Press. Retrieved June 19, 2019 from https://www.sciencedirect.com/book/9780125184069/usability-engineering.

[81]

K. N. Ochsner. 2000. Are affective events richly recollected or simply familiar? The experience and process of recognizing feelings past. Journal of Experimental Psychology. General 129, 2 (2000), 242–261.

[82]

Kenneth R. Ohnemus and David W. Biers. 1993. Retrospective versus concurrent thinking-out-loud in usability testing. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 37, 17 (1993), 1127–1131.

[83]

Mary M. Omodei and Jim McLennan. 1994. Studying complex decision making in natural settings: Using a head-mounted video camera to study competitive orienteering. Perceptual and Motor Skills 79, 3, Pt 2 (1994), 1411–1425.

[84]

Andrew Perrin. 2015. Social Media Usage: 2005–2015. Retrieved from http://www.pewinternet.org/2015/10/08/social-networking-usage-2005-2015/.

[85]

Linda W. P. Peute, Nicolette F. de Keizer, and Monique W. M. Jaspers. 2015. The value of retrospective and concurrent think aloud in formative usability testing of a physician data query tool. Journal of Biomedical Informatics 55 (2015), 1–10.

Digital Library

[86]

Pierre Philippot, Celine Baeyens, and Celine Douilliez. 2006. Specifying emotional information: Regulation of emotional intensity via executive processes. Emotion (Washington, D.C.) 6, 4 (2006), 560–571.

[87]

Rosalind W. Picard, Szymon Fedor, and Yadid Ayzenberg. 2016. Multiple arousal theory and daily-life electrodermal activity asymmetry. Emotion Review 8, 1 (2016), 62–75.

[88]

Jenny Preece, Yvonne Rogers, and Helen Sharp. 2019. Interaction Design: Beyond Human-Computer Interaction. Retrieved May 2, 2019 from https://www.wiley.com/en-us/Interaction+Design%3A+Beyond+Human+Computer+Interaction%2C+4th+Edition-p-9781119020752.

[89]

Jenny S. Radesky, Caroline J. Kistin, Barry Zuckerman, Katie Nitzberg, Jamie Gross, Margot Kaplan-Sanoff, Marilyn Augustyn, and Michael Silverstein. 2014. Patterns of mobile device use by caregivers and children during meals in fast food restaurants. Pediatrics 133, 4 (2014), e843–9.

[90]

Donald A. Redelmeier and Daniel Kahneman. 1996. Patients’ memories of painful medical treatments: Real-time and retrospective evaluations of two minimally invasive procedures. Pain 66, 1 (1996), 3–8.

[91]

Donald A. Redelmeier, Joel Katz, and Daniel Kahneman. 2003. Memories of colonoscopy: A randomized trial. Pain 104, 1–2 (2003), 187–194.

[92]

Paul Rozin and Edward B. Royzman. 2001. Negativity bias, negativity dominance, and contagion. Personality and Social Psychology Review 5, 4 (2001), 296–320.

[93]

D. M. Russell and M. Oren. 2009. Retrospective cued recall: A method for accurately recalling previous user behaviors. In Proceedings of the 42nd Hawaii International Conference on System Sciences (HICSS’09). 1–9.

Digital Library

[94]

J. A. Russell and A. Mehrabian. 1974. Distinguishing anger and anxiety in terms of emotional response factors. Journal of Consulting and Clinical Psychology 42, 1 (1974), 79–83. Retrieved June 19, 2019 from http://www.ncbi.nlm.nih.gov/ /4814102.

[95]

James A. Russell and Albert Mehrabian. 1977. Evidence for a three-factor theory of emotions. Journal of Research in Personality 11, 3 (1977), 273–294.

[96]

J. Edward Russo, Eric J. Johnson, and Debra L. Stephens. 1989. The validity of verbal protocols. Memory & Cognition 17, 6 (1989), 759–769.

[97]

J. Edward Russo. 1979. A software system for the collection of retrospective protocols prompted by eye fixations. Behavior Research Methods & Instrumentation 11, 2 (1979), 177–179.

[98]

Sinué Salgado and Osman Skjold Kingo. 2019. How is physiological arousal related to self-reported measures of emotional intensity and valence of events and their autobiographical memories? Consciousness and Cognition 75.

[99]

L. Salmerón, J. Naumann, V. García, and I. Fajardo. 2017. Scanning and deep processing of information in hypertext: An eye tracking and cued retrospective think-aloud study. Journal of Computer Assisted Learning 33, 3 (2017), 222–233.

[100]

Klaus R. Scherer. 2005. What are emotions? And how can they be measured? Social Science Information 44, 4 (2005), 695–729.

[101]

Christie N. Scollon, Chu Kim-Prieto, and Ed Diener. 2003. Experience sampling: Promises and pitfalls, strengths and weaknesses. Journal of Happiness Studies 4, 1 (2003), 5–34.

[102]

Abigail J. Sellen, Andrew Fogg, Mike Aitken, Steve Hodges, Carsten Rother, and Ken Wood. 2007. Do life-logging technologies support memory for the past? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’07). 81.

Digital Library

[103]

N. Sadat Shami, Jeffrey T. Hancock, Christian Peter, Michael Muller, and Regan Mandryk. 2008. Measuring affect in HCI: Going beyond the individual. In Proceedings of the CHI’08 Extended Abstracts on Human Factors in Computing Systems (CHI EA’08). 3901–3904.

Digital Library

[104]

Adam P. R. Smith, Richard N. A. Henson, Michael D. Rugg, and Raymond J. Dolan. 2005. Modulation of retrieval processing reflects accuracy of emotional source memory. Learning & Memory 12, 5 (2005), 472–479.

[105]

Anna Ståhl, Kristina Höök, Martin Svensson, Alex S. Taylor, and Marco Combetto. 2009. Experiencing the affective diary. Personal Ubiquitous Computing 13, 5 (2009), 365–378.

Digital Library

[106]

Arthur A. Stone, Joseph E. Schwartz, John M. Neale, Ssaul Shiffman, Christine A. Marco, Mary Hickcox, Jean Paty, Laura S. Porter, and Laura J. Cruise. 1998. A comparison of coping assessed by ecological momentary assessment and retrospective recall. Journal of Personality and Social Psychology 74, 6 (1998), 1670–1680.

[107]

K. Lynn Taylor and Jean-Paul Dionne. 2000. Accessing problem-solving strategy knowledge: The complementary use of concurrent verbal protocols and retrospective debriefing. Journal of Educational Psychology 92, 3 (2000), 413–425.

[108]

Jonathan R. Zadra and Gerald L. Clore. 2011. Emotion and perception: The role of affective information. Wiley Interdisciplinary Reviews. Cognitive Science 2, 6 (2011), 676–685.

[109]

Feng Zhou. 2014. Emotion prediction from physiological signals: A comparison study between visual and auditory elicitors. Interacting with Computers 26, 3 (2014), 285–302.

[110]

Philippe Zimmermann. 2008. Beyond Usability—Measuring Aspects of User Experience. Ph.D. Dissertation. ETH Zürich Diss. Zürich, Switzerland, ETH NO. 17901.

Cited By

Hertzum M(2024)Concurrent or Retrospective Thinking Aloud in Usability Tests: A Meta-Analytic ReviewACM Transactions on Computer-Human Interaction10.1145/366532731:3(1-29)Online publication date: 17-May-2024
https://dl.acm.org/doi/10.1145/3665327
Chanmas GTaveekitworachai PYou XThawonmas RNukoolkit CDajpratham P(2024)Driving assistant using generative AI pre-generated messages in simulator-based driving assessment: A step towards low-cost simulator-based driving assessmentHeliyon10.1016/j.heliyon.2024.e3594110:16(e35941)Online publication date: Aug-2024
https://doi.org/10.1016/j.heliyon.2024.e35941
Borawska AMateja A(2024)Unveiling the User Experience: A Synthesis of Cognitive Neuroscience Methods in Digital Product DesignAdvances in Information Systems Development10.1007/978-3-031-57189-3_10(199-218)Online publication date: 18-Jul-2024
https://doi.org/10.1007/978-3-031-57189-3_10
Show More Cited By

Index Terms

Do You Feel the Same? On the Robustness of Cued-Recall Debriefing for User Experience Evaluation
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI
    2. HCI design and evaluation methods
      1. Laboratory experiments

Recommendations

Asserting Real-Time Emotions through Cued-Recall: Is it Valid?
NordiCHI '16: Proceedings of the 9th Nordic Conference on Human-Computer Interaction

Asserting emotions through free-recall is commonly used to evaluate user experience (UX) of interactive systems. From psychology we know that free-recall of emotions leads to a significant memory bias where participants rely on a few of the most intense ...
A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies

The popularity of computer games has exploded in recent years, yet methods of evaluating user emotional state during play experiences lag far behind. There are few methods of assessing emotional state, and even fewer methods of quantifying emotion ...
Emotional user experience

Emotional experience has become an important topic in human-technology interaction research and design. Nevertheless, such research and design often lacks a proper explanatory basis and methodologically robust operationalisation. In this article, a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer-Human Interaction

ACM Transactions on Computer-Human Interaction Volume 28, Issue 4

August 2021

297 pages

ISSN:1073-0516

EISSN:1557-7325

DOI:10.1145/3477419

Editor:
Kristina Höök
KTH Royal Institute of Technology, Sweden

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2021

Accepted: 01 March 2021

Revised: 01 August 2020

Received: 01 December 2019

Published in TOCHI Volume 28, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
313
Total Downloads

Downloads (Last 12 months)85
Downloads (Last 6 weeks)12

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hertzum M(2024)Concurrent or Retrospective Thinking Aloud in Usability Tests: A Meta-Analytic ReviewACM Transactions on Computer-Human Interaction10.1145/366532731:3(1-29)Online publication date: 17-May-2024
https://dl.acm.org/doi/10.1145/3665327
Chanmas GTaveekitworachai PYou XThawonmas RNukoolkit CDajpratham P(2024)Driving assistant using generative AI pre-generated messages in simulator-based driving assessment: A step towards low-cost simulator-based driving assessmentHeliyon10.1016/j.heliyon.2024.e3594110:16(e35941)Online publication date: Aug-2024
https://doi.org/10.1016/j.heliyon.2024.e35941
Borawska AMateja A(2024)Unveiling the User Experience: A Synthesis of Cognitive Neuroscience Methods in Digital Product DesignAdvances in Information Systems Development10.1007/978-3-031-57189-3_10(199-218)Online publication date: 18-Jul-2024
https://doi.org/10.1007/978-3-031-57189-3_10
Klimek AMannheim ISchouten GWouters EPeeters M(2023)Wearables measuring electrodermal activity to assess perceived stress in care: a scoping reviewActa Neuropsychiatrica10.1017/neu.2023.19(1-11)Online publication date: 24-Mar-2023
https://doi.org/10.1017/neu.2023.19
Gorson JCunningham KWorsley MO'Rourke E(2022)Using Electrodermal Activity Measurements to Understand Student Emotions While ProgrammingProceedings of the 2022 ACM Conference on International Computing Education Research - Volume 110.1145/3501385.3543981(105-119)Online publication date: 3-Aug-2022
https://dl.acm.org/doi/10.1145/3501385.3543981
Wang JLiu TLiu ZChai Y(2022)Exploring the influencing factors of wall‐following behavior in a virtual reality fire evacuation gameComputer Animation and Virtual Worlds10.1002/cav.212234:1Online publication date: 20-Aug-2022
https://doi.org/10.1002/cav.2122

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents