extended-abstract

Towards Designing for Multimodal Remembering: Findings from an Interview Study

Author:

Ege OtenenAuthors Info & Claims

CHI EA '24: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

Article No.: 616, Pages 1 - 6

https://doi.org/10.1145/3613905.3647965

Published: 11 May 2024 Publication History

Abstract

Humans experience, remember, and express the world through their senses. These different modalities are also essential in forming and recalling autobiographical memories. People remember the color of the shirt they wore, the voice of their partner, and the taste of their mother’s soup. They also express their memories with different modalities by telling it with their words or showing pictures. Although a rising number of studies focused on developing technologies to support remembering, studies have yet to take the opportunity to explore the role of modalities to aid memories. We interviewed 12 participants about their memories by allowing them to remember and express memories with different modalities to understand better how to design for multimodal remembering. This paper is a first step toward developing a framework that aims to explore multimodal remembering, improve design practices by providing insights for future memory technologies.

References

[1]

Donna Rose Addis, Alana T Wong, and Daniel L Schacter. 2008. Age-related changes in the episodic simulation of future events. Psychological Science 19, 1 (2008), 33–41. http://www.jstor.org/stable/40064797

[2]

Judith Amores Fernandez, Nirmita Mehra, Bjoern Rasch, and Pattie Maes. 2023. Olfactory Wearables for Mobile Targeted Memory Reactivation. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–20. https://doi.org/10.1145/3544548.3580892

Digital Library

[3]

Mirjam Augstein and Thomas Neumayr. 2019. A human-centered taxonomy of interaction modalities and devices. Interacting with Computers 31, 1 (2019), 27–58. https://doi.org/10.1093/iwc/iwz003

[4]

Cagla Aydin. 2018. The differential contributions of visual imagery constructs on autobiographical thinking. Memory 26, 2 (2018), 189–200. https://doi.org/10.1080/09658211.2017.1340483

[5]

Cagla Aydin, Tilbe Göksun, Ege Otenen, Selma Berfin Tanis, and Yağmur Damla Şentürk. 2023. The role of gestures in autobiographical memory. Plos One 18, 2 (2023), e0281748. https://doi.org/10.1371/journal.pone.0281748

[6]

Dorthe Berntsen and David C Rubin. 2004. Cultural life scripts structure recall from autobiographical memory. Memory & Cognition 32, 3 (2004), 427–442. https://doi.org/10.3758/bf03195836

[7]

Olessia Blajenkova, Maria Kozhevnikov, and Michael A Motes. 2006. Object-spatial imagery: a new self-report imagery questionnaire. Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition 20, 2 (2006), 239–263. https://doi.org/10.1002/acp.1182

[8]

Georgina Browne, Emma Berry, Narinder Kapur, Steve Hodges, Gavin Smyth, Peter Watson, and Ken Wood. 2011. SenseCam improves memory for recent events and quality of life in a patient with memory retrieval difficulties. Memory 19, 7 (2011), 713–722. https://doi.org/10.1080/09658211.2011.614622

[9]

Kelly Caine. 2016. Local standards for sample size at CHI. In Proceedings of the 2016 CHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 981–992. https://doi.org/10.1145/2858036.2858498

Digital Library

[10]

Simon Chu and John J Downes. 2000. Odour-evoked autobiographical memories: Psychological investigations of Proustian phenomena. Chemical Senses 25, 1 (2000), 111–116. https://doi.org/10.1093/chemse/25.1.111

[11]

Martin A Conway. 2005. Memory and the self. Journal of Memory and Language 53, 4 (2005), 594–628. https://doi.org/10.1016/j.jml.2005.08.005

[12]

Martin A Conway and Christopher W Pleydell-Pearce. 2000. The construction of autobiographical memories in the self-memory system.Psychological Review 107, 2 (2000), 261.

[13]

Masashi Crete-Nishihata, Ronald M Baecker, Michael Massimi, Deborah Ptak, Rachelle Campigotto, Liam D Kaufman, Adam M Brickman, Gary R Turner, Joshua R Steinerman, and Sandra E Black. 2012. Reconstructing the past: personal memory technologies are not just personal and not just for memory. Human–Computer Interaction 27, 1-2 (2012), 92–123. https://doi.org/10.1080/07370024.2012.656062

[14]

Maaike J De Bruijn and Michael Bender. 2018. Olfactory cues are more effective than visual cues in experimentally triggering autobiographical memories. Memory 26, 4 (2018), 547–558. https://doi.org/10.1080/09658211.2017.1381744

[15]

Lina Dib, Daniela Petrelli, and Steve Whittaker. 2010. Sonic souvenirs: exploring the paradoxes of recorded sound for family remembering. In Proceedings of the 2010 ACM conference on Computer supported cooperative work. Association for Computing Machinery, New York, NY, USA, 391–400. https://doi.org/10.1145/1718918.1718985

Digital Library

[16]

Tilman Dingler, Passant El Agroudy, Rufat Rzayev, Lars Lischke, Tonja Machulla, and Albrecht Schmidt. 2021. Memory Augmentation Through Lifelogging: Opportunities and Challenges. Springer International Publishing, Cham, 47–69. https://doi.org/10.1007/978-3-030-30457-7_3

[17]

Alexandra Ernst, Julie MF Bertrand, Virginie Voltzenlogel, Céline Souchay, and Christopher JA Moulin. 2021. The Proust machine: What a public science event tells us about autobiographical memory and the five senses. Frontiers in Psychology 11 (2021), 623910. https://doi.org/10.3389/fpsyg.2020.623910

[18]

Tom Gayler, Corina Sas, and Vaiva Kalnikaite. 2023. “It took me back 25 years in one bound”: self-generated flavor-based cues for self-defining memories in later life. Human–Computer Interaction 38, 5-6 (2023), 417–458. https://doi.org/10.1080/07370024.2022.2107518

[19]

Jim Gemmell, Gordon Bell, and Roger Lueder. 2006. MyLifeBits: a personal database for everything. Commun. ACM 49, 1 (2006), 88–95. https://doi.org/10.1145/1107458.1107460

Digital Library

[20]

Seth J Gillihan, Jennifer Kessler, and Martha J Farah. 2007. Memories affect mood: Evidence from covert experimental assignment to positive, neutral, and negative memory recall. Acta Psychologica 125, 2 (2007), 144–154. https://doi.org/10.1016/j.actpsy.2006.07.009

[21]

Lorna Goddard, Linda Pring, and Nick Felmingham. 2005. The effects of cue modality on the quality of personal memories retrieved. Memory 13, 1 (2005), 79–86. https://doi.org/10.1080/09658210344000594

[22]

James M Goodman and Sliman J Bensmaia. 2020. The neural mechanisms of touch and proprioception at the somatosensory periphery. Neuron 104, 5 (2020), 1000–1009. https://doi.org/10.1016/B978-0-12-805408-6.00014-2

[23]

Steve Hodges, Emma Berry, and Ken Wood. 2011. SenseCam: A wearable camera that stimulates and rehabilitates autobiographical memory. Memory 19, 7 (2011), 685–696. https://doi.org/10.1080/09658211.2011.605591

[24]

Francesco Ianì. 2019. Embodied memories: Reviewing the role of the body in memory processes. Psychonomic Bulletin & Review 26 (2019), 1747–1766. https://doi.org/10.3758/s13423-019-01674-x

[25]

Jolanda Jetten, Catherine Haslam, Cara Pugliese, James Tonks, and S Alexander Haslam. 2010. Declining autobiographical memory and the loss of identity: Effects on well-being. Journal of Clinical and Experimental Neuropsychology 32, 4 (2010), 408–416. https://doi.org/10.1080/13803390903140603

[26]

Marcia K Johnson, Mary A Foley, Aurora G Suengas, and Carol L Raye. 1988. Phenomenal characteristics of memories for perceived and imagined autobiographical events.Journal of Experimental Psychology: General 117, 4 (1988), 371. https://doi.org/10.1037/0096-3445.117.4.371

[27]

Subin Kim, Sangsu Jang, Jin-young Moon, Minjoo Han, and Young-Woo Park. 2022. Slide2Remember: an Interactive Wall Frame Enriching Reminiscence Experiences by Providing Re-encounters of Taken Photos and Heard Music in a Similar Period. In Designing Interactive Systems Conference. Association for Computing Machinery, New York, NY, USA, 288–300. https://doi.org/10.1145/3532106.3533456

Digital Library

[28]

Marie Kirk and Dorthe Berntsen. 2018. A short cut to the past: Cueing via concrete objects improves autobiographical memory retrieval in Alzheimer’s disease patients. Neuropsychologia 110 (2018), 113–122. https://doi.org/10.1016/j.neuropsychologia.2017.06.034

[29]

Igor Knez, Louise Ljunglöf, Artin Arshamian, and Johan Willander. 2017. Self-grounding visual, auditory and olfactory autobiographical memories. Consciousness and Cognition 52 (2017), 1–8. https://doi.org/10.1016/j.concog.2017.04.008

[30]

Artie Konrad, Ellen Isaacs, and Steve Whittaker. 2016. Technology-mediated memory: is technology altering our memories and interfering with well-being?ACM Transactions on Computer-Human Interaction (TOCHI) 23, 4 (2016), 1–29. https://doi.org/10.1145/2934667

Digital Library

[31]

Ethan Kross, Ozlem Ayduk, and Walter Mischel. 2005. When asking “why” does not hurt distinguishing rumination from reflective processing of negative emotions. Psychological Science 16, 9 (2005), 709–715. https://doi.org/10.1111/j.1467-9280.2005.01600.x

[32]

Huy Viet Le, Sarah Clinch, Corina Sas, Tilman Dingler, Niels Henze, and Nigel Davies. 2016. Impact of video summary viewing on episodic memory recall: Design guidelines for video summarizations. In Proceedings of the 2016 CHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 4793–4805. https://doi.org/10.1145/2858036.2858413

Digital Library

[33]

Brian Levine, Eva Svoboda, Janine F Hay, Gordon Winocur, and Morris Moscovitch. 2002. Aging and autobiographical memory: dissociating episodic from semantic retrieval.Psychology and Aging 17, 4 (2002), 677. https://doi.org/10.1037/0882-7974.17.4.677

[34]

Eleanor A Maguire and Sinéad L Mullally. 2013. The hippocampus: a manifesto for change.Journal of Experimental Psychology: General 142, 4 (2013), 1180. https://doi.org/10.1037/a0033650

[35]

Ali Mair, Marie Poirier, and Martin A Conway. 2019. Memory for staged events: Supporting older and younger adults’ memory with SenseCam. Quarterly Journal of Experimental Psychology 72, 4 (2019), 717–728. https://doi.org/10.1177/1747021818765038

[36]

Nathalie Overdevest, Rakesh Patibanda, Aryan Saini, Elise Van Den Hoven, and Florian ‘Floyd’ Mueller. 2023. Towards Designing for Everyday Embodied Remembering: Findings from a Diary Study. In Proceedings of the 2023 ACM Designing Interactive Systems Conference. Association for Computing Machinery, New York, NY, USA, 2611–2624. https://doi.org/10.1145/3563657.3595999

Digital Library

[37]

Hettie Roebuck and Gary Lupyan. 2020. The internal representations questionnaire: Measuring modes of thinking. Behavior Research Methods 52 (2020), 2053–2070. https://doi.org/10.3758/s13428-020-01354-y

[38]

David C Rubin, Elisabeth Groth, and Debra J Goldsmith. 1984. Olfactory cuing of autobiographical memory. The American Journal of Psychology 97, 4 (1984), 493–507. https://doi.org/10.2307/1422158

[39]

David C Rubin, Robert W Schrauf, and Daniel L Greenberg. 2003. Belief and recollection of autobiographical memories. Memory & Cognition 31 (2003), 887–901. https://doi.org/10.3758/bf03196443

[40]

Carol D Ryff and Corey Lee M Keyes. 1995. The structure of psychological well-being revisited.Journal of Personality and Social Psychology 69, 4 (1995), 719. https://doi.org/10.1037//0022-3514.69.4.719

[41]

Rufat Rzayev, Tilman Dingler, and Niels Henze. 2018. Reflectivediary: fostering human memory through activity summaries created from implicit data collection. In Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia. Association for Computing Machinery, New York, NY, USA, 285–291. https://doi.org/10.1145/3282894.3282907

Digital Library

[42]

Corina Sas, Tomasz Fratczak, Matthew Rees, Hans Gellersen, Vaiva Kalnikaite, Alina Coman, and Kristina Höök. 2013. AffectCam: arousal- augmented sensecam for richer recall of episodic memories. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems (Paris, France) (CHI EA ’13). Association for Computing Machinery, New York, NY, USA, 1041–1046. https://doi.org/10.1145/2468356.2468542

Digital Library

[43]

Peggy L St. Jacques and Brian Levine. 2007. Ageing and autobiographical memory for emotional and neutral events. Memory 15, 2 (2007), 129–144. https://doi.org/10.1080/09658210601119762

[44]

Angelina R Sutin and Richard W Robins. 2007. Phenomenology of autobiographical memories: The memory experiences questionnaire. Memory 15, 4 (2007), 390–411. https://doi.org/10.1080/09658210701256654

[45]

Elise Van den Hoven and Ali Mazalek. 2011. Grasping gestures: Gesturing with physical artifacts. AI EDAM 25, 3 (2011), 255–271. https://doi.org/10.1017/S0890060411000072

[46]

Jayne Wallace, Peter C Wright, John McCarthy, David Philip Green, James Thomas, and Patrick Olivier. 2013. A design-led inquiry into personhood in dementia. In Proceedings of the SIGCHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 2617–2626. https://doi.org/10.1145/2470654.2481363

Digital Library

[47]

Johan Willander, Sverker Sikström, and Kristina Karlsson. 2015. Multimodal retrieval of autobiographical memories: Sensory information contributes differently to the recollection of events. Frontiers in Psychology 6 (2015), 1681. https://doi.org/10.3389/fpsyg.2015.01681

[48]

Katrin Wolf, Albrecht Schmidt, Agon Bexheti, and Marc Langheinrich. 2014. Lifelogging: You’re wearing a camera?IEEE Pervasive Computing 13, 03 (2014), 8–12. https://doi.org/10.1109/MPRV.2014.53

Cited By

Bosco COtenen EOsorio Torres JNguyen VChheda DPeng XJessup NHimes ACureton BLu YHill CHendrie HBarnes PShih P(2024)Designing a multimodal and culturally relevant ADRD generative AI tool for Black American informal caregivers: A cognitive walk-through usability study. (Preprint)JMIR Aging10.2196/60566Online publication date: 11-Jun-2024
https://doi.org/10.2196/60566

Index Terms

Towards Designing for Multimodal Remembering: Findings from an Interview Study
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques
  2. Interaction design
    1. Interaction design process and methods
      1. Interface design prototyping
      2. User centered design

Index terms have been assigned to the content through auto-classification.

Recommendations

Towards Designing for Everyday Embodied Remembering: Findings from a Diary Study
DIS '23: Proceedings of the 2023 ACM Designing Interactive Systems Conference

Our bodies play an important part in our remembering practices, for example when we can remember passwords by typing, even if we cannot verbalise them. An increasing number of technologies are being developed to support remembering. However, so far, they ...
Pensieve: supporting everyday reminiscence
CHI '10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Reminiscing is a valuable activity that people of all ages spontaneously and informally partake in as part of their everyday lives. This paper discusses the design and use of Pensieve, a system that supports everyday reminiscence by emailing memory ...
Designing Memory Probes to Inform Dialogue
DIS '17: Proceedings of the 2017 Conference on Designing Interactive Systems

To investigate the phenomenon that occurs during interactions between used objects and autobiographical memories, which are both ever-changing and imbedded with personal significance, an adapted probing method capable of managing these complex qualities ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '24: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

May 2024

4761 pages

ISBN:9798400703317

DOI:10.1145/3613905

Editors:
Florian Floyd Mueller
Monash University
,
Penny Kyburz
The Australian National University
,
Julie R. Williamson
University of Glasgow
,
Corina Sas
Lancaster University

Copyright © 2024 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 May 2024

Check for updates

Author Tags

Qualifiers

Extended-abstract
Research
Refereed limited

Conference

CHI '24

Sponsor:

CHI '24: CHI Conference on Human Factors in Computing Systems

May 11 - 16, 2024

HI, Honolulu, USA

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
677
Total Downloads

Downloads (Last 12 months)677
Downloads (Last 6 weeks)67

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bosco COtenen EOsorio Torres JNguyen VChheda DPeng XJessup NHimes ACureton BLu YHill CHendrie HBarnes PShih P(2024)Designing a multimodal and culturally relevant ADRD generative AI tool for Black American informal caregivers: A cognitive walk-through usability study. (Preprint)JMIR Aging10.2196/60566Online publication date: 11-Jun-2024
https://doi.org/10.2196/60566

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Table of Contents