research-article

Exploring Mental Models for Transparent and Controllable Recommender Systems: A Qualitative Study

Authors:

Johannes Kunkel,

Jürgen ZieglerAuthors Info & Claims

UMAP '20: Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization

Pages 183 - 191

https://doi.org/10.1145/3340631.3394841

Published: 13 July 2020 Publication History

Abstract

While online content is personalized to an increasing degree, eg. using recommender systems (RS), the rationale behind personalization and how users can adjust it typically remains opaque. This was often observed to have negative effects on the user experience and perceived quality of RS. As a result, research increasingly has taken user-centric aspects such as transparency and control of a RS into account, when assessing its quality. However, we argue that too little of this research has investigated the users' perception and understanding of RS in their entirety. In this paper, we explore the users' mental models of RS. More specifically, we followed the qualitative grounded theory methodology and conducted 10 semi-structured face-to-face interviews with typical and regular Netflix users. During interviews participants expressed high levels of uncertainty and confusion about the RS in Netflix. Consequently, we found a broad range of different mental models. Nevertheless, we also identified a general structure underlying all of these models, consisting of four steps: data acquisition, inference of user profile, comparison of user profiles or items, and generation of recommendations. Based on our findings, we discuss implications to design more transparent, controllable, and user friendly RS in the future.

Supplementary Material

VTT File (3340631.3394841.vtt)

Download
15.76 KB

MP4 File (3340631.3394841.mp4)

Supplemental Video

Download
35.70 MB

References

[1]

Svetlin Bostandjiev, John O'Donovan, and Tobias Höllerer. 2012. TasteWeights: A Visual Interactive Hybrid Recommender System. In Proceedings of the Sixth ACM Conference on Recommender Systems (RecSys '12). ACM, New York, NY, USA, 35--42. https://doi.org/10.1145/2365952.2365964

Digital Library

[2]

André Calero Valdez, Martina Ziefle, and Katrien Verbert. 2016. HCI for Recommender Systems: The Past, the Present and the Future. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys '16). ACM, New York, NY, USA, 123--126. https://doi.org/10.1145/2959100.2959158

Digital Library

[3]

Shuo Chang, F. Maxwell Harper, and Loren Terveen. 2015. Using Groups of Items for Preference Elicitation in Recommender Systems. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15). ACM, New York, NY, USA, 1258--1269. https://doi.org/10.1145/2675133.2675210

Digital Library

[4]

Leon Ciechanowski, Aleksandra Przegalinska, Mikolaj Magnuski, and Peter Gloor. 2019. In the shades of the uncanny valley: An experimental study of human--chatbot interaction. Future Generation Computer Systems, Vol. 92 (March 2019), 539--548. https://doi.org/10.1016/j.future.2018.01.055

Digital Library

[5]

Juliet Corbin and Anselm Strauss. 2008. Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory .SAGE Publications, Thousand Oaks, California. https://doi.org/10.4135/9781452230153

[6]

Henriette Cramer, Vanessa Evers, Satyan Ramlal, Maarten Van Someren, Lloyd Rutledge, Natalia Stash, Lora Aroyo, and Bob Wielinga. 2008. The effects of transparency on trust in and acceptance of a content-based art recommender. User Modeling and User-Adapted Interaction, Vol. 18, 5 (Aug. 2008), 455. https://doi.org/10.1007/s11257-008--9051--3

Digital Library

[7]

Will Dahlgreen. 2016. Streaming wars: the actors Netflix and Amazon customers want to see. https://yougov.co.uk/topics/politics/articles-reports/2016/01/14/streaming-wars-actors-netflix-and-amazon-customers Retrieved January, 15, 2020 from

[8]

Deloitte. 2017. Welchen Video-on-Demand-Anbieter nutzen Sie? https://de.statista.com/statistik/daten/studie/443820/umfrage/genutzte-video-on-demand-anbieter-in-deutschland/ Retrieved April, 24, 2020 from

[9]

Malin Eiband, Hanna Schneider, Mark Bilandzic, Julian Fazekas-Con, Mareike Haug, and Heinrich Hussmann. 2018. Bringing Transparency Design into Practice. In 23rd International Conference on Intelligent User Interfaces (IUI '18). ACM, New York, NY, USA, 211--223. https://doi.org/10.1145/3172944.3172961

Digital Library

[10]

Muheeb Faizan Ghori, Arman Dehpanah, Jonathan Gemmell, Hamed Qahri-Saremi, and Bamshad Mobasher. 2019. Does the User Have A Theory of the Recommender? A Pilot Study. In Proceedings of Joint Workshop on Interfaces and Human Decision Making for Recommender Systems. CEUR-WS.org, 77--85.

[11]

Carlos A. Gomez-Uribe and Neil Hunt. 2015. The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM Transactions on Management Information Systems, Vol. 6, 4 (Dec. 2015), 13:1--13:19. https://doi.org/10.1145/2843948

[12]

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A Survey of Methods for Explaining Black Box Models. Comput. Surveys, Vol. 51, 5, Article Article 93 (Aug. 2018), 42 pages. https://doi.org/10.1145/3236009

Digital Library

[13]

Jonathan L. Herlocker, Joseph A. Konstan, and John Riedl. 2000. Explaining Collaborative Filtering Recommendations. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (CSCW '00). ACM, 241--250. https://doi.org/10.1145/358916.358995

Digital Library

[14]

Christie Kodama, Beth St. Jean, Mega Subramaniam, and Natalie Greene Taylor. 2017. There's a creepy guy on the other end at Google!: engaging middle school students in a drawing activity to elicit their mental models of Google. Information Retrieval Journal, Vol. 20, 5 (Oct. 2017), 403--432. https://doi.org/10.1007/s10791-017--9306-x

Digital Library

[15]

Joseph A. Konstan and John Riedl. 2012. Recommender systems: from algorithms to user experience. User Modeling and User-Adapted Interaction, Vol. 22, 1--2 (March 2012), 101--123. https://doi.org/10.1007/s11257-011--9112-x

Digital Library

[16]

Todd Kulesza, Simone Stumpf, Margaret Burnett, and Irwin Kwan. 2012. Tell Me More?: The Effects of Mental Model Soundness on Personalizing an Intelligent Agent. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1--10. https://doi.org/10.1145/2207676.2207678

Digital Library

[17]

Johannes Kunkel, Tim Donkers, Catalin-Mihai Barbu, and Jürgen Ziegler. 2018a. Trust-Related Effects of Expertise and Similarity Cues in Human-Generated Recommendations. In Companion Proceedings of the 23rd International on Intelligent User Interfaces: 2nd Workshop on Theory-Informed User Modeling for Tailoring and Personalizing Interfaces (HUMANIZE). http://ceur-ws.org/Vol-2068/humanize5.pdf

[18]

Johannes Kunkel, Tim Donkers, Lisa Michael, Catalin-Mihai Barbu, and Jürgen Ziegler. 2019. Let Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, 1--12. https://doi.org/10.1145/3290605.3300717

Digital Library

[19]

Johannes Kunkel, Benedikt Loepp, and Jürgen Ziegler. 2017. A 3D Item Space Visualization for Presenting and Manipulating User Preferences in Collaborative Filtering. In Proceedings of the 22nd International Conference on Intelligent User Interfaces (IUI '17). ACM, New York, NY, USA, 3--15. https://doi.org/10.1145/3025171.3025189

Digital Library

[20]

Johannes Kunkel, Benedikt Loepp, and Jürgen Ziegler. 2018b. Understanding Latent Factors Using a GWAP. In Proceedings of the Late-Breaking Results track part of the Twelfth ACM Conference on Recommender Systems (RecSys'18). https://arxiv.org/pdf/1808.10260.pdf

[21]

Benedikt Loepp, Tim Donkers, Timm Kleemann, and Jürgen Ziegler. 2019. Interactive recommending with Tag-Enhanced Matrix Factorization (TagMF). International Journal of Human-Computer Studies 121 (Jan. 2019), 21--41. https://doi.org/10.1016/j.ijhcs.2018.05.002

[22]

Benedikt Loepp, Tim Hussein, and Jürgen Ziegler. 2014. Choice-based preference elicitation for collaborative filtering recommender systems. In Proceedings of the 32nd International Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 3085--3094. https://doi.org/10.1145/2556288.2557069

Digital Library

[23]

Julian McAuley and Jure Leskovec. 2013. Hidden Factors and Hidden Topics: Understanding Rating Dimensions with Review Text. In Proceedings of the 7th ACM Conference on Recommender Systems (RecSys '13). ACM, New York, NY, USA, 165--172. https://doi.org/10.1145/2507157.2507163

Digital Library

[24]

Sean M. McNee, John Riedl, and Joseph A. Konstan. 2006. Being Accurate is Not Enough: How Accuracy Metrics Have Hurt Recommender Systems. In CHI '06 Extended Abstracts on Human Factors in Computing Systems (CHI EA '06). ACM, New York, NY, USA, 1097--1101. https://doi.org/10.1145/1125451.1125659

[25]

Janice M. Morse. 2015. "Data Were Saturated...". Qualitative Health Research, Vol. 25, 5 (2015), 587--588. https://doi.org/10.1177/1049732315576699

[26]

Donald A. Norman. 1983. Some Observations on Mental Models. In Mental Models, Dedre Gentner and Albert L. Stevens (Eds.). Psychology Press, New York, NY, USA, 7--14.

[27]

Donald A. Norman. 1988. The design of everyday things. Basic Books, Inc., New York, NY, USA.

Digital Library

[28]

John O'Donovan, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and Tobias Höllerer. 2008. PeerChooser: Visual Interactive Recommendation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, New York, NY, USA, 1085--1088. https://doi.org/10.1145/1357054.1357222

Digital Library

[29]

Pearl Pu, Li Chen, and Rong Hu. 2011. A User-centric Evaluation Framework for Recommender Systems. In Proceedings of the Fifth ACM Conference on Recommender Systems (RecSys '11). ACM, New York, NY, USA, 157--164. https://doi.org/10.1145/2043932.2043962

Digital Library

[30]

Lingyun Qiu and Izak Benbasat. 2009. Evaluating Anthropomorphic Product Recommendation Agents: A Social Relationship Perspective to Designing Information Systems. Journal of Management Information Systems, Vol. 25, 4 (Dec. 2009), 145--182. https://doi.org/10.2753/MIS0742--1222250405

Digital Library

[31]

Al Mamunur Rashid, Istvan Albert, Dan Cosley, Shyong K. Lam, Sean M. McNee, Joseph A. Konstan, and John Riedl. 2002. Getting to Know You: Learning New User Preferences in Recommender Systems. In Proceedings of the 7th International Conference on Intelligent User Interfaces (IUI '02). ACM, New York, NY, USA, 127--134.

Digital Library

[32]

Quentin Roy, Futian Zhang, and Daniel Vogel. 2019. Automation Accuracy Is Good, but High Controllability May Be Better. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, 520:1--520:8. https://doi.org/10.1145/3290605.3300750

Digital Library

[33]

David E. Rumelhart and Donald A. Norman. 1983. Representation in Memory.

[34]

Ben Shneiderman and Pattie Maes. 1997. Direct Manipulation vs. Interface Agents. interactions, Vol. 4, 6 (Nov. 1997), 42--61. https://doi.org/10.1145/267505.267514

[35]

Itamar Simonson. 2005. Determinants of Customers' Responses to Customized Offers: Conceptual Framework and Research Propositions. Journal of Marketing, Vol. 69, 1 (Jan. 2005), 32--45. https://doi.org/10.1509/jmkg.69.1.32.55512

[36]

Rashmi Sinha and Kirsten Swearingen. 2002. The role of transparency in recommender systems. In CHI '02 Extended Abstracts on Human Factors in Computing Systems (CHI EA '02). ACM, New York, NY, USA, 830--831. https://doi.org/10.1145/506443.506619

Digital Library

[37]

E. Isaac Sparling and Shilad Sen. 2011. Rating: how difficult is it?. In Proceedings of the Fifth ACM Conference on Recommender Systems (RecSys '11). ACM, New York, NY, USA, 149--156.

Digital Library

[38]

Statista.com. 2018. Most popular video streaming services in the United States as of July 2018, by monthly average users. https://www.statista.com/statistics/910875/us-most-popular-video-streaming-services-by-monthly-average-users/s Retrieved January, 15, 2020 from

[39]

Anselm Strauss and Juliet Corbin. 1994. Grounded theory methodology. In Handbook of qualitative research, Norman K. Denzin and Yvonna S. Lincoln (Eds.). SAGE Publications, Thousand Oaks, CA, USA, 273--285.

[40]

Nava Tintarev and Judith Masthoff. 2015. Explaining Recommendations: Design and Evaluation. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer US, Boston, MA, USA, 353--382. https://doi.org/10.1007/978--1--4899--7637--6textunderscore10

[41]

Helma Torkamaan, Catalin-Mihai Barbu, and Jürgen Ziegler. 2019. How Can They Know That? A Study of Factors Affecting the Creepiness of Recommendations. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys '19). ACM, New York, NY, USA, 423--427. https://doi.org/10.1145/3298689.3346982

Digital Library

[42]

Chun-Hua Tsai and Peter Brusilovsky. 2019. Explaining Recommendations in an Interactive Hybrid Social Recommender. In Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI '19). ACM, New York, NY, USA, 391--396. https://doi.org/10.1145/3301275.3302318

Digital Library

Cited By

Sarıkaya B(2024)CONCEPTUAL REVIEW OF ARTIFICIAL INTELLIGENCE; DIFFERENCES BETWEEN HUMAN AND MACHINE LEARNINGTurkish Online Journal of Design Art and Communication10.7456/tojdac.1464262Online publication date: 20-May-2024
https://doi.org/10.7456/tojdac.1464262
Rezk ASimkute AVines JElsden CEvans MJones RLuger E(2024)User-Centric Tensions: Exploring Perceived Benefits and (Dis)comfort in Media PersonalisationProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685365(1-13)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3679318.3685365
Himbeault LLatulipe C(2024)An Initial Exploration of Code Diagram Query EffectivenessProceedings of the 26th Western Canadian Conference on Computing Education10.1145/3660650.3660664(1-7)Online publication date: 2-May-2024
https://dl.acm.org/doi/10.1145/3660650.3660664
Show More Cited By

Index Terms

Exploring Mental Models for Transparent and Controllable Recommender Systems: A Qualitative Study
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems

Recommendations

User Personality and User Satisfaction with Recommender Systems

In this study, we show that individual users' preferences for the level of diversity, popularity, and serendipity in recommendation lists cannot be inferred from their ratings alone. We demonstrate that we can extract strong signals about individual ...
Acquiring User Information Needs for Recommender Systems
WI-IAT '13: Proceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT) - Volume 03

Most recommender systems attempt to use collaborative filtering, content-based filtering or hybrid approach to recommend items to new users. Collaborative filtering recommends items to new users based on their similar neighbours, and content-based ...
A Scalable, Accurate Hybrid Recommender System
WKDD '10: Proceedings of the 2010 Third International Conference on Knowledge Discovery and Data Mining

Recommender systems apply machine learning techniques for filtering unseen information and can predict whether a user would like a given resource. There are three main types of recommender systems: collaborative filtering, content-based filtering, and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UMAP '20: Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization

July 2020

426 pages

ISBN:9781450368612

DOI:10.1145/3340631

Editors:
Tsvi Kuflik
University of Haifa, Israel
,
Ilaria Torre
University of Genoa, Italy
,
Robin Burke
University of Colorado, Boulder, USA
,
Cristina Gena
University of Turin, Italy

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Deutsche Forschungsgemeinschaft

Conference

UMAP '20

Sponsor:

UMAP '20: 28th ACM Conference on User Modeling, Adaptation and Personalization

July 14 - 17, 2020

Genoa, Italy

Acceptance Rates

Overall Acceptance Rate 162 of 633 submissions, 26%

Upcoming Conference

UMAP '25

Sponsor:
sigchi
sigchi

33rd ACM Conference on User Modeling, Adaptation and Personalization

June 16 - 19, 2025

New York City , NY , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

41
Total Citations
View Citations
981
Total Downloads

Downloads (Last 12 months)226
Downloads (Last 6 weeks)15

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sarıkaya B(2024)CONCEPTUAL REVIEW OF ARTIFICIAL INTELLIGENCE; DIFFERENCES BETWEEN HUMAN AND MACHINE LEARNINGTurkish Online Journal of Design Art and Communication10.7456/tojdac.1464262Online publication date: 20-May-2024
https://doi.org/10.7456/tojdac.1464262
Rezk ASimkute AVines JElsden CEvans MJones RLuger E(2024)User-Centric Tensions: Exploring Perceived Benefits and (Dis)comfort in Media PersonalisationProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685365(1-13)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3679318.3685365
Himbeault LLatulipe C(2024)An Initial Exploration of Code Diagram Query EffectivenessProceedings of the 26th Western Canadian Conference on Computing Education10.1145/3660650.3660664(1-7)Online publication date: 2-May-2024
https://dl.acm.org/doi/10.1145/3660650.3660664
Ge YLiu SFu ZTan JLi ZXu SLi YXian YZhang Y(2024)A Survey on Trustworthy Recommender SystemsACM Transactions on Recommender Systems10.1145/3652891Online publication date: 13-Apr-2024
https://doi.org/10.1145/3652891
Suchmann LKrämer NZiegler JBoratto LGena CMarras MGermanakos PPopescus E(2024)Dynamic Ridge Plot Sliders: Supporting Users' Understanding of the Item Space Structure and Feature Dependencies in Interactive Recommender SystemsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664872(106-113)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3631700.3664872
Rezk ASimkute ALuger EVines JElsden CEvans MJones R(2024)Agency Aspirations: Understanding Users' Preferences And Perceptions Of Their Role In Personalised News CurationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642634(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642634
Panagoulias DVirvou MTsihrintzis G(2024)A novel framework for artificial intelligence explainability via the Technology Acceptance Model and Rapid Estimate of Adult Literacy in Medicine using machine learningExpert Systems with Applications10.1016/j.eswa.2024.123375248(123375)Online publication date: Aug-2024
https://doi.org/10.1016/j.eswa.2024.123375
Mohanthy SVinya VTejasvi KPadmaja JYadla SGodavarthi S(2024)Explainable AI Methods and ApplicationsExplainable AI in Health Informatics10.1007/978-981-97-3705-5_2(33-48)Online publication date: 8-Jul-2024
https://doi.org/10.1007/978-981-97-3705-5_2
Smits Avan der Zandt Lvan Turnhout K(2024)Why Designers Must Contribute to Responsible AIArtificial Intelligence in HCI10.1007/978-3-031-60611-3_7(87-104)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-60611-3_7
Tejasvi KVinya VPadmaja JBegum RJabbar M(2024)Explainable Artificial Intelligence (XAI) for Managing Customer Needs in E-Commerce: A Systematic ReviewRole of Explainable Artificial Intelligence in E-Commerce10.1007/978-3-031-55615-9_2(17-31)Online publication date: 26-Apr-2024
https://doi.org/10.1007/978-3-031-55615-9_2
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents