research-article

How Intention Informed Recommendations Modulate Choices: A Field Study of Spoken Word Content

Authors:

Michael Sobolev,

Christina Tsangouri,

Deborah EstrinAuthors Info & Claims

WWW '19: The World Wide Web Conference

Pages 2169 - 2180

https://doi.org/10.1145/3308558.3313540

Published: 13 May 2019 Publication History

Abstract

People's content choices are ideally driven by their intentions, aspirations, and plans. However, in reality, choices may be modulated by recommendation systems which are typically trained to promote popular items and to reinforce users' historical behavior. As a result, the utility and user experience of content consumption can be affected implicitly and undesirably. To study this problem, we conducted a 2 × 2 randomized controlled field experiment (105 urban college students) to compare the effects of intention informed recommendations with classical intention agnostic systems. The study was conducted in the context of spoken word web content (podcasts) which is often consumed through subscription sites or apps. We modified a commercial podcast app to include (1) a recommender that takes into account users' stated intentions at onboarding, and (2) a Collaborative Filtering (CF) recommender during daily use. Our study suggests that: (1) intention-aware recommendations can significantly raise users' interactions (subscriptions and listening) with channels and episodes related to intended topics by over 24%, even if such a recommender is only used during onboarding, and (2) the CF-based recommender doubles users' explorations on episodes from not-subscribed channels and improves satisfaction for users onboarded with the intention-aware recommender.

References

[1]

Gediminas Adomavicius and Alexander Tuzhilin. 2011. Context-aware recommender systems. In Recommender systems handbook. Springer, 217-253.

[2]

Deepak Agarwal, Bee-Chung Chen, Pradheep Elango, and Raghu Ramakrishnan. 2013. Content recommendation on web portals. Commun. ACM 56, 6 (2013), 92-101.

Digital Library

[3]

Eytan Bakshy, Solomon Messing, and Lada A Adamic. 2015. Exposure to ideologically diverse news and opinion on Facebook. Science 348, 6239 (2015), 1130-1132.

[4]

Oded Berger-Tal, Jonathan Nathan, Ehud Meron, and David Saltz. 2014. The exploration-exploitation dilemma: a multidisciplinary framework. PloS one 9, 4 (2014), e95693.

[5]

Jana Besser, Martha Larson, and Katja Hofmann. 2010. Podcast search: User goals and retrieval technologies. Online information review(2010).

[6]

Robert J Boik. 1979. Interactions, partial interactions, and interaction contrasts in the analysis of variance.Psychological Bulletin 86, 5 (1979), 1084.

[7]

Danah Boyd. 2010. Streams of content, limited attention: The flow of information through social media.Educause Review 45, 5 (2010), 26.

[8]

Allison JB Chaney, Brandon M Stewart, and Barbara E Engelhardt. 2017. How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility. arXiv preprint arXiv:1710.11214(2017).

[9]

Li Chen and Pearl Pu. 2012. Critiquing-based recommenders: survey and emerging trends. User Modeling and User-Adapted Interaction 22, 1-2 (2012), 125-150.

Digital Library

[10]

Justin Cheng, Caroline Lo, and Jure Leskovec. 2017. Predicting intent using activity logs: How goal specificity and temporal range affect user behavior. In Proceedings of the 26th International Conference on World Wide Web Companion. International World Wide Web Conferences Steering Committee, 593-601.

Digital Library

[11]

Elizabeth M Daly, Werner Geyer, and David R Millen. 2010. The network effects of recommending social connections. In Proceedings of the fourth ACM conference on Recommender systems. ACM, 301-304.

Digital Library

[12]

Zhicheng Dou, Ruihua Song, and Ji-Rong Wen. 2007. A large-scale evaluation and analysis of personalized search strategies. In Proceedings of the 16th international conference on World Wide Web. ACM, 581-590.

Digital Library

[13]

Michael D Ekstrand, Mucun Tian, Ion Madrazo Azpiazu, Jennifer D Ekstrand, Oghenemaro Anuyah, David McNeill, and Maria Soledad Pera. 2018. All The Cool Kids, How Do They Fit In?: Popularity and Demographic Biases in Recommender Evaluation and Effectiveness. In Conference on Fairness, Accountability and Transparency. 172-186.

[14]

Michael D Ekstrand and Martijn C Willemsen. 2016. Behaviorism is not enough: better recommendations through listening to users. In Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 221-224.

Digital Library

[15]

Andrew J Elliot and Judith M Harackiewicz. 1994. Goal setting, achievement orientation, and intrinsic motivation: A mediational analysis.Journal of personality and social psychology 66, 5(1994), 968.

[16]

Seth Flaxman, Sharad Goel, and Justin M Rao. 2016. Filter bubbles, echo chambers, and online news consumption. Public opinion quarterly 80, S1 (2016), 298-320.

[17]

Marguerite Fuller, E Tsagkias, Eamonn Newman, Jana Besser, Martha Larson, Gareth JF Jones, M Rijke, 2008. Using term clouds to represent segment-level semantic content of podcasts. (2008).

[18]

Peter M Gollwitzer. 1999. Implementation intentions: strong effects of simple plans.American psychologist 54, 7 (1999), 493.

[19]

Masataka Goto and Jun Ogata. 2011. PodCastle: Recent advances of a spoken document retrieval service improved by anonymous user contributions. In Twelfth Annual Conference of the International Speech Communication Association.

[20]

Kartik Hosanagar, Daniel Fleder, Dokyun Lee, and Andreas Buja. 2013. Will the global village fracture into tribes? Recommender systems and their effects on consumer fragmentation. Management Science 60, 4 (2013), 805-823.

Digital Library

[21]

Cheng-Kang Hsieh, Longqi Yang, Honghao Wei, Mor Naaman, and Deborah Estrin. 2016. Immersive recommendation: News and event recommendations using personal digital traces. In Proceedings of the 25th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee.

Digital Library

[22]

Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative filtering for implicit feedback datasets. In Data Mining, 2008. ICDM'08. Eighth IEEE International Conference on. Ieee, 263-272.

Digital Library

[23]

Michael Inzlicht, Brandon J Schmeichel, and C Neil Macrae. 2014. Why self-control seems (but may not be) limited. Trends in cognitive sciences 18, 3 (2014), 127-133.

[24]

Dietmar Jannach, Paul Resnick, Alexander Tuzhilin, and Markus Zanker. 2016. Recommender systems-beyond matrix completion. Commun. ACM 59, 11 (2016), 94-102.

Digital Library

[25]

Jie Kang, Kyle Condiff, Shuo Chang, Joseph A Konstan, Loren Terveen, and F Maxwell Harper. 2017. Understanding how people use natural language to ask for recommendations. In Proceedings of the Eleventh ACM Conference on Recommender Systems. ACM, 229-237.

Digital Library

[26]

Bart P Knijnenburg, Saadhika Sivakumar, and Daricia Wilkinson. 2016. Recommender systems for self-actualization. In Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 11-14.

Digital Library

[27]

Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M Henne. 2009. Controlled experiments on the web: survey and practical guide. Data mining and knowledge discovery 18, 1 (2009), 140-181.

Digital Library

[28]

Mounia Lalmas and Liangjie Hong. 2018. Tutorial on Metrics of User Engagement: Applications to News, Search and E-Commerce. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM, 781-782.

Digital Library

[29]

Uichin Lee, Zhenyu Liu, and Junghoo Cho. 2005. Automatic identification of user goals in web search. In Proceedings of the 14th international conference on World Wide Web. ACM, 391-400.

Digital Library

[30]

Leonard A Marascuilo and Joel R Levin. 1970. Appropriate post hoc comparisons for interaction and nested hypotheses in analysis of variance designs: The elimination of type IV errors. American Educational Research Journal 7, 3 (1970), 397-421.

[31]

Katherine L Milkman, Dolly Chugh, and Max H Bazerman. 2009. How can decision making be improved?Perspectives on psychological science 4, 4 (2009), 379-383.

[32]

Katherine L Milkman, Todd Rogers, and Max H Bazerman. 2008. Harnessing our inner angels and demons: What we have learned about want/should conflicts and how that knowledge can help us reduce short-sighted decision making. Perspectives on Psychological Science 3, 4 (2008), 324-338.

[33]

Junta Mizuno, Jun Ogata, and Masataka Goto. 2008. A similar content retrieval method for podcast episodes. In Spoken Language Technology Workshop, 2008. SLT 2008. IEEE. IEEE.

[34]

Tien T Nguyen, Pik-Mai Hui, F Maxwell Harper, Loren Terveen, and Joseph A Konstan. 2014. Exploring the filter bubble: the effect of using recommender systems on content diversity. In Proceedings of the 23rd international conference on World wide web. ACM, 677-686.

Digital Library

[35]

Jun Ogata and Masataka Goto. 2009. PodCastle: Collaborative training of acoustic models on the basis of wisdom of crowds for podcast transcription. In Tenth Annual Conference of the International Speech Communication Association.

[36]

Eli Pariser. 2011. The filter bubble: How the new personalized web is changing what we read and how we think. Penguin.

Digital Library

[37]

Edison Research. 2017. The Podcast Consumer 2017. http://www.edisonresearch.com/the-podcast-consumer-2017/

[38]

William W Ronan, Gary P Latham, and SB Kinne. 1973. Effects of goal setting and supervision on worker behavior in an industrial situation.journal of Applied Psychology 58, 3 (1973), 302.

[39]

Daniel E Rose and Danny Levinson. 2004. Understanding user goals in web search. In Proceedings of the 13th international conference on World Wide Web. ACM, 13-19.

Digital Library

[40]

Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as treatments: debiasing learning and evaluation. In Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume 48. JMLR. org.

Digital Library

[41]

Amit Sharma, Jake M Hofman, and Duncan J Watts. 2015. Estimating the causal impact of recommendation systems from observational data. In Proceedings of the Sixteenth ACM Conference on Economics and Computation. ACM, 453-470.

Digital Library

[42]

Ana-Andreea Stoica, Christopher Riederer, and Augustin Chaintreau. 2018. Algorithmic Glass Ceiling in Social Networks: The effects of social recommendations on network diversity. In Proceedings of the 2018 World Wide Web Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 923-932.

Digital Library

[43]

Jessica Su, Aneesh Sharma, and Sharad Goel. 2016. The effect of recommendations on network structure. In Proceedings of the 25th international conference on World Wide Web. International World Wide Web Conferences Steering Committee, 1157-1167.

Digital Library

[44]

Jaime Teevan, Susan T Dumais, and Daniel J Liebling. 2008. To personalize or not to personalize: modeling queries with variation in user intent. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 163-170.

Digital Library

[45]

Choon Hui Teo, Houssam Nassif, Daniel Hill, Sriram Srinivasan, Mitchell Goodman, Vijai Mohan, and SVN Vishwanathan. 2016. Adaptive, personalized diversity for visual discovery. In Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 35-38.

Digital Library

[46]

Sabina Tomkins, Steven Isley, Ben London, and Lise Getoor. 2018. Sustainability at scale: towards bridging the intention-behavior gap with sustainable recommendations. In Proceedings of the 12th ACM Conference on Recommender Systems. ACM, 214-218.

Digital Library

[47]

Manos Tsagkias, Martha Larson, and Maarten De Rijke. 2010. Predicting podcast preference: An analysis framework and its application. Journal of the American Society for information Science and Technology 61, 2 (2010), 374-391.

Digital Library

[48]

Ryen W White and Steven M Drucker. 2007. Investigating behavioral variability in web search. In Proceedings of the 16th international conference on World Wide Web. ACM, 21-30.

Digital Library

[49]

Jacob O Wobbrock, Leah Findlater, Darren Gergle, and James J Higgins. 2011. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 143-146.

Digital Library

[50]

Longqi Yang, Eugene Bagdasaryan, Joshua Gruenstein, Cheng-Kang Hsieh, and Deborah Estrin. 2018. OpenRec: A Modular Framework for Extensible and Adaptable Recommendation Algorithms. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM, 664-672.

Digital Library

[51]

Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. 2018. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In Proceedings of the 12th ACM Conference on Recommender Systems. ACM, 279-287.

Digital Library

[52]

Longqi Yang, Cheng-Kang Hsieh, Hongjian Yang, John P Pollak, Nicola Dell, Serge Belongie, Curtis Cole, and Deborah Estrin. 2017. Yum-me: a personalized nutrient-based meal recommender system. ACM Transactions on Information Systems (TOIS) 36, 1 (2017), 7.

Digital Library

[53]

Longqi Yang, Michael Sobolev, Christina Tsangouri, and Deborah Estrin. 2018. Understanding user interactions with podcast recommendations delivered via voice. In Proceedings of the 12th ACM Conference on Recommender Systems. ACM, 190-194.

Digital Library

[54]

Mike Yeomans, Anuj Shah, Sendhil Mullainathan, and Jon Kleinberg. 2016. Making sense of recommendations. Preprint at http://scholar. harvard. edu/files/sendhil/files/recommenders55_01. pdf (2016).

Cited By

Zhao YLi TSobolev MRubin JLam WCatolino GPoshyvanyk D(2024)Digital Wellbeing Redefined: Toward User-Centric Approach for Positive Social Media EngagementProceedings of the IEEE/ACM 11th International Conference on Mobile Software Engineering and Systems10.1145/3647632.3651392(95-98)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3647632.3651392
Sadeghian AOtarkhani A(2023)Data-driven digital nudging: a systematic literature review and future agendaBehaviour & Information Technology10.1080/0144929X.2023.2286535(1-29)Online publication date: 29-Nov-2023
https://doi.org/10.1080/0144929X.2023.2286535
Rafique WHafid AQadir J(2023)Developing smart city services using intent‐aware recommendation systems: A surveyTransactions on Emerging Telecommunications Technologies10.1002/ett.472834:4Online publication date: 12-Jan-2023
https://doi.org/10.1002/ett.4728
Show More Cited By

Recommendations

User Similarity Adjustment for Improved Recommendations
MIKE 2015: Proceedings of the Third International Conference on Mining Intelligence and Knowledge Exploration - Volume 9468

Recommender systems are becoming more and more attractive in both research and commercial communities due to Information overload problem and the popularity of the Internet applications. Collaborative Filtering, a popular branch of recommendation ...
A field study of related video recommendations: newest, most similar, or most relevant?
RecSys '18: Proceedings of the 12th ACM Conference on Recommender Systems

Many video sites recommend videos related to the one a user is watching. These recommendations have been shown to influence what users end up exploring and are an important part of a recommender system. Plenty of methods have been proposed to recommend ...
Explanations that are Intrinsic to Recommendations
UMAP '18: Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization

Explanations can give credibility to recommendations and help users to make better choices. In current recommender systems, explanation is a step that comes after recommendation. In this paper, we describe an approach that turns recommender systems on ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

WWW '19: The World Wide Web Conference

May 2019

3620 pages

ISBN:9781450366748

DOI:10.1145/3308558

Editors:
Ling Liu
Georgia Tech, USA
,
Ryen White
Microsoft Research, USA

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

IW3C2: International World Wide Web Conference Committee

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '19

WWW '19: The Web Conference

May 13 - 17, 2019

CA, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
552
Total Downloads

Downloads (Last 12 months)32
Downloads (Last 6 weeks)4

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhao YLi TSobolev MRubin JLam WCatolino GPoshyvanyk D(2024)Digital Wellbeing Redefined: Toward User-Centric Approach for Positive Social Media EngagementProceedings of the IEEE/ACM 11th International Conference on Mobile Software Engineering and Systems10.1145/3647632.3651392(95-98)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3647632.3651392
Sadeghian AOtarkhani A(2023)Data-driven digital nudging: a systematic literature review and future agendaBehaviour & Information Technology10.1080/0144929X.2023.2286535(1-29)Online publication date: 29-Nov-2023
https://doi.org/10.1080/0144929X.2023.2286535
Rafique WHafid AQadir J(2023)Developing smart city services using intent‐aware recommendation systems: A surveyTransactions on Emerging Telecommunications Technologies10.1002/ett.472834:4Online publication date: 12-Jan-2023
https://doi.org/10.1002/ett.4728
Nazari ZChandar PFazelnia GEdwards CCarterette BLalmas M(2022)Choice of Implicit Signal Matters: Accounting for User Aspirations in Podcast RecommendationsProceedings of the ACM Web Conference 202210.1145/3485447.3512115(2433-2441)Online publication date: 25-Apr-2022
https://dl.acm.org/doi/10.1145/3485447.3512115
Piao JZhang GXu FChen ZZheng YGao CLi Y(2021)Bringing Friends into the Loop of Recommender Systems: An Exploratory StudyProceedings of the ACM on Human-Computer Interaction10.1145/34795835:CSCW2(1-26)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479583
Olivia Snooks KLindley JRichards DWhitham R(2021)Context-Aware WearablesExtended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411763.3450367(1-9)Online publication date: 8-May-2021
https://dl.acm.org/doi/10.1145/3411763.3450367
Jones RZamani HSchedl MChen CReddy SClifton AKarlgren JHashemi HPappu ANazari ZYang LSemerci OBouchard HCarterette BDiaz FShah CSuel TCastells PJones RSakai T(2021)Current Challenges and Future Directions in Podcast Information AccessProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3462805(1554-1565)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3404835.3462805
Dean SRich SRecht BHildebrandt MCastillo CCelis ERuggieri STaylor LZanfir-Fortuna G(2020)Recommendations and user agencyProceedings of the 2020 Conference on Fairness, Accountability, and Transparency10.1145/3351095.3372866(436-445)Online publication date: 27-Jan-2020
https://dl.acm.org/doi/10.1145/3351095.3372866
Sobolev M(undefined)Digital Nudging: Using Technology to Nudge for GoodSSRN Electronic Journal10.2139/ssrn.3889831
https://doi.org/10.2139/ssrn.3889831

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents