opinion

Measuring the Business Value of Recommender Systems

Authors:

Dietmar Jannach,

Michael JugovacAuthors Info & Claims

ACM Transactions on Management Information Systems (TMIS), Volume 10, Issue 4

Article No.: 16, Pages 1 - 23

https://doi.org/10.1145/3370082

Published: 10 December 2019 Publication History

Abstract

Recommender Systems are nowadays successfully used by all major web sites—from e-commerce to social media—to filter content and make suggestions in a personalized way. Academic research largely focuses on the value of recommenders for consumers, e.g., in terms of reduced information overload. To what extent and in which ways recommender systems create business value is, however, much less clear, and the literature on the topic is scattered. In this research commentary, we review existing publications on field tests of recommender systems and report which business-related performance measures were used in such real-world deployments. We summarize common challenges of measuring the business value in practice and critically discuss the value of algorithmic improvements and offline experiments as commonly done in academic environments. Overall, our review indicates that various open questions remain both regarding the realistic quantification of the business effects of recommenders and the performance assessment of recommendation algorithms in academia.

References

[1]

Sue F. Abdinnour-Helm, Barbara S. Chaparro, and Steven M. Farmer. 2005. Using the end-user computing satisfaction (EUCS) instrument to measure satisfaction with a web site. Decis. Sci. 36, 2 (2005), 341--364.

[2]

Himan Abdollahpouri, Gediminas Adomavicius, Robin Burke, Ido Guy, Dietmar Jannach, Toshihiro Kamishima, Jan Krasnodebski, and Luiz Pizzato. 2019. Beyond personalization: Research directions in multistakeholder recommendation. Retrieved from https://arxiv.org/abs/1905.01986.

[3]

Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. 2017. Recommender systems as multistakeholder environments. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization (UMAP’17). 347--348.

Digital Library

[4]

Fernando Amat, Ashok Chandrashekar, Tony Jebara, and Justin Basilico. 2018. Artwork personalization at Netflix. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys’18). 487--488.

Digital Library

[5]

Xavier Amatriain and Justin Basilico. 2012. Netflix Recommendations: Beyond the 5 stars. Retrieved from https://medium.com/netflix-techblog/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429.

[6]

Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009. Improvements that don’t add up: Ad-hoc retrieval results since 1998. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM’09). 601--610.

Digital Library

[7]

Riccardo Bambini, Paolo Cremonesi, and Roberto Turrin. 2011. Recommender Systems Handbook. Springer, 299--331. Francesco Ricci, Lior Rokach, Bracha Shapira, and Paul B. Kantor (Eds.).

[8]

Mathieu Bastian, Matthew Hayes, William Vaughan, Sam Shah, Peter Skomoroch, Hyungjin Kim, Sal Uryasev, and Christopher Lloyd. 2014. LinkedIn skills: Large-scale topic extraction and inference. In Proceedings of the 8th ACM Conference on Recommender Systems (RecSys’14). 1--8.

Digital Library

[9]

Joeran Beel, Marcel Genzmehr, Stefan Langer, Andreas Nürnberger, and Bela Gipp. 2013. A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation. In Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation (RepSys’13). 7--14.

Digital Library

[10]

Jöran Beel and Stefan Langer. 2015. A comparison of offline evaluations, online evaluations, and user studies in the context of research-paper recommender systems. In Proceedings of the 22nd International Conference on Theory and Practice of Digital Libraries (TPDL’15). 153--168.

[11]

Joeran Beel, Stefan Langer, and Bela Gipp. 2017. TF-IDuF: A novel term-weighting scheme for user modeling based on users’ personal document collections. In Proceedings of the iConference 2017. 452--459.

[12]

Anand V. Bodapati. 2008. Recommendation systems with purchase data. J. Market. Res. 45, 1 (2008), 77--93.

[13]

Frederik J. Zuiderveen Borgesius, Damian Trilling, Judith Moeller, Balazs Bodo, Claes H. de Vreese, and Natali Helberger. 2016. Should we worry about filter bubbles? Internet Policy Rev. 5, 1 (2016).

[14]

Yuri M. Brovman, Marie Jacob, Natraj Srinivasan, Stephen Neola, Daniel Galron, Ryan Snyder, and Paul Wang. 2016. Optimizing similar item recommendations in a semi-structured marketplace to maximize conversion. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys’16). 199--202.

Digital Library

[15]

Diego Carraro and Derek Bridge. 2019. Debiased offline evaluation of recommender systems: A weighted-sampling approach (extended abstract). In Proceedings of the ACM RecSys Workshop on Reinforcement and Robust Estimators for Recommendation (REVEAL’19).

[16]

Olivier Chapelle, Thorsten Joachims, Filip Radlinski, and Yisong Yue. 2012. Large-scale validation and analysis of interleaved search evaluation. ACM Trans. Info. Syst. 30, 1 (2012), 6:1--6:41.

Digital Library

[17]

Pei-Yu Chen, Yen-Chun Chou, and R. J. Kauffman. 2009. Community-based recommender systems: Analyzing business models from a systems operator’s perspective. In Proceedings of the 42nd Hawaii International Conference on System Sciences (HICSS’09). 1--10.

[18]

Ye Chen and John F. Canny. 2011. Recommending ephemeral items at web scale. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’11). 1013--1022.

[19]

Paolo Cremonesi, Franca Garzotto, and Roberto Turrin. 2012. Investigating the persuasion potential of recommender systems from a quality perspective: An empirical study. ACM Trans. Interact. Intell. Syst. 2, 2 (June 2012), 11:1--11:41.

Digital Library

[20]

Paolo Cremonesi, Franca Garzotto, and Roberto Turrin. 2013. User-centric vs. system-centric evaluation of recommender systems. In Proceedings of the 14th International Conference on Human-Computer Interaction (INTERACT’13). 334--351.

[21]

Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019. Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In Proceedings of the ACM Conference on Recommender Systems (RecSys’19). Copenhagen.

Digital Library

[22]

Abhinandan S. Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. 2007. Google news personalization: Scalable online collaborative filtering. In Proceedings of the 16th International Conference on World Wide Web (WWW’07). 271--280.

Digital Library

[23]

James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, and Dasarathi Sampath. 2010. The YouTube video recommendation system. In Proceedings of the 4th ACM Conference on Recommender Systems (RecSys’10). 293--296.

Digital Library

[24]

Alex Deng, Ya Xu, Ron Kohavi, and Toby Walker. 2013. Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In Proceedings of the 6th ACM International Conference on Web Search and Data Mining (WSDM’13). 123--132.

Digital Library

[25]

M. Benjamin Dias, Dominique Locher, Ming Li, Wael El-Deredy, and Paulo J. G. Lisboa. 2008. The value of personalised recommender systems to e-business: A case study. In Proceedings of the ACM Conference on Recommender Systems (RecSys’08). 291--294.

[26]

William J. Doll and Gholamreza Torkzadeh. 1988. The measurement of end-user computing satisfaction. MIS Quart. 12, 2 (1988), 259--274.

Digital Library

[27]

Marcos Aurélio Domingues, Fabien Gouyon, Alípio Mário Jorge, José Paulo Leal, João Vinagre, Luís Lemos, and Mohamed Sordo. 2013. Combining usage and content in an online recommendation system for music in the Long Tail. Int. J. Multimedia Info. Retriev. 2, 1 (2013), 3--13.

[28]

Michael D. Ekstrand, F. Maxwell Harper, Martijn C. Willemsen, and Joseph A. Konstan. 2014. User perception of differences in recommender algorithms. In Proceedings of the 8th ACM Conference on Recommender Systems (RecSys’14). 161--168.

[29]

Gerhard Friedrich and Markus Zanker. 2011. A taxonomy for generating explanations in recommender systems. AI Mag. 32, 3 (June 2011), 90--98.

[30]

Florent Garcin, Boi Faltings, Olivier Donatsch, Ayar Alazzawi, Christophe Bruttin, and Amr Huber. 2014. Offline and online evaluation of news recommender systems at Swissinfo.ch. In Proceedings of the 8th ACM Conference on Recommender Systems (RecSys’14). 169--176.

Digital Library

[31]

Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. 2018. Offline A/B testing for recommender systems. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining (WSDM’18). 198--206.

Digital Library

[32]

Carlos A. Gomez-Uribe and Neil Hunt. 2015. The Netflix recommender system: Algorithms, business value, and innovation. Trans. Manage. Info. Syst. 6, 4 (2015), 13:1--13:19.

[33]

Bryan Gumm. 2013. A/B Testing: The Most Powerful Way to Turn Clicks into Customers. Wiley, 180--193. Dan Siroker and Pete Koomen (Eds.).

[34]

Azleen Ilias, Norazah Bte Mohd Suki, Mohd Rushdan Yasoa, and Mohd Zulkeflee Abd Razak. 2008. The end-user computing satisfaction (EUCS) on computerized accounting system (CAS): How they perceived? J. Internet Bank. Comm. 13, 1 (2008), 1--18.

[35]

Rolf Jagerman, Ilya Markov, and Maarten de Rijke. 2019. When people change their mind: Off-policy evaluation in non-stationary recommendation environments. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining (WSDM’19). 447--455.

Digital Library

[36]

Jincheul Jang, Dapeng Zhao, Woneui Hong, Youkyoung Park, and Mun Yong Yi. 2016. Uncovering the underlying factors of smart TV UX over time: A multi-study, mixed-method approach. In Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video (TVX’16). 3--12.

Digital Library

[37]

Dietmar Jannach and Gediminas Adomavicius. 2016. Recommendations with a purpose. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys’16). 7--10.

Digital Library

[38]

Dietmar Jannach and Gediminas Adomavicius. 2017. Price and profit awareness in recommender systems. In Proceedings of the Workshop on Value-Aware and Multi-Stakeholder Recommendation (VAMS’17).

[39]

Dietmar Jannach and Kolja Hegelich. 2009. A case study on the effectiveness of recommendations in the mobile Internet. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys’09). 205--208.

Digital Library

[40]

Dietmar Jannach and Lukas Lerche. 2017. Offline performance vs. subjective quality experience: A case study in video game recommendation. In Proceedings of the ACM Symposium on Applied Computing (SAC’17). 1649--1654.

Digital Library

[41]

Dietmar Jannach, Lukas Lerche, and Michael Jugovac. 2015. Adaptation and evaluation of recommendations for short-term shopping goals. In Proceedings of the 9th ACM Conference on Recommender Systems (RecSys’15). 211--218.

Digital Library

[42]

Dietmar Jannach, Lukas Lerche, and Michael Jugovac. 2015. Item familiarity as a possible confounding factor in user-centric recommender systems evaluation. i-com J. Interact. Media 14, 1 (2015), 29--39.

[43]

Dietmar Jannach, Lukas Lerche, Iman Kamehkhosh, and Michael Jugovac. 2015. What recommenders recommend: An analysis of recommendation biases and possible countermeasures. User Model. User-Adapt. Interact. 25, 5 (2015), 427--491.

Digital Library

[44]

Dietmar Jannach, Malte Ludewig, and Lukas Lerche. 2017. Session-based item recommendation in e-commerce: On short-term intents, reminders, trends, and discounts. User-Model. User-Adapt. Interact. 27, 3--5 (2017), 351--392.

Digital Library

[45]

Dietmar Jannach, Paul Resnick, Alexander Tuzhilin, and Markus Zanker. 2016. Recommender systems—Beyond matrix completion. Commun. ACM 59, 11 (2016), 94--102.

Digital Library

[46]

Dietmar Jannach, Oren Sar Shalom, and Joseph A. Konstan. 2019. Towards more impactful recommender systems research. In Proceedings of the ACM RecSys Workshop on the Impact of Recommender Systems (ImpactRS’19).

[47]

Ray Jiang, Silvia Chiappa, Tor Lattimore, András György, and Pushmeet Kohli. 2019. Degenerate feedback loops in recommender systems. In Proceedings of the Conference on Artificial Intelligence, Ethics, and Society (AIES’19). 383--390.

Digital Library

[48]

Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased learning-to-rank with biased feedback. In Proceedings of the 17th International Joint Conference on Artificial Intelligence (IJCAI’17). 781--789.

Digital Library

[49]

Michael Jugovac, Dietmar Jannach, and Lukas Lerche. 2017. Efficient optimization of multiple recommendation quality factors according to individual user tendencies. Expert Syst. Appl. 81 (2017), 321--331.

Digital Library

[50]

Iman Kamehkhosh and Dietmar Jannach. 2017. User perception of next-track music recommendations. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization (UMAP’17). 113--121.

Digital Library

[51]

Iman Kamehkhosh, Dietmar Jannach, and Geoffray Bonnin. 2018. How automated recommendations affect the playlist creation behavior of users. In Proceedings of the Workshop on Intelligent Music Interfaces for Listening and Creation (MILC’18).

[52]

Marius Kaminskas and Derek Bridge. 2016. Diversity, serendipity, novelty, and coverage: A survey and empirical analysis of beyond-accuracy objectives in recommender systems. ACM Trans. Interact. Intell. Syst. 7, 1 (2016), 2:1--2:42.

Digital Library

[53]

Adem Karahoca, Erkan Bayraktar, Ekrem Tatoglu, and Dilek Karahoca. 2010. Information system design for a hospital emergency department: A usability analysis of software prototypes. J. Biomed. Info. 43, 2 (2010), 224--232.

Digital Library

[54]

J. Katukuri, T. Könik, R. Mukherjee, and S. Kolay. 2014. Recommending similar items in large-scale online marketplaces. In Proceedings of the IEEE International Conference on Big Data 2014. 868--876.

[55]

J. Katukuri, T. Konik, R. Mukherjee, and S. Kolay. 2015. Post-purchase recommendations in large-scale online marketplaces. In Proceedings of the IEEE International Conference on Big Data (BigData’15). 1299--1305.

[56]

Evan Kirshenbaum, George Forman, and Michael Dugan. 2012. A live comparison of methods for personalized article recommendation at Forbes.com. In Machine Learning and Knowledge Discovery in Databases. Springer, 51--66.

[57]

Julia Kiseleva, Melanie J. I. Müller, Lucas Bernardi, Chad Davis, Ivan Kovacek, Mats Stafseng Einarsen, Jaap Kamps, Alexander Tuzhilin, and Djoerd Hiemstra. 2015. Where to go on your next trip? Optimizing travel destinations based on user preferences. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’15).

[58]

Sören Köcher, Michael Jugovac, Dietmar Jannach, and Hartmut Holzmüller. 2019. New hidden persuaders: An investigation of attribute-level anchoring effects of product recommendations. J. Retail. 95 (2019), 24--41. Issue 1.

[59]

Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, and Ya Xu. 2012. Trustworthy online controlled experiments: Five puzzling outcomes explained. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’12). 786--794.

Digital Library

[60]

Joseph A. Konstan and John Riedl. 2012. Recommender systems: From algorithms to user experience. User Model. User-Adapt. Interact. 22, 1 (2012), 101--123.

Digital Library

[61]

R. D. Lawrence, G. S. Almasi, V. Kotlyar, M. S. Viveros, and S. S. Duri. 2001. Personalization of supermarket product recommendations. Data Min. Knowl. Discov. 5, 1 (2001), 11--32.

Digital Library

[62]

Dokyun Lee and Kartik Hosanagar. 2014. Impact of recommender systems on sales volume and diversity. In Proceedings of the International Conference on Information Systems (ICIS’14).

[63]

Dokyun Lee and Kartik Hosanagar. 2019. How do recommender systems affect sales diversity? A cross-category investigation via randomized field experiment. Info. Syst. Res. 30, 1 (2019), 239--259.

[64]

Lukas Lerche, Dietmar Jannach, and Malte Ludewig. 2016. On the value of reminders within e-commerce recommendations. In Proceedings of the ACM Conference on User Modeling, Adaptation, and Personalization (UMAP’16). 27--35.

Digital Library

[65]

Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. 2011. Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In Proceedings of the 4th ACM International Conference on Web Search and Data Mining (WSDM’11). 297--306.

Digital Library

[66]

Jimmy Lin. 2019. The neural hype and comparisons against weak baselines. SIGIR Forum 52, 2 (Jan. 2019), 40--51.

Digital Library

[67]

Z. C. Lipton and J. Steinhardt. 2018. Troubling trends in machine learning scholarship. Retrieved from https://arxiv.org/abs/1807.03341.

[68]

Jiahui Liu, Peter Dolan, and Elin Rønby Pedersen. 2010. Personalized news recommendation based on click behavior. In Proceedings of the 15th International Conference on Intelligent User Interfaces (IUI’10). 31--40.

Digital Library

[69]

Malte Ludewig and Dietmar Jannach. 2018. Evaluation of session-based recommendation algorithms. User-Model. User-Adapt. Interact. 28, 4--5 (2018), 331--390.

Digital Library

[70]

Malte Ludewig and Dietmar Jannach. 2019. User-centric evaluation of session-based recommendations for an automated radio station. In Proceedings of the ACM Conference on Recommender Systems (RecSys’19).

Digital Library

[71]

Andrii Maksai, Florent Garcin, and Boi Faltings. 2015. Predicting online performance of news recommender systems through richer evaluation metrics. In Proceedings of the 9th ACM Conference on Recommender Systems (RecSys’15). 179--186.

Digital Library

[72]

Vicki McKinney, Yoon Kanghyun, and Fatemeh “Mariam” Zahedi. 2002. The measurement of web-customer satisfaction: An expectation and disconfirmation approach. Info. Syst. Res. 13, 3 (2002), 296--315.

[73]

Sean M. McNee, Istvan Albert, Dan Cosley, Prateep Gopalkrishnan, Shyong K. Lam, Al Mamunur Rashid, Joseph A. Konstan, and John Riedl. 2002. On the recommending of citations for research papers. In Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW’02). 116--125.

Digital Library

[74]

Sean M. McNee, John Riedl, and Joseph A. Konstan. 2006. Being accurate is not enough: How accuracy metrics have hurt recommender systems. In CHI’06 Extended Abstracts on Human Factors in Computing Systems (CHI EA’06). 1097--1101.

Digital Library

[75]

Ajinkya More, Linas Baltrunas, Nikos Vlassis, and Justin Basilico. 2019. Recap: Designing a more efficient estimator for off-policy evaluation in bandits with large action spaces. In Proceedings of the ACM RecSys Workshop on Reinforcement and Robust Estimators for Recommendation (REVEAL’19).

[76]

Mehrbakhsh Nilashi, Dietmar Jannach, Othman bin Ibrahim, Mohammad Dalvi Esfahani, and Hossein Ahmadi. 2016. Recommendation quality, transparency, and website quality for trust-building in recommendation agents. Electr. Comm. Res. Appl. 19 (2016), 70--84.

Digital Library

[77]

Ingrid Nunes and Dietmar Jannach. 2017. A systematic review and taxonomy of explanations in decision support and recommender systems. User-Model. User-Adapt. Interact. 27, 3--5 (2017), 393--444.

Digital Library

[78]

Lijing Qin, Shouyuan Chen, and Xiaoyan Zhu. 2014. Contextual combinatorial bandit and its application on diversified online recommendation. In Proceedings of the SIAM International Conference on Data Mining (SDM’14). 461--469.

[79]

Massimo Quadrana, Paolo Cremonesi, and Dietmar Jannach. 2018. Sequence-aware recommender systems. Comput. Surveys 54 (2018), 1--36. Issue 1.

Digital Library

[80]

Filip Radlinski, Robert Kleinberg, and Thorsten Joachims. 2008. Learning diverse rankings with multi-armed bandits. In Proceedings of the 25th International Conference on Machine Learning (ICML’08). 784--791.

Digital Library

[81]

Mahmuda Rahman and Jae C. Oh. 2018. Graph bandit for diverse user coverage in online recommendation. Appl. Intell. 48, 8 (2018), 1979--1995.

Digital Library

[82]

Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI’09). 452--461.

Digital Library

[83]

Mario Rodriguez, Christian Posse, and Ethan Zhang. 2012. Multiple objective optimization in recommender systems. In Proceedings of the 6th ACM Conference on Recommender Systems (RecSys’12). 11--18.

Digital Library

[84]

Marco Rossetti, Fabio Stella, and Markus Zanker. 2016. Contrasting offline and online results when evaluating recommendation algorithms. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys’16). 31--34.

Digital Library

[85]

Alan Said, Domonkos Tikk, Yue Shi, Martha Larson, Klara Stumpf, and Paolo Cremonesi. 2012. Recommender systems evaluation: A 3D benchmark. In Proceedings of the Workshop on Recommendation Utility Evaluation: Beyond RMSE (RUE’12). 21--23.

[86]

Guy Shani, David Heckerman, and Ronen I. Brafman. 2005. An MDP-based recommender system. J. Mach. Learn. Res. 6 (2005), 1265--1295.

Digital Library

[87]

Dong-Hee Shin and Won-Young Kim. 2008. Applying the technology acceptance model and flow theory to cyworld user behavior: Implication of the Web2.0 user acceptance. CyberPsychol. Behav. 11, 3 (2008), 378--382.

[88]

Barry Smyth, Paul Cotter, and Stephen Oman. 2007. Enabling intelligent content discovery on the mobile Internet. In Proceedings of the 22nd AAAI Conference on Artificial Intelligence (AAAI’07). 1744--1751.

[89]

Ellen Spertus, Mehran Sahami, and Orkut Buyukkokten. 2005. Evaluating similarity measures: A large-scale study in the Orkut social network. In Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery in Data Mining (KDD’05). 678--684.

Digital Library

[90]

Harald Steck. 2010. Training and testing of recommender systems on data missing not at random. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’10). 713--722.

Digital Library

[91]

Idan Szpektor, Yoelle Maarek, and Dan Pelleg. 2013. When relevance is not enough: Promoting diversity and freshness in personalized question recommendation. In Proceedings of the 22nd International Conference on World Wide Web (WWW’13). 1249--1260.

Digital Library

[92]

Kar Yan Tam and Shuk Ying Ho. 2005. Web personalization as a persuasion strategy: An elaboration likelihood model perspective. Info. Syst. Res. 16, 3 (2005), 271--291.

Digital Library

[93]

Nava Tintarev and Judith Masthoff. 2012. Evaluating the effectiveness of explanations for recommender systems. User Model. User-Adapt. Interact. 22, 4 (2012), 399--439.

Digital Library

[94]

Saúl Vargas and Pablo Castells. 2011. Rank and relevance in novelty and diversity metrics for recommender systems. In Proceedings of the 5th ACM Conference on Recommender Systems (RecSys’11). 109--116.

Digital Library

[95]

Kiri Wagstaff. 2012. Machine learning that matters. In Proceedings of the 29th International Conference on Machine Learning (ICML’12). 529--536.

[96]

Wayne Wobcke, Alfred Krzywicki, Yang Sok, Xiongcai Cai, Michael Bain, Paul Compton, and Ashesh Mahidadia. 2015. A deployed people-to-people recommender system in online dating. AI Mag. 36, 3 (2015), 5--18.

[97]

Bo Xiao and Izak Benbasat. 2007. E-commerce product recommendation agents: Use, characteristics, and impact. MIS Quart. 31, 1 (2007), 137--209.

Digital Library

[98]

Ye Xu, Zang Li, Abhishek Gupta, Ahmet Bugdayci, and Anmol Bhasin. 2014. Modeling professional similarity by mining professional career trajectories. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’14). 1945--1954.

Digital Library

[99]

Jeonghee Yi, Ye Chen, Jie Li, Swaraj Sett, and Tak W. Yan. 2013. Predictive model performance: Offline and online evaluations. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’13). 1294--1302.

[100]

Kyung-Hyan Yoo, Ulrike Gretzel, and Markus Zanker. 2012. Persuasive Recommender Systems: Conceptual Background and Implications. Springer.

[101]

Kyung-Hyan Yoo, Ulrike Gretzel, and Markus Zanker. 2013. Persuasive Recommender Systems—Conceptual Background and Implications. Springer.

[102]

Markus Zanker, Marcel Bricman, Sergiu Gordea, Dietmar Jannach, and Markus Jessenitschnig. 2006. Persuasive online-selling in quality and taste domains. In Proceedings of the 7th International Conference on E-Commerce and Web Technologies (EC-Web’06). 51--60.

Digital Library

[103]

Markus Zanker, Matthias Fuchs, Wolfram Höpken, Mario Tuta, and Nina Müller. 2008. Evaluating recommender systems in tourism—A case study from Austria. In Proceedings of the International Conference on Information and Communication Technologies in Tourism (ENTER’08), Peter O’Connor, Wolfram Höpken, and Ulrike Gretzel (Eds.). 24--34.

[104]

Mi Zhang and Neil Hurley. 2008. Avoiding monotony: Improving the diversity of recommendation lists. In Proceedings of the ACM Conference on Recommender Systems (RecSys’08). 123--130.

Digital Library

[105]

Hua Zheng, Dong Wang, Qi Zhang, Hang Li, and Tinghao Yang. 2010. Do clicks measure recommendation relevancy?: An empirical user study. In Proceedings of the 4th ACM Conference on Recommender Systems (RecSys’10). 249--252.

Digital Library

[106]

Tao Zhou, Zoltán Kuscsik, Jian-Guo Liu, Matúš Medo, Joseph Rushton Wakeling, and Yi-Cheng Zhang. 2010. Solving the apparent diversity-accuracy dilemma of recommender systems. Proc. Natl. Acad. Sci. U.S.A. 107, 10 (2010), 4511--4515.

[107]

Cai-Nicolas Ziegler, Sean M. McNee, Joseph A. Konstan, and Georg Lausen. 2005. Improving recommendation lists through topic diversification. In Proceedings of the 14th International Conference on World Wide Web (WWW’05). 22--32.

Digital Library

Cited By

Mansoorizadeh M(2024)Development of a recommender system for technical and professional training courses to skill training applicants, job seekers and skill learnersQuarterly Journal of Skill Training10.61186/irtvto.12.46.23512:46(235-252)Online publication date: 1-Mar-2024
https://doi.org/10.61186/irtvto.12.46.235
Goli ALambrecht AYoganarasimhan H(2024)A Bias Correction Approach for Interference in Ranking ExperimentsMarketing Science10.1287/mksc.2022.004643:3(590-614)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.1287/mksc.2022.0046
Bi XYang MAdomavicius G(2024)Consumer Acquisition for Recommender SystemsInformation Systems Research10.1287/isre.2023.122935:1(339-362)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1287/isre.2023.1229
Show More Cited By

Index Terms

Measuring the Business Value of Recommender Systems
1. General and reference
  1. Cross-computing tools and techniques
    1. Evaluation
  2. Document types
    1. Surveys and overviews
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems
  2. World Wide Web
    1. Web applications
      1. Electronic commerce

Recommendations

Tagcloud-based explanation with feedback for recommender systems
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

Personalized recommender systems aim to push only the relevant items and information directly to the users without requiring them to browse through millions of web resources. The challenge of these systems is to achieve a high user acceptance rate on ...
A systematic review of value-aware recommender systems
Abstract
Research on recommender systems (RSs) has traditionally focused on the design of systems capable of suggesting items of interest for users. However, often the most important expectation for RSs used in commercial applications is to improve the ...
Highlights
- A comprehensive systematic review specialized on value-aware recommender systems.
- Algorithms, commercial applications and value categories are explored in depth.
- Limitations of the state of the art and future research directions ...
A framework for tag-aware recommender systems

In social tagging system, a user annotates a tag to an item. The tagging information is utilized in recommendation process. In this paper, we propose a hybrid item recommendation method to mitigate limitations of existing approaches and propose a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Management Information Systems

ACM Transactions on Management Information Systems Volume 10, Issue 4

December 2019

98 pages

ISSN:2158-656X

EISSN:2158-6578

DOI:10.1145/3374918

Editor:
Daniel Zeng
Chinese Academy of Sciences, China

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 December 2019

Accepted: 01 October 2019

Revised: 01 September 2019

Received: 01 December 2018

Published in TMIS Volume 10, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Opinion
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

124
Total Citations
View Citations
1,873
Total Downloads

Downloads (Last 12 months)389
Downloads (Last 6 weeks)28

Reflects downloads up to 13 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Mansoorizadeh M(2024)Development of a recommender system for technical and professional training courses to skill training applicants, job seekers and skill learnersQuarterly Journal of Skill Training10.61186/irtvto.12.46.23512:46(235-252)Online publication date: 1-Mar-2024
https://doi.org/10.61186/irtvto.12.46.235
Goli ALambrecht AYoganarasimhan H(2024)A Bias Correction Approach for Interference in Ranking ExperimentsMarketing Science10.1287/mksc.2022.004643:3(590-614)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.1287/mksc.2022.0046
Bi XYang MAdomavicius G(2024)Consumer Acquisition for Recommender SystemsInformation Systems Research10.1287/isre.2023.122935:1(339-362)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1287/isre.2023.1229
Yuan ZChen AWang YSun T(2024)How Recommendation Affects Customer Search: A Field ExperimentInformation Systems Research10.1287/isre.2022.0294Online publication date: 4-Mar-2024
https://doi.org/10.1287/isre.2022.0294
Ge YLiu SFu ZTan JLi ZXu SLi YXian YZhang Y(2024)A Survey on Trustworthy Recommender SystemsACM Transactions on Recommender Systems10.1145/3652891Online publication date: 13-Apr-2024
https://doi.org/10.1145/3652891
Vrijenhoek SDaniil SSandel JHollink L(2024)Diversity of What? On the Different Conceptualizations of Diversity in Recommender SystemsProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658926(573-584)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658926
Rajapakse DLeith D(2024)User Cold-Start Learning in Recommender Systems using Monte Carlo Tree SearchACM Transactions on Recommender Systems10.1145/36180023:1(1-23)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3618002
Kweon WKang SJang SYu HChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Top-Personalized-K RecommendationProceedings of the ACM on Web Conference 202410.1145/3589334.3645417(3388-3399)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645417
Chronis CVarlamis IHimeur YSayed AAL-Hasan TNhlabatsi ABensaali FDimitrakopoulos G(2024)A Survey on the use of Federated Learning in Privacy-Preserving Recommender SystemsIEEE Open Journal of the Computer Society10.1109/OJCS.2024.33963445(227-247)Online publication date: 2024
https://doi.org/10.1109/OJCS.2024.3396344
Sinclair-Desgagné B(2024)On the (non-) reliance on algorithms—A decision-theoretic accountJournal of Mathematical Psychology10.1016/j.jmp.2024.102844119(102844)Online publication date: Apr-2024
https://doi.org/10.1016/j.jmp.2024.102844
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents