Lower-Left Partial AUC: An Effective and Efficient Optimization Metric for Recommendation
Authors: 
Wentao Shi, Chenxu Wang, Fuli Feng, Yang Zhang, Wenjie Wang, Junkang Wu, 
Xiangnan HeAuthors Info & Claims
Wentao Shi, Chenxu Wang, Fuli Feng, Yang Zhang, Wenjie Wang, Junkang Wu, Xiangnan HeAuthors Info & ClaimsPages 3253 - 3264
Abstract
Optimization metrics are crucial for building recommendation systems at scale. However, an effective and efficient metric for practical use remains elusive. While Top-K ranking metrics are the gold standard for optimization, they suffer from significant computational overhead. Alternatively, the more efficient accuracy and AUC metrics often fall short of capturing the true targets of recommendation tasks, leading to suboptimal performance. To overcome this dilemma, we propose a new optimization metric, Lower-Left Partial AUC (LLPAUC), which is computationally efficient like AUC but strongly correlates with Top-K ranking metrics. Compared to AUC, LLPAUC considers only the partial area under the ROC curve in the Lower-Left corner to push the optimization focus on Top-K. We provide theoretical validation of the correlation between LLPAUC and Top-K ranking metrics and demonstrate its robustness to noisy user feedback. We further design an efficient point-wise recommendation loss to maximize LLPAUC and evaluate it on three datasets, validating its effectiveness and robustness.
Supplemental Material
MP4 File
Supplemental video
- Download
- 21.16 MB
References
[1]
Stephen Boyd and Lieven Vandenberghe. 2004. Convex Optimization. Cambridge University Press.
[2]
Andrew P Bradley. 1997. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern recognition, Vol. 30, 7 (1997), 1145--1159.
[3]
Amar Budhiraja, Gaurush Hiranandani, Navya Yarrabelly, Ayush Choure, Oluwasanmi Koyejo, and Prateek Jain. 2020. Rich-Item Recommendations for Rich-Users via GCNN: Exploiting Dynamic and Static Side Information. CoRR, Vol. abs/2001.10495 (2020).
[4]
Christopher J. C. Burges, Robert Ragno, and Quoc Viet Le. 2006. Learning to Rank with Nonsmooth Cost Functions. In NIPS. MIT Press, 193--200.
[5]
Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. 2020. Bias and Debias in Recommender System: A Survey and Future Directions. CoRR, Vol. abs/2010.03240 (2020).
[6]
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In RecSys. ACM, 191--198.
[7]
Lori E. Dodd and Margaret S. Pepe. 2003. Partial AUC Estimation and Regression. Biometrics, Vol. 59, 3 (2003), 614--623.
[8]
Yanbo Fan, Siwei Lyu, Yiming Ying, and Bao-Gang Hu. 2017. Learning with Average Top-k Loss. In NIPS. 497--505.
[9]
Wei Gao and Zhi-Hua Zhou. 2015. On the Consistency of AUC Pairwise Optimization. In IJCAI. AAAI Press, 939--945.
[10]
Yuan Gao, Xiang Wang, Xiangnan He, Huamin Feng, and Yong-Dong Zhang. 2023 a. Rumor detection with self-supervised learning on texts and social graph. Frontiers Comput. Sci., Vol. 17, 4 (2023), 174611.
[11]
Yuan Gao, Xiang Wang, Xiangnan He, Zhenguang Liu, Huamin Feng, and Yongdong Zhang. 2023 b. Addressing Heterophily in Graph Anomaly Detection: A Perspective of Graph Spectrum. In WWW. ACM, 1528--1538.
[12]
Yuan Gao, Xiang Wang, Xiangnan He, Zhenguang Liu, Huamin Feng, and Yongdong Zhang. 2023 c. Alleviating Structural Distribution Shift in Graph Anomaly Detection. In WSDM. ACM, 357--365.
[13]
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep Sparse Rectifier Neural Networks. In AISTATS (JMLR Proceedings, Vol. 15). JMLR.org, 315--323.
[14]
Jon Atle Gulla, Lemei Zhang, Peng Liu, Ö zlem Ö zgö bek, and Xiaomeng Su. 2017. The Adressa dataset for news recommendation. In WI. ACM, 1042--1048.
[15]
J.A. Hanley and Barbara Mcneil. 1982. The Meaning and Use of the Area Under a Receiver Operating Characteristic (ROC) Curve. Radiology, Vol. 143 (05 1982), 29--36. https://doi.org/10.1148/radiology.143.1.7063747
[16]
Ruining He and Julian J. McAuley. 2016. Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering. In WWW. ACM, 507--517.
[17]
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yong-Dong Zhang, and Meng Wang. 2020. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In SIGIR. ACM, 639--648.
[18]
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural Collaborative Filtering. In WWW. ACM, 173--182.
[19]
Gaurush Hiranandani, Warut Vijitbenjaronk, Sanmi Koyejo, and Prateek Jain. 2020. Optimization and Analysis of the pAp@k Metric for Recommender Systems. In ICML (Proceedings of Machine Learning Research, Vol. 119). PMLR, 4260--4270.
[20]
Abhishek Kumar, Harikrishna Narasimhan, and Andrew Cotter. 2021. Implicit rate-constrained optimization of non-decomposable objectives. In ICML (Proceedings of Machine Learning Research, Vol. 139). PMLR, 5861--5871.
[21]
Kelong Mao, Jieming Zhu, Jinpeng Wang, Quanyu Dai, Zhenhua Dong, Xi Xiao, and Xiuqiang He. 2021. SimpleX: A Simple and Strong Baseline for Collaborative Filtering. In CIKM. ACM, 1243--1252.
[22]
Donna Katzman McClish. 1989. Analyzing a Portion of the ROC Curve. Medical Decision Making, Vol. 9, 3 (1989), 190--195. https://doi.org/10.1177/0272989X8900900307 2668680.
[23]
Harikrishna Narasimhan and Shivani Agarwal. 2013a. A Structural SVM Based Approach for Optimizing Partial AUC. In ICML (1) (JMLR Workshop and Conference Proceedings, Vol. 28). JMLR.org, 516--524.
[24]
Harikrishna Narasimhan and Shivani Agarwal. 2013b. SVM(_pAUC )(^tight ): a new support vector method for optimizing partial AUC based on a tight convex upper bound. In KDD. ACM, 167--175.
[25]
Harikrishna Narasimhan and Shivani Agarwal. 2017. Support Vector Algorithms for Optimizing the Partial Area under the ROC Curve. Neural Computation, Vol. 29, 7 (07 2017), 1919--1963.
[26]
Hang Pan, Jiawei Chen, Fuli Feng, Wentao Shi, Junkang Wu, and Xiangnan He. 2023. Discriminative-Invariant Representation Learning for Unbiased Recommendation. In IJCAI. ijcai.org, 2270--2278.
[27]
Yash Patel, Giorgos Tolias, and Jir'i Matas. 2022. Recall@k Surrogate Loss with Large Batches and Similarity Mixup. In CVPR. IEEE, 7492--7501.
[28]
Zi-Hao Qiu, Quanqi Hu, Yongjian Zhong, Lijun Zhang, and Tianbao Yang. 2022. Large-scale Stochastic Optimization of NDCG Surrogates for Deep Learning with Provable Convergence. In ICML (Proceedings of Machine Learning Research, Vol. 162). PMLR, 18122--18152.
[29]
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian Personalized Ranking from Implicit Feedback. In UAI. AUAI Press, 452--461.
[30]
Huiyang Shao, Qianqian Xu, Zhiyong Yang, Shilong Bao, and Qingming Huang. 2022. Asymptotically Unbiased Instance-wise Regularized Partial AUC Optimization: Theory and Algorithm. In NeurIPS.
[31]
Wentao Shi, Jiawei Chen, Fuli Feng, Jizhi Zhang, Junkang Wu, Chongming Gao, and Xiangnan He. 2023. On the Theories Behind Hard Negative Sampling for Recommendation. In WWW. ACM, 812--822.
[32]
Yongxiang Tang, Wentao Bai, Guilin Li, Xialong Liu, and Yu Zhang. 2022. CROLoss: Towards a Customizable Loss for Retrieval Models in Recommender Systems. In CIKM. ACM, 1916--1924.
[33]
Wenjie Wang, Fuli Feng, Xiangnan He, Liqiang Nie, and Tat-Seng Chua. 2021. Denoising Implicit Feedback for Recommendation. In WSDM. ACM, 373--381.
[34]
Wenjie Wang, Yiyan Xu, Fuli Feng, Xinyu Lin, Xiangnan He, and Tat-Seng Chua. 2023. Diffusion Recommender Model. CoRR, Vol. abs/2304.04971 (2023).
[35]
Xuanhui Wang, Cheng Li, Nadav Golbandi, Michael Bendersky, and Marc Najork. 2018. The LambdaLoss Framework for Ranking Metric Optimization. In CIKM. ACM, 1313--1322.
[36]
Le Wu, Xiangnan He, Xiang Wang, Kun Zhang, and Meng Wang. 2023. A Survey on Accuracy-Oriented Neural Recommendation: From Collaborative Filtering to Information-Rich Recommendation. IEEE Trans. Knowl. Data Eng., Vol. 35, 5 (2023), 4425--4445.
[37]
Hanfang Yang, Kun Lu, Xiang Lyu, and Feifang Hu. 2019. Two-way partial AUC and its properties. Statistical Methods in Medical Research, Vol. 28, 1 (2019), 184--195.
[38]
Jheng-Hong Yang, Chih-Ming Chen, Chuan-Ju Wang, and Ming-Feng Tsai. 2018. HOP-rec: high-order proximity for implicit recommendation. In Proceedings of the 12th ACM conference on recommender systems. 140--144.
[39]
Zhiyong Yang, Qianqian Xu, Shilong Bao, Yuan He, Xiaochun Cao, and Qingming Huang. 2021. When All We Need is a Piece of the Pie: A Generic Framework for Optimizing Two-way Partial AUC. In ICML (Proceedings of Machine Learning Research, Vol. 139). PMLR, 11820--11829.
[40]
Zhiyong Yang, Qianqian Xu, Shilong Bao, Yuan He, Xiaochun Cao, and Qingming Huang. 2023. Optimizing Two-Way Partial AUC With an End-to-End Framework. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 45, 8 (2023), 10228--10246.
[41]
Yao Yao, Qihang Lin, and Tianbao Yang. 2022. Large-scale Optimization of Partial AUC in a Range of False Positive Rates. In NeurIPS.
[42]
Weinan Zhang, Tianqi Chen, Jun Wang, and Yong Yu. 2013. Optimizing top-n collaborative filtering via dynamic negative item sampling. In SIGIR. ACM, 785--788.
[43]
Dixian Zhu, Gang Li, Bokun Wang, Xiaodong Wu, and Tianbao Yang. 2022. When AUC meets DRO: Optimizing Partial AUC for Deep Learning with Non-Convex Convergence Guarantee. In ICML (Proceedings of Machine Learning Research, Vol. 162). PMLR, 27548--27573.
Index Terms
- Lower-Left Partial AUC: An Effective and Efficient Optimization Metric for Recommendation
Recommendations
An effective recommendation method for cold start new users using trust and distrust networks
Recommendation systems analyze the purchasing behavior (e.g., item ratings) of users to learn about their preferences and recommend products or services that may be of interest to them. However, as new users require time to become familiar with ...
Activity Recommendation with Partners
Recommending social activities, such as watching movies or having dinner, is a common function found in social networks or e-commerce sites. Besides certain websites which manage activity-related locations (e.g., foursquare.com), many items on product ...
Partial AUC maximization in a linear combination of dichotomizers
Classifier combination is a useful and common methodology to design an effective classification system. A large number of combination rules has been proposed hitherto, mostly aimed at minimizing the error rate. Recently, some methods have been presented ...
Comments
Information & Contributors
Information
Published In
May 2024
4826 pages
ISBN:9798400701719
DOI:10.1145/3589334
- General Chairs:
- Tat-Seng Chua,
- Chong-Wah Ngo,
- Proceedings Chair:
- Roy Ka-Wei Lee,
- Program Chairs:
- Ravi Kumar,
- Hady W. Lauw
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- the CCCD Key Lab of Ministry of Culture and Tourism
- the National Natural Science Foundation of China
Conference
WWW '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,899 of 8,196 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 151Total Downloads
- Downloads (Last 12 months)151
- Downloads (Last 6 weeks)7
Reflects downloads up to 23 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in