short-paper

RGRecSys: A Toolkit for Robustness Evaluation of Recommender Systems

Authors:

Shelby Heinecke,

Yongfeng Zhang,

Caiming XiongAuthors Info & Claims

WSDM '22: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining

Pages 1597 - 1600

https://doi.org/10.1145/3488560.3502192

Published: 15 February 2022 Publication History

Abstract

Robust machine learning is an increasingly important topic that focuses on developing models resilient to various forms of imperfect data. Due to the pervasiveness of recommender systems in online technologies, researchers have carried out several robustness studies focusing on data sparsity and profile injection attacks. Instead, we propose a more holistic view of robustness for recommender systems that encompasses multiple dimensions - robustness with respect to sub-populations, transformations, distributional disparity, attack, and data sparsity. While there are several libraries that allow users to compare different recommender system models, there is no software library for comprehensive robustness evaluation of recommender system models under different scenarios. As our main contribution, we present a robustness evaluation toolkit, Robustness Gym for RecSys (RGRecSys), that allows us to quickly and uniformly evaluate the robustness of recommender system models.

Supplementary Material

MP4 File (WSDM22-dp018.mp4)

Presentation video for the paper "RGRecSys: A Toolkit for Robustness Evaluation of Recommender Systems".

Download
70.70 MB

References

[1]

Fabio Aiolli. 2013. Efficient top-n recommendation for very large scale binary rated datasets. In Proceedings of the 7th ACM Conference on Recommender Systems .

Digital Library

[2]

Adam Barth, Collin Jackson, and John C Mitchell. 2008. Robust defenses for cross-site request forgery. In Proceedings of the 15th ACM conference on Computer and communications security .

Digital Library

[3]

Longbing Cao. 2016. Non-iid recommender systems: A review and framework of recommendation paradigm shifting. Engineering (2016).

[4]

Hanxiong Chen, Shaoyun Shi, Yunqi Li, and Yongfeng Zhang. 2021. Neural Collaborative Reasoning. In Proceedings of the Web Conference 2021 .

Digital Library

[5]

Michael D Ekstrand, Michael Ludwig, Joseph A Konstan, and John T Riedl. 2011. Rethinking the recommender research ecosystem: reproducibility, openness, and lenskit. In Proceedings of the fifth ACM conference on Recommender systems .

Digital Library

[6]

Karan Goel, Nazneen Fatema Rajani, Jesse Vig, Zachary Taschdjian, Mohit Bansal, and Christopher Ré. [n.d.]. Robustness Gym: Unifying the NLP Evaluation Landscape. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics .

[7]

Guibing Guo, Jie Zhang, Zhu Sun, and Neil Yorke-Smith. [n.d.]. LibRec: A Java Library for Recommender Systems. In UMAP Workshops .

[8]

F Maxwell Harper and Joseph A Konstan. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (2015).

[9]

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020. Lightgcn: Simplifying and powering graph convolution network for recommendation. In SIGIR .

Digital Library

[10]

Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural Collaborative Filtering. In WWW. ACM .

[11]

Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management .

Digital Library

[12]

Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased learning-to-rank with biased feedback. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining .

Digital Library

[13]

Joonseok Lee, Mingxuan Sun, and Guy Lebanon. 2012. Prea: Personalized recommendation algorithms toolkit. Journal of Machine Learning Research (2012).

[14]

Jingjing Li, Mengmeng Jing, Ke Lu, Lei Zhu, Yang Yang, and Zi Huang. 2019. From zero-shot learning to cold-start recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence .

Digital Library

[15]

Yunqi Li, Hanxiong Chen, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2021. User-oriented Fairness in Recommendation. In WWW .

[16]

Zohreh Ovaisi, Ragib Ahsan, Yifan Zhang, Kathryn Vasilaky, and Elena Zheleva. 2020. Correcting for selection bias in learning-to-rank systems. In Proceedings of The Web Conference .

Digital Library

[17]

Zohreh Ovaisi, Kathryn Vasilaky, and Elena Zheleva. 2021. Propensity-Independent Bias Recovery in Offline Learning-to-Rank Systems. In Proceedings of ACM SIGIR Conference on Research and Development in Information Retrieval .

Digital Library

[18]

Steffen Rendle. 2010. Factorization machines. In 2010 IEEE International conference on data mining .

Digital Library

[19]

Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007. Predicting clicks: estimating the click-through rate for new ads. In Proceedings of the 16th international conference on World Wide Web .

Digital Library

[20]

Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. 2021. Towards Out-Of-Distribution Generalization: A Survey. arXiv preprint arXiv:2108.13624 (2021).

[21]

Ashudeep Singh and Thorsten Joachims. 2018. Fairness of exposure in rankings. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining .

Digital Library

[22]

Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & cross network for ad click predictions. In Proceedings of the ADKDD'17 .

Digital Library

[23]

Jun Xiao, Hao Ye, Xiangnan He, Hanwang Zhang, Fei Wu, and Tat-Seng Chua. 2017. Attentional factorization machines: Learning the weight of feature interactions via attention networks. arXiv preprint arXiv:1708.04617 (2017).

[24]

Shijie Zhang, Hongzhi Yin, Tong Chen, Quoc Viet Nguyen Hung, Zi Huang, and Lizhen Cui. 2020. Gcn-based user representation learning for unifying robust recommendation and fraudster detection. In SIGIR .

[25]

Yongfeng Zhang and Xu Chen. 2020. Explainable recommendation: A survey and new perspectives. Foundations and Trends in Information Retrieval (2020).

[26]

Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Kaiyuan Li, Yushuo Chen, Yujie Lu, Hui Wang, Changxin Tian, Xingyu Pan, et al. 2020. RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms. arXiv preprint arXiv:2011.01731 (2020).

[27]

Lei Zheng, Chun-Ta Lu, Fei Jiang, Jiawei Zhang, and Philip S Yu. 2018. Spectral collaborative filtering. In Proceedings of ACM conference on recommender systems .

Digital Library

[28]

Sijin Zhou, Xinyi Dai, Haokun Chen, Weinan Zhang, Kan Ren, Ruiming Tang, Xiuqiang He, and Yong Yu. 2020. Interactive recommender system via knowledge graph-enhanced reinforcement learning. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval .

Digital Library

Cited By

Ma AYu YShi CGuo ZChua T(2024)Cross-view hypergraph contrastive learning for attribute-aware recommendationInformation Processing and Management: an International Journal10.1016/j.ipm.2024.10370161:4Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1016/j.ipm.2024.103701
Chizari NTajfar KMoreno-García M(2023)Bias Assessment Approaches for Addressing User-Centered Fairness in GNN-Based Recommender SystemsInformation10.3390/info1402013114:2(131)Online publication date: 17-Feb-2023
https://doi.org/10.3390/info14020131
Liu HLin HZhang XMa FChen HWang LYu HZhang XFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Boosting Meta-Learning Cold-Start Recommendation with Graph Neural NetworkProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615283(4105-4109)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615283
Show More Cited By

Index Terms

RGRecSys: A Toolkit for Robustness Evaluation of Recommender Systems
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems

Recommendations

Naïve filterbots for robust cold-start recommendations
KDD '06: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining

The goal of a recommender system is to suggest items of interest to a user based on historical behavior of a community of users. Given detailed enough history, item-based collaborative filtering (CF) often performs as well or better than almost any ...
Is trust robust?: an analysis of trust-based recommendation
IUI '06: Proceedings of the 11th international conference on Intelligent user interfaces

Systems that adapt to input from users are susceptible to attacks from those same users. Recommender systems are common targets for such attacks since there are financial, political and many other motivations for influencing the promotion or demotion of ...
Rating aggregation in collaborative filtering systems
RecSys '09: Proceedings of the third ACM conference on Recommender systems

Recommender systems based on user feedback rank items by aggregating users' ratings in order to select those that are ranked highest. Ratings are usually aggregated using a weighted arithmetic mean. However, the mean is quite sensitive to outliers and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WSDM '22: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining

February 2022

1690 pages

ISBN:9781450391320

DOI:10.1145/3488560

General Chairs:
K. Selcuk Candan
Arizona State University, USA
,
Huan Liu
Arizona State University, USA
,
Program Chairs:
Leman Akoglu
Carnegie Mellon University, USA
,
Xin Luna Dong
Meta Platforms, Inc. (former Facebook), USA
,
Jiliang Tang
Michigan State University, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 February 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

WSDM '22

Sponsor:

WSDM '22: The Fifteenth ACM International Conference on Web Search and Data Mining

February 21 - 25, 2022

AZ, Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 498 of 2,863 submissions, 17%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
231
Total Downloads

Downloads (Last 12 months)64
Downloads (Last 6 weeks)12

Reflects downloads up to 21 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ma AYu YShi CGuo ZChua T(2024)Cross-view hypergraph contrastive learning for attribute-aware recommendationInformation Processing and Management: an International Journal10.1016/j.ipm.2024.10370161:4Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1016/j.ipm.2024.103701
Chizari NTajfar KMoreno-García M(2023)Bias Assessment Approaches for Addressing User-Centered Fairness in GNN-Based Recommender SystemsInformation10.3390/info1402013114:2(131)Online publication date: 17-Feb-2023
https://doi.org/10.3390/info14020131
Liu HLin HZhang XMa FChen HWang LYu HZhang XFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Boosting Meta-Learning Cold-Start Recommendation with Graph Neural NetworkProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615283(4105-4109)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615283
Wang QWu CLian DChen E(2023)Securing recommender system via cooperative trainingWorld Wide Web10.1007/s11280-023-01214-726:6(3915-3943)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1007/s11280-023-01214-7
Rezaimehr FDadkhah C(2023)T&TRS: robust collaborative filtering recommender systems against attacksMultimedia Tools and Applications10.1007/s11042-023-16641-x83:11(31701-31731)Online publication date: 18-Sep-2023
https://doi.org/10.1007/s11042-023-16641-x
Wang QLian DWu CChen E(2022)Towards Robust Recommender Systems via Triple Cooperative DefenseWeb Information Systems Engineering – WISE 202210.1007/978-3-031-20891-1_40(564-578)Online publication date: 31-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-20891-1_40
Ge YLiu SFu ZTan JLi ZXu SLi YXian YZhang Y(undefined)A Survey on Trustworthy Recommender SystemsACM Transactions on Recommender Systems10.1145/3652891
https://dl.acm.org/doi/10.1145/3652891

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents