research-article

EliMRec: Eliminating Single-modal Bias in Multimedia Recommendation

Authors:

Xianglin HuangAuthors Info & Claims

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

Pages 687 - 695

https://doi.org/10.1145/3503161.3548404

Published: 10 October 2022 Publication History

Abstract

The main idea of multimedia recommendation is to introduce the profile content of multimedia documents as an auxiliary, so as to endow recommenders with generalization ability and gain better performance. However, recent studies using non-uniform datasets roughly fuse single-modal features into multi-modal features and adopt the strategy of directly maximizing the likelihood of user preference scores, leading to the single-modal bias. Owing to the defect in architecture, there is still room for improvement for recent multimedia recommendation.

In this paper, we propose EliMRec, a generic and modal-agnostic framework to eliminate the single-modal bias in multimedia recommendation. From our observation, biased predictive reasoning is influenced directly by the single modality rather than considering the all given multiple views of the item. Through the novel perspective of causal inference, we manage to explain the single-modal issue and exploit the inner working of multi-modal fusion. To eliminate single-modal bias, we enhance the bias-capture ability of a general multimedia recommendation framework and imagine several counterfactual worlds that control one modality variant with other modality fixed or blank. Truth to be told, counterfactual analysis enables us to identify and eliminate bias lying in the direct effect from single-modal features to the preference score. Extensive experiments on real-world datasets demonstrate that our method significantly improves over several state-of-the-art baselines like LightGCN and MMGCN. Codes are available at https://github.com/Xiaohao-Liu/EliMRec.

References

[1]

Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A Simple but Tough-to-Beat Baseline for Sentence Embeddings. In ICLR 2017.

[2]

Remi Cadene, Corentin Dancette, Hedi Ben-younes, Matthieu Cord, and Devi Parikh. 2020. RUBi: Reducing Unimodal Biases in Visual Question Answering. arXiv:1906.10169 [cs] (2020).

[3]

Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. 2020. Bias and Debias in Recommender System: A Survey and Future Directions. (2020).

[4]

Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nie, Wei Liu, and Tat-Seng Chua. 2017. Attentive Collaborative Filtering: Multimedia Recommendation with Item- and Component-Level Attention. In SIGIR. 335--344.

[5]

Konstantina Christakopoulou, Madeleine Traverse, Trevor Potter, Emma Marriott, Daniel Li, Chris Haulk, Ed H. Chi, and Minmin Chen. 2020. Deconfounding User Satisfaction Estimation from Response Rate Bias .Association for Computing Machinery, 450--455.

[6]

Dan Geiger, Thomas Verma, and Judea Pearl. 1990. d-separation: From theorems to algorithms. In Machine Intelligence and Pattern Recognition. Vol. 10. Elsevier, 139--148.

[7]

Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Chia Laguna Resort, Sardinia, Italy, May 13--15, 2010 (JMLR Proceedings, Vol. 9). JMLR.org, 249--256.

[8]

Madelyn Glymour, Judea Pearl, and Nicholas P Jewell. 2016. Causal inference in statistics: A primer .John Wiley & Sons.

[9]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27--30, 2016. IEEE Computer Society, 770--778.

[10]

Ruining He and Julian J. McAuley. 2016. VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback. In AAAI. 144--150.

[11]

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yong-Dong Zhang, and Meng Wang. 2020. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In Proceedings of the 43rd International ACM conference on research and development in Information Retrieval, Virtual Event, China, July 25--30, 2020. ACM, 639--648.

Digital Library

[12]

Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural Collaborative Filtering. In Proceedings of the 26th International Conference on World Wide Web, Perth, Australia, April 3--7, 2017. ACM, 173--182.

Digital Library

[13]

Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, and Kevin W. Wilson. 2017. CNN architectures for large-scale audio classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, 2017. 131--135.

[14]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, San Diego, CA, USA, May 7--9, 2015, Conference Track Proceedings.

[15]

Yehuda Koren. 2008. Factorization meets the neighborhood: a multifaceted collaborative filtering model. In Proceedings of the 14th ACM International Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, August 24--27, 2008, Ying Li, Bing Liu, and Sunita Sarawagi (Eds.). ACM, 426--434.

Digital Library

[16]

Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat seng Chua. 2022. Invariant Grounding for Video Question Answering. In CVPR.

[17]

Dugang Liu, Pengxiang Cheng, Zhenhua Dong, Xiuqiang He, Weike Pan, and Zhong Ming. 2020. A general knowledge distillation framework for counterfactual recommendation via uniform data. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 831--840.

Digital Library

[18]

Fan Liu, Zhiyong Cheng, Changchang Sun, Yinglong Wang, Liqiang Nie, and Mohan S. Kankanhalli. 2019. User Diverse Preference Modeling by Multimodal Attentive Metric Learning. In Proceedings of the 27th ACM International Conference on Multimedia, MM 2019. ACM, 1526--1534.

[19]

Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. 2021. Counterfactual VQA: A Cause-Effect Look at Language Bias. arXiv:2006.04315 [cs] (2021).

[20]

Judea Pearl. 2009. Causality. Cambridge university press.

[21]

Judea Pearl. 2010. Causal inference. Causality: Objectives and Assessment (2010), 39--58.

[22]

Judea Pearl. 2013. Direct and indirect effects. arXiv preprint arXiv:1301.2300 (2013).

[23]

Judea Pearl and Dana Mackenzie. 2018. The book of why: the new science of cause and effect .Basic books.

[24]

Zhen Qin, Suming J Chen, Donald Metzler, Yongwoo Noh, Jingzheng Qin, and Xuanhui Wang. 2020. Attribute-based propensity for unbiased learning in recommender systems: Algorithm and case studies. In SIGKDD.

[25]

Steffen Rendle. 2010. Factorization Machines. In The 10th IEEE International Conference on Data Mining, Sydney, Australia, 14--17 December 2010, Geoffrey I. Webb, Bing Liu, Chengqi Zhang, Dimitrios Gunopulos, and Xindong Wu (Eds.). IEEE Computer Society, 995--1000.

[26]

Steen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian Personalized Ranking from Implicit Feedback. (2009), 10.

Digital Library

[27]

Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as Treatments: Debiasing Learning and Evaluation. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19--24, 2016 (JMLR Workshop and Conference Proceedings, Vol. 48). JMLR.org, 1670--1679.

[28]

Juntao Tan, Shuyuan Xu, Yingqiang Ge, Yunqi Li, Xu Chen, and Yongfeng Zhang. 2021. Counterfactual Explainable Recommendation. 1784--1793.

[29]

Kaihua Tang, Mingyuan Tao, and Hanwang Zhang. 2021. Adversarial Visual Robustness by Causal Intervention. arXiv preprint arXiv:2106.09534 (2021).

[30]

Zhulin Tao, Xiaohao Liu, Yewei Xia, Xiang Wang, Lifang Yang, Xianglin Huang, and Tat-Seng Chua. 2022. Self-supervised Learning for Multimedia Recommendation. IEEE Transactions on Multimedia (2022). https://doi.org/10.1109/TMM.2022.3187556

Digital Library

[31]

Zhulin Tao, Yinwei Wei, Xiang Wang, Xiangnan He, Xianglin Huang, and Tat-Seng Chua. 2020. MGAT: Multimodal Graph Attention Network for Recommendation. Inf. Process. Manag., Vol. 57, 5 (2020), 102277.

[32]

Wenjie Wang, Fuli Feng, Xiangnan He, Xiang Wang, and Tat-Seng Chua. 2021a. Deconfounded Recommendation for Alleviating Bias Amplification. SIGKDD (2021), 1717--1725.

[33]

Wenjie Wang, Fuli Feng, Xiangnan He, Hanwang Zhang, and Tat-Seng Chua. 2021b. Clicks can be Cheating: Counterfactual Recommendation for Mitigating Clickbait Issue. SIGIR (2021), 1288--1297.

[34]

Xiang Wang, Xiangnan He, Liqiang Nie, and Tat-Seng Chua. 2017. Item Silk Road: Recommending Items from Information Domains to Social Users. In Proceedings of the 40th International ACM Conference on Research and Development in Information Retrieval, August 7--11, 2017. ACM, 185--194.

Digital Library

[35]

Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019. Neural Graph Collaborative Filtering. In Proceedings of the 42nd International ACM Conference on Research and Development in Information Retrieval, Paris, France, July 21--25, 2019. ACM, 165--174.

Digital Library

[36]

Xiang Wang, Yingxin Wu, An Zhang, Fuli Feng, Xiangnan He, and Tat-Seng Chua. 2022. Reinforced Causal Explainer for Graph Neural Networks. TPAMI (2022).

[37]

Xiang Wang, Yingxin Wu, An Zhang, Xiangnan He, and Tat seng Chua. 2021c. Towards Multi-Grained Explainability for Graph Neural Networks. In NeurIPS.

[38]

Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. 2021d. Combating Selection Biases in Recommender Systems with a Few Unbiased Ratings. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 427--435.

Digital Library

[39]

Tianxin Wei, Fuli Feng, Jiawei Chen, Ziwei Wu, Jinfeng Yi, and Xiangnan He. 2021a. Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System. ACM SIGKDD (2021), 1791--1800.

[40]

Yinwei Wei, Xiang Wang, Qi Li, Liqiang Nie, Yan Li, Xuanping Li, and Tat-Seng Chua. 2021b. Contrastive learning for cold-start recommendation. In Proceedings of the 29th ACM International Conference on Multimedia. 5382--5390.

Digital Library

[41]

Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, and Tat-Seng Chua. 2020. Graph-refined convolutional network for multimedia recommendation with implicit feedback. In Proceedings of the 28th ACM international conference on multimedia. 3541--3549.

Digital Library

[42]

Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. 2019. MMGCN: Multi-modal Graph Convolution Network for Personalized Recommendation of Micro-video. In Proceedings of the 27th ACM International Conference on Multimedia. ACM, 1437--1445.

Digital Library

[43]

Le Wu, Xiangnan He, Xiang Wang, Kun Zhang, and Meng Wang. 2021. A Survey on Neural Recommendation: From Collaborative Filtering to Information-rich Recommendation. arXiv:2104.13030 [cs] (Oct. 2021).

[44]

Yingxin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat seng Chua. 2022. Discovering Invariant Rationales for Graph Neural Networks. In ICLR.

[45]

Xu Yang, Hanwang Zhang, Guojun Qi, and Jianfei Cai. 2021. Causal Attention for Vision-Language Tasks. arXiv:2103.03493 [cs] (2021).

[46]

Jingwei Yi, Fangzhao Wu, Chuhan Wu, Qifei Li, Guangzhong Sun, and Xing Xie. 2021. DebiasedRec: Bias-aware User Modeling and Click Prediction for Personalized News Recommendation. arXiv:2104.07360 [cs] (2021).

[47]

Zhongqi Yue, Tan Wang, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua. 2021. Counterfactual Zero-Shot and Open-Set Visual Recognition. arXiv:2103.00887 [cs] (2021).

[48]

Yongfeng Zhang, Qingyao Ai, Xu Chen, and W. Bruce Croft. 2017. Joint Representation Learning for Top-N Recommendation with Heterogeneous Information Sources. In CIKM 2017. ACM, 1449--1458.

[49]

Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. 2021. Causal Intervention for Leveraging Popularity Bias in Recommendation. SIGIR (2021), 11--20.

[50]

Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2020. Graph neural networks: A review of methods and applications. AI Open, Vol. 1 (2020), 57--81.

Cited By

Wang WBai HHuang JWan YYuan YQiu HPeng NLyu MCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)New Job, New Gender? Measuring the Social Bias in Image Generation ModelsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681433(3781-3789)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681433
Zhang TMin WLiu TJiang SRui Y(2024)Toward Egocentric Compositional Action Anticipation with Adaptive Semantic DebiasingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363333320:5(1-21)Online publication date: 11-Jan-2024
https://dl.acm.org/doi/10.1145/3633333
Shang YGao CChen JJin DLi YChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Improving Item-side Fairness of Multimodal Recommendation via Modality DebiasingProceedings of the ACM Web Conference 202410.1145/3589334.3648156(4697-4705)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3648156
Show More Cited By

Index Terms

EliMRec: Eliminating Single-modal Bias in Multimedia Recommendation
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems

Recommendations

Learning Hybrid Behavior Patterns for Multimedia Recommendation
MM '22: Proceedings of the 30th ACM International Conference on Multimedia

Multimedia recommendation aims to predict user preferences where users interact with multimodal items. Collaborative filtering based on graph convolutional networks manifests impressive performance gains in multimedia recommendation. This is attributed ...
A Multimodal Single-Branch Embedding Network for Recommendation in Cold-Start and Missing Modality Scenarios
RecSys '24: Proceedings of the 18th ACM Conference on Recommender Systems

Most recommender systems adopt collaborative filtering (CF) and provide recommendations based on past collective interactions. Therefore, the performance of CF algorithms degrades when few or no interactions are available, a scenario referred to as cold-...
Multi-View Graph Convolutional Network for Multimedia Recommendation
MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Multimedia recommendation has received much attention in recent years. It models user preferences based on both behavior information and item multimodal information. Though current GCN-based methods achieve notable success, they suffer from two ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

October 2022

7537 pages

ISBN:9781450392037

DOI:10.1145/3503161

General Chairs:
João Magalhães
NOVA University of Lisbon, Portugal
,
Alberto del Bimbo
University of Florence, Italy
,
Shin'ichi Satoh
National Institute of Informatics, Japan
,
Nicu Sebe
University of Trento, Italy
,
Program Chairs:
Xavier Alameda-Pineda
Inria, Grenoble, France
,
Qin Jin
Renmin University of China, China
,
Vincent Oria
New Jersey Institute of Technology, USA
,
Laura Toni
University College London, UK

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

the National Key Research and Development Program of China
the Fundamental Research Funds for the Central Universities
the GHfund B

Conference

MM '22

Sponsor:

SIGMM

MM '22: The 30th ACM International Conference on Multimedia

October 10 - 14, 2022

Lisboa, Portugal

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
477
Total Downloads

Downloads (Last 12 months)169
Downloads (Last 6 weeks)29

Reflects downloads up to 28 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang WBai HHuang JWan YYuan YQiu HPeng NLyu MCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)New Job, New Gender? Measuring the Social Bias in Image Generation ModelsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681433(3781-3789)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681433
Zhang TMin WLiu TJiang SRui Y(2024)Toward Egocentric Compositional Action Anticipation with Adaptive Semantic DebiasingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363333320:5(1-21)Online publication date: 11-Jan-2024
https://dl.acm.org/doi/10.1145/3633333
Shang YGao CChen JJin DLi YChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Improving Item-side Fairness of Multimodal Recommendation via Modality DebiasingProceedings of the ACM Web Conference 202410.1145/3589334.3648156(4697-4705)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3648156
Li LQin YJi WZhou YZimmermann R(2024)Domain-Wise Invariant Learning for Panoptic Scene Graph GenerationICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10447193(3165-3169)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10447193
Liu XZeng HZhang WYang L(2024)Collaborative Denoising Shilling Attack for Recommendation Systems2024 27th International Conference on Computer Supported Cooperative Work in Design (CSCWD)10.1109/CSCWD61410.2024.10580115(1424-1429)Online publication date: 8-May-2024
https://doi.org/10.1109/CSCWD61410.2024.10580115
Malitesta DCornacchia GPomo CDi Noia TJi WWei YZheng ZFei HChua T(2023)On Popularity Bias of Multimodal-aware Recommender Systems: A Modalities-driven AnalysisProceedings of the 1st International Workshop on Deep Multimodal Learning for Information Retrieval10.1145/3606040.3617441(59-68)Online publication date: 29-Oct-2023
https://doi.org/10.1145/3606040.3617441
Huang SLi HLi QZheng CLiu LEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Pareto Invariant Representation Learning for Multimedia RecommendationProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612591(6410-6419)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612591
Shang YGao CChen JJin DMa HLi YEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Enhancing Adversarial Robustness of Multi-modal Recommendation via Modality BalancingProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612337(6274-6282)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612337

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents