tutorial

Multi-Modal Recommender Systems: Hands-On Exploration

Authors:

Quoc-Tuan Truong,

Hady LauwAuthors Info & Claims

RecSys '21: Proceedings of the 15th ACM Conference on Recommender Systems

Pages 834 - 837

https://doi.org/10.1145/3460231.3473324

Published: 13 September 2021 Publication History

Abstract

Recommender systems typically learn from user-item preference data such as ratings and clicks. This information is sparse in nature, i.e., observed user-item preferences often represent less than 5% of possible interactions. One promising direction to alleviate data sparsity is to leverage auxiliary information that may encode additional clues on how users consume items. Examples of such data (referred to as modalities) are social networks, item’s descriptive text, product images. The objective of this tutorial is to offer a comprehensive review of recent advances to represent, transform and incorporate the different modalities into recommendation models. Moreover, through practical hands-on sessions, we consider cross model/modality comparisons to investigate the importance of different methods and modalities. The hands-on exercises are conducted with Cornac (https://cornac.preferred.ai ), a comparative framework for multimodal recommender systems. The materials are made available on https://preferred.ai/recsys21-tutorial/.

References

[1]

David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. JMLR 3(2003), 993–1022.

Digital Library

[2]

Allison JB Chaney, David M Blei, and Tina Eliassi-Rad. 2015. A probabilistic model for using social networks in personalized item recommendation. In RecSys. 43–50.

[3]

Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nie, Wei Liu, and Tat-Seng Chua. 2017. Attentive collaborative filtering: Multimedia recommendation with item-and component-level attention. In SIGIR. 335–344.

[4]

Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. 2019. Graph neural networks for social recommendation. In WWW. 417–426.

[5]

Ruining He and Julian McAuley. 2016. VBPR: visual bayesian personalized ranking from implicit feedback. In AAAI.

[6]

Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In WWW. 173–182.

[7]

Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative filtering for implicit feedback datasets. In ICDM. 263–272.

[8]

Wang-Cheng Kang, Chen Fang, Zhaowen Wang, and Julian McAuley. 2017. Visually-aware fashion recommendation and design with generative image models. In ICDM. 207–216.

[9]

Donghyun Kim, Chanyoung Park, Jinoh Oh, Sungyoung Lee, and Hwanjo Yu. 2016. Convolutional matrix factorization for document context-aware recommendation. In RecSys. 233–240.

[10]

Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix factorization techniques for recommender systems. Computer 42, 8 (2009), 30–37.

Digital Library

[11]

Wonsung Lee, Kyungwoo Song, and Il-Chul Moon. 2017. Augmented variational autoencoders for collaborative filtering with auxiliary information. In CIKM. 1139–1148.

[12]

Chenyi Lei, Dong Liu, Weiping Li, Zheng-Jun Zha, and Houqiang Li. 2016. Comparative deep learning of hybrid representations for image recommendations. In CVPR. 2545–2553.

[13]

Piji Li, Zihao Wang, Zhaochun Ren, Lidong Bing, and Wai Lam. 2017. Neural rating regression with abstractive tips generation for recommendation. In SIGIR.

[14]

Xiaopeng Li and James She. 2017. Collaborative variational autoencoder for recommender systems. In SIGKDD.

[15]

Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. 2018. Variational autoencoders for collaborative filtering. In WWW.

[16]

Zhongqi Lu, Zhicheng Dou, Jianxun Lian, Xing Xie, and Qiang Yang. 2015. Content-based collaborative filtering for news topic recommendation. In AAAI.

[17]

Hao Ma, Haixuan Yang, Michael R Lyu, and Irwin King. 2008. Sorec: social recommendation using probabilistic matrix factorization. In CIKM. 931–940.

[18]

Hao Ma, Dengyong Zhou, Chao Liu, Michael R Lyu, and Irwin King. 2011. Recommender systems with social regularization. In WSDM. 287–296.

[19]

Julian McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In RecSys. 165–172.

[20]

Andriy Mnih and Russ R Salakhutdinov. 2008. Probabilistic matrix factorization. In NIPS. 1257–1264.

[21]

Trong T Nguyen and Hady W Lauw. 2016. Representation learning for homophilic preferences. In Proceedings of the 10th ACM Conference on Recommender Systems. 317–324.

Digital Library

[22]

Wei Niu, James Caverlee, and Haokai Lu. 2018. Neural personalized ranking for image recommendation. In WSDM. 423–431.

[23]

Chanyoung Park, Donghyun Kim, Jinoh Oh, and Hwanjo Yu. 2017. Do” Also-Viewed” Products Help User Rating Prediction?. In WWW. 1113–1122.

[24]

Rendle S., Freudenthaler C., Gantner Z., and Schmidt-Thieme L.2012. BPR: Bayesian personalized ranking from implicit feedback. In UAI.

[25]

Aghiles Salah and Hady W Lauw. 2018. A bayesian latent variable model of user preferences with item context. IJCAI.

[26]

Aghiles Salah and Hady W Lauw. 2018. Probabilistic collaborative representation learning for personalized item recommendation. In UAI.

[27]

Aghiles Salah and Mohamed Nadif. 2017. Social regularized von Mises–Fisher mixture model for item recommendation. Data Mining and Knowledge Discovery 31, 5 (2017), 1218–1241.

Digital Library

[28]

Aghiles Salah, Quoc-Tuan Truong, and Hady W Lauw. 2020. Cornac: A Comparative Framework for Multimodal Recommender Systems.JMLR 21(2020), 95–1.

[29]

Ajit P Singh and Geoffrey J Gordon. 2008. Relational learning via collective matrix factorization. In SIGKDD. 650–658.

[30]

Quoc-Tuan Truong and Hady Lauw. 2019. Multimodal review generation for recommender systems. In WWW. 1864–1874.

[31]

Quoc-Tuan Truong, Aghiles Salah, and Hady W Lauw. 2021. Bilateral Variational Autoencoder for Collaborative Filtering. In WSDM. 292–300.

[32]

Quoc-Tuan Truong, Aghiles Salah, Thanh-Binh Tran, Jingyao Guo, and Hady W. Lauw. 2021. Exploring Cross-Modality Utilization in Recommender Systems. IEEE Internet Computing(2021).

[33]

Dashun Wang, Dino Pedreschi, Chaoming Song, Fosca Giannotti, and Albert-Laszlo Barabasi. 2011. Human mobility, social ties, and link prediction. In SIGKDD. 1100–1108.

[34]

Hao Wang, Naiyan Wang, and Dit-Yan Yeung. 2015. Collaborative deep learning for recommender systems. In SIGKDD.

[35]

Suhang Wang, Yilin Wang, Jiliang Tang, Kai Shu, Suhas Ranganath, and Huan Liu. 2017. What your images reveal: Exploiting visual contents for point-of-interest recommendation. In WWW. 391–400.

[36]

Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019. Kgat: Knowledge graph attention network for recommendation. In SIGKDD. 950–958.

Digital Library

[37]

Weilong Yao, Jing He, Hua Wang, Yanchun Zhang, and Jie Cao. 2015. Collaborative topic ranking: Leveraging item meta-data for sparsity reduction. In AAAI.

[38]

Haochao Ying, Liang Chen, Yuwen Xiong, and Jian Wu. 2016. Collaborative deep ranking: A hybrid pair-wise recommendation algorithm with implicit feedback. In PAKDD.

[39]

Fuzheng Zhang, Nicholas Jing Yuan, Defu Lian, Xing Xie, and Wei-Ying Ma. 2016. Collaborative knowledge base embedding for recommender systems. In SIGKDD. 353–362.

[40]

Shuai Zhang, Lina Yao, and Xiwei Xu. 2017. Autosvd++ an efficient hybrid collaborative filtering model via contractive auto-encoders. In SIGIR.

[41]

Yongfeng Zhang, Qingyao Ai, Xu Chen, and W Bruce Croft. 2017. Joint representation learning for top-n recommendation with heterogeneous information sources. In CIKM. 1449–1458.

[42]

Lei Zheng, Vahid Noroozi, and Philip S Yu. 2017. Joint deep modeling of users and items using reviews for recommendation. In WSDM.

Cited By

Malitesta DCornacchia GPomo CMerra FDi Noia TDi Sciascio E(2024)Formalizing Multimedia Recommendation through Multimodal Deep LearningACM Transactions on Recommender Systems10.1145/3662738Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3662738
Ong DTruong QLauw HChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Cornac-AB: An Open-Source Recommendation Framework with Native A/B Testing IntegrationCompanion Proceedings of the ACM on Web Conference 202410.1145/3589335.3651241(1027-1030)Online publication date: 13-May-2024
https://doi.org/10.1145/3589335.3651241
Yang HRang RXing LZhang LCai HGuo MSun J(2024)V-GMR: a variational autoencoder-based heterogeneous graph multi-behavior recommendation modelApplied Intelligence10.1007/s10489-024-05360-x54:4(3337-3350)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1007/s10489-024-05360-x
Show More Cited By

Multi-Modal Recommender Systems: Hands-On Exploration
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals

Recommendations

Multi-modal Knowledge Graphs for Recommender Systems
CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge Management

Recommender systems have shown great potential to solve the information explosion problem and enhance user experience in various online applications. To tackle data sparsity and cold start problems in recommender systems, researchers propose knowledge ...
An exploration of improving collaborative recommender systems via user-item subgroups
WWW '12: Proceedings of the 21st international conference on World Wide Web

Collaborative filtering (CF) is one of the most successful recommendation approaches. It typically associates a user with a group of like-minded users based on their preferences over all the items, and recommends to the user those items enjoyed by ...
Multi-Criteria Recommender Systems: A Survey and a Method to Learn New User's Profile

A Recommender System RS works much better for users when it has more information. In Collaborative Filtering, where users' preferences are expressed as ratings, the more ratings elicited, the more accurate the recommendations. New users present a big ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

RecSys '21: Proceedings of the 15th ACM Conference on Recommender Systems

September 2021

883 pages

ISBN:9781450384582

DOI:10.1145/3460231

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 September 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Tutorial
Research
Refereed limited

Funding Sources

National Research Foundation Singapore

Conference

RecSys '21

Sponsor:

RecSys '21: Fifteenth ACM Conference on Recommender Systems

September 27 - October 1, 2021

Amsterdam, Netherlands

Acceptance Rates

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Upcoming Conference

RecSys '24

Sponsor:
sigchi

18th ACM Conference on Recommender Systems

October 14 - 18, 2024

Bari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
831
Total Downloads

Downloads (Last 12 months)147
Downloads (Last 6 weeks)10

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Malitesta DCornacchia GPomo CMerra FDi Noia TDi Sciascio E(2024)Formalizing Multimedia Recommendation through Multimodal Deep LearningACM Transactions on Recommender Systems10.1145/3662738Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3662738
Ong DTruong QLauw HChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Cornac-AB: An Open-Source Recommendation Framework with Native A/B Testing IntegrationCompanion Proceedings of the ACM on Web Conference 202410.1145/3589335.3651241(1027-1030)Online publication date: 13-May-2024
https://doi.org/10.1145/3589335.3651241
Yang HRang RXing LZhang LCai HGuo MSun J(2024)V-GMR: a variational autoencoder-based heterogeneous graph multi-behavior recommendation modelApplied Intelligence10.1007/s10489-024-05360-x54:4(3337-3350)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1007/s10489-024-05360-x
Foniakov MBardukov AMakarov I(2024)Application of Multimodal Machine Learning for Image Recommendation SystemsRecent Trends in Analysis of Images, Social Networks and Texts10.1007/978-3-031-67008-4_18(235-249)Online publication date: 30-Jul-2024
https://doi.org/10.1007/978-3-031-67008-4_18
Wang JWang XZhao G(2023)Task Recommendation via Heterogeneous Multi-modal Features and Decision Fusion in Mobile CrowdsensingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/362623920:3(1-20)Online publication date: 10-Nov-2023
https://dl.acm.org/doi/10.1145/3626239
Eichinger TKüpper A(2023)Distributed Data Minimization for Decentralized Collaborative Filtering SystemsProceedings of the 24th International Conference on Distributed Computing and Networking10.1145/3571306.3571400(140-149)Online publication date: 4-Jan-2023
https://dl.acm.org/doi/10.1145/3571306.3571400
Liu KXue FGuo DWu LLi SHong R(2023)MEGCF: Multimodal Entity Graph Collaborative Filtering for Personalized RecommendationACM Transactions on Information Systems10.1145/354410641:2(1-27)Online publication date: 3-Apr-2023
https://dl.acm.org/doi/10.1145/3544106
Wei WHuang CXia LZhang C(2023)Multi-Modal Self-Supervised Learning for RecommendationProceedings of the ACM Web Conference 202310.1145/3543507.3583206(790-800)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583206
Li SGuo DLiu KHong RXue FChen HDuh WHuang HKato MMothe JPoblete B(2023)Multimodal Counterfactual Learning Network for Multimedia-based RecommendationProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591739(1539-1548)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591739
Liu KXue FGuo DSun PQian SHong R(2023)Multimodal Graph Contrastive Learning for Multimedia-Based RecommendationIEEE Transactions on Multimedia10.1109/TMM.2023.325110825(9343-9355)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TMM.2023.3251108

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents