short-paper

Attention Mixtures for Time-Aware Sequential Recommendation

Authors:

Guillaume Salha-Galvan,

Romain HennequinAuthors Info & Claims

SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 1821 - 1826

https://doi.org/10.1145/3539618.3591951

Published: 18 July 2023 Publication History

Abstract

Transformers emerged as powerful methods for sequential recommendation. However, existing architectures often overlook the complex dependencies between user preferences and the temporal context. In this short paper, we introduce MOJITO, an improved Transformer sequential recommender system that addresses this limitation. MOJITO leverages Gaussian mixtures of attention-based temporal context and item embedding representations for sequential modeling. Such an approach permits to accurately predict which items should be recommended next to users depending on past actions and the temporal context. We demonstrate the relevance of our approach, by empirically outperforming existing Transformers for sequential recommendation on several real-world datasets.

References

[1]

Gediminas Adomavicius and Alexander Tuzhilin. 2011. Context-Aware Recommender Systems. Recommender Systems Handbook (2011), 217--253.

[2]

Srinadh Bhojanapalli, Ayan Chakrabarti, Himanshu Jain, Sanjiv Kumar, Michal Lukasik, and Andreas Veit. 2021. Eigen Analysis of Self-Attention and its Reconstruction from Partial Computation. arXiv preprint arXiv:2106.08823 (2021).

[3]

David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research, Vol. 3 (2003), 993--1022.

Digital Library

[4]

Théo Bontempelli, Benjamin Chapus, Francc ois Rigaud, Mathieu Morlon, Marin Lorant, and Guillaume Salha-Galvan. 2022. Flow Moods: Recommending Music by Moods on Deezer. Proceedings of the 16th ACM Conference on Recommender Systems (2022), 452--455.

Digital Library

[5]

Chao Chen, Haoyu Geng, Nianzu Yang, Junchi Yan, Daiyue Xue, Jianping Yu, and Xiaokang Yang. 2021. Learning Self-Modulating Attention in Continuous Time Space with Applications to Sequential Recommendation. Proceedings of the 38th International Conference on Machine Learning (2021), 1606--1616.

[6]

Ching-Wei Chen, Paul Lamere, Markus Schedl, and Hamed Zamani. 2018. RecSys Challenge 2018: Automatic Music Playlist Continuation. Proceedings of the 12th ACM Conference on Recommender Systems (2018), 527--528.

Digital Library

[7]

Sung Min Cho, Eunhyeok Park, and Sungjoo Yoo. 2020. MEANTIME: Mixture of Attention Mechanisms with Multi-Temporal Embeddings for Sequential Recommendation. Proceedings of the 14th ACM Conference on Recommender Systems (2020), 515--520.

Digital Library

[8]

Hanwen Du, Hui Shi, Pengpeng Zhao, Deqing Wang, Victor Sheng, Yanchi Liu, Guanfeng Liu, and Lei Zhao. 2022. Contrastive Learning with Bidirectional Transformers for Sequential Recommendation. Proceedings of the 31st ACM International Conference on Information and Knowledge Management (2022), 396--405.

Digital Library

[9]

Hui Fang, Danning Zhang, Yiheng Shu, and Guibing Guo. 2020. Deep Learning for Sequential Recommendation: Algorithms, Influential Factors, and Evaluations. ACM Transactions on Information Systems (TOIS), Vol. 39, 1 (2020), 1--42.

Digital Library

[10]

Lei Guo, Jinyu Zhang, Tong Chen, Xinhua Wang, and Hongzhi Yin. 2022. Reinforcement Learning-Enhanced Shared-Account Cross-Domain Sequential Recommendation. IEEE Transactions on Knowledge and Data Engineering (2022).

Digital Library

[11]

Casper Hansen, Christian Hansen, Lucas Maystre, Rishabh Mehrotra, Brian Brost, Federico Tomasi, and Mounia Lalmas. 2020. Contextual and Sequential User Embeddings for Large-Scale Music Recommendation. Proceedings of the 14th ACM Conference on Recommender Systems (2020), 53--62.

Digital Library

[12]

F Maxwell Harper and Joseph A Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems, Vol. 5, 4 (2015), 1--19.

Digital Library

[13]

Balazs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016. Session-Based Recommendations with Recurrent Neural Networks. Proceedings of the 4th International Conference on Learning Representations (2016).

[14]

Haoji Hu, Xiangnan He, Jinyang Gao, and Zhi-Li Zhang. 2020. Modeling Personalized Item Frequency Information for Next-Basket Recommendation. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (2020), 1071--1080.

Digital Library

[15]

Lin Jing, Weike Pan, and Zhong Ming. 2020. FISSA: Fusing Item Similarity Models with Self-Attention Networks for Sequential Recommendation. Proceedings of the 14th ACM Conference on Recommender Systems (2020), 130--139.

[16]

Santosh Kabbur, Xia Ning, and George Karypis. 2013. Factored Item Similarity Models for Top-N Recommender Systems. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2013), 659--667.

Digital Library

[17]

Wang-Cheng Kang and Julian McAuley. 2018. Self-Attentive Sequential Recommendation. Proceedings of the 20th International Conference on Data Mining (2018), 197--206.

[18]

Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations (2015).

[19]

Cheng-Te Li, Cheng Hsu, and Yang Zhang. 2022. FairSR: Fairness-Aware Sequential Recommendation through Multi-Task Learning with Preference Graph Embeddings. ACM Transactions on Intelligent Systems and Technology (TIST), Vol. 13, 1 (2022), 1--21.

Digital Library

[20]

Jiacheng Li, Yujie Wang, and Julian McAuley. 2020. Time Interval Aware Self-Attention for Sequential Recommendation. Proceedings of the 13th International Conference on Web Search and Data Mining (2020), 322--330.

Digital Library

[21]

Xiaoyang Liu, Chong Liu, Pinzheng Wang, Rongqin Zheng, Lixin Zhang, Leyu Lin, Zhijun Chen, and Liangliang Fu. 2022. UFNRec: Utilizing False Negative Samples for Sequential Recommendation. arXiv preprint arXiv:2208.04116 (2022).

[22]

Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image-Based Recommendations on Styles and Substitutes. Proceedings of the 38th ACM SIGIR Conference on Research and Development in Information Retrieval (2015), 43--52.

Digital Library

[23]

Paul Michel, Omer Levy, and Graham Neubig. 2019. Are Sixteen Heads Really Better Than One? Advances in Neural Information Processing Systems, Vol. 32 (2019).

[24]

Tan M. Nguyen, Tam Nguyen, Hai Do, Khai Nguyen, Vishwanath Saragadam, Minh Pham, Duy Khuong Nguyen, Nhat Ho, and Stanley J. Osher. 2022. Improving Transformer with an Admixture of Attention Heads. Advances in Neural Information Processing Systems 35 (2022).

[25]

Chanyoung Park, Donghyun Kim, Xing Xie, and Hwanjo Yu. 2018. Collaborative Translational Metric Learning. Proceedings of the 2018 IEEE International Conference on Data Mining (2018), 367--376.

[26]

Bruno L Pereira, Alberto Ueda, Gustavo Penha, Rodrygo LT Santos, and Nivio Ziviani. 2019. Online Learning to Rank for Sequential Music Recommendation. Proceedings of the 13th ACM Conference on Recommender Systems (2019), 237--245.

Digital Library

[27]

Aleksandr Petrov and Craig MacDonald. 2022. Effective and Efficient Training for Sequential Recommendation using Recency Sampling. Proceedings of the 16th ACM Conference on Recommender Systems (2022), 81--91.

Digital Library

[28]

Ruihong Qiu, Jingjing Li, Zi Huang, and Hongzhi Yin. 2019. Rethinking the Item Order in Session-based Recommendation with Graph Neural Networks. Proceedings of the 28th ACM International Conference on Information and Knowledge Management (2019), 579--588.

Digital Library

[29]

Massimo Quadrana, Paolo Cremonesi, and Dietmar Jannach. 2018. Sequence-Aware Recommender Systems. ACM Computing Surveys, Vol. 51, 4 (2018), 1--36.

Digital Library

[30]

Ahmed Rashed, Shereen Elsayed, and Lars Schmidt-Thieme. 2022. Context and Attribute-Aware Sequential Recommendation via Cross-Attention. Proceedings of the 16th ACM Conference on Recommender Systems (2022), 71--80.

Digital Library

[31]

Pengjie Ren, Zhumin Chen, Jing Li, Zhaochun Ren, Jun Ma, and Maarten de Rijke. 2019. RepeatNet: A Repeat Aware Neural Recommendation Machine for Session-Based Recommendation. Proceedings of the 33rd AAAI Conference on Artificial Intelligence (2019), 4806--4813.

Digital Library

[32]

Markus Schedl. 2016. The LFM-1b Dataset for Music Retrieval and Recommendation. Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval (2016), 103--110.

Digital Library

[33]

Markus Schedl. 2019. Deep learning in music recommendation systems. Frontiers in Applied Mathematics and Statistics (2019), 44.

[34]

Markus Schedl, Hamed Zamani, Ching-Wei Chen, Yashar Deldjoo, and Mehdi Elahi. 2018. Current Challenges and Visions in Music Recommender Systems Research. International Journal of Multimedia Information Retrieval, Vol. 7, 2 (2018), 95--116.

[35]

Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. Proceedings of the 28th ACM International Conference on Information and Knowledge Management (2019), 1441--1450.

Digital Library

[36]

Zhu Sun, Di Yu, Hui Fang, Jie Yang, Xinghua Qu, Jie Zhang, and Cong Geng. 2020. Are We Evaluating Rigorously? Benchmarking Recommendation for Reproducible Evaluation and Fair Comparison. Proceedings of the 14th ACM Conference on Recommender Systems (2020), 23--32.

Digital Library

[37]

Yi Tay, Anh Tuan Luu, and Cheung Hui Siu. 2018. Latent Relational Metric Learning via Memory-Based Attention for Collaborative Ranking. Proceedings of the 27th World Wide Web Conference (2018), 729--739.

Digital Library

[38]

Viet-Anh Tran, Guillaume Salha-Galvan, Romain Hennequin, and Manuel Moussallam. 2021. Hierarchical Latent Relation Modeling for Collaborative Metric Learning. Proceedings of the 15th ACM Conference on Recommender Systems (2021), 302--309.

Digital Library

[39]

Xuan Tuan Trinh and Minh Phuong Tu. 2017. 3D Convolutional Networks for Session-Based Recommendation with Content Features. Proceedings of the 11th ACM Conference on Recommender Systems (2017), 138--146.

[40]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ?ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. Advances in Neural Information Processing Systems 30 (2017), 5998--6008.

[41]

Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing Multi-Head Self-Attention: Specialized Heads do the Heavy Lifting, the Rest can be Pruned. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019), 5797--5808.

[42]

Shoujin Wang, Longbing Cao, Yan Wang, Quan Z. Sheng, Mehmet Orgun, and Defu Lian. 2021. A Survey on Session-Based Recommender Systems. ACM Computing Surveys, Vol. 54, 7 (2021), 1--38.

[43]

Liwei Wu, Shuqing Li, Cho-Jui Hsieh, and James Sharpnack. 2020. SSE-PT: Sequential Recommendation via Personalized Transformer. Proceedings of the 14th ACM Conference on Recommender Systems (2020), 328--337.

Digital Library

[44]

Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019. Session-Based Recommendation with Graph Neural Networks. Proceedings of the 33rd AAAI Conference on Artificial Intelligence (2019), 346--353.

Digital Library

[45]

Lianghao Xia, Chao Huang, Yong Xu, and Jian Pei. 2022. Multi-Behavior Sequential Recommendation with Temporal Graph Transformer. IEEE Transactions on Knowledge and Data Engineering (2022).

Digital Library

[46]

Xuefeng Xian, Ligang Fang, and Shiming Sun. 2021. Learning Heterogeneous Temporal Patterns of User Preference for Timely Recommendation. Proceedings of the 30th Web Conference (2021), 1274--1283.

[47]

Xu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu, Jinyang Gao, Jiandong Zhang, Bolin Ding, and Bin Cui. 2022. Contrastive Learning for Sequential Recommendation. Proceedings of the 2022 IEEE 38th International Conference on Data Engineering (2022), 1259--1273.

[48]

Chengfeng Xu, Pengpeng Zhao, Yanchi Liu, Victor S. Sheng, Jiajie Xu, Fuzhen Zhuang, Junhua Fang, and Xiaofang Zhou. 2019b. Graph Contextualized Self-attention Network for Session-Based Recommendation. Proceedings of the 28th International Joint Conference on Artificial Intelligence (2019), 3940--3946.

Digital Library

[49]

Da Xu, Chuanwei Ruan, Sushant Kumar, Evren Korpeoglu, and Kannan Achan. 2019a. Self-Attention with Functional Time Representation Learning. Advances in Neural Information Processing Systems, Vol. 32 (2019).

[50]

Yuhao Yang, Chao Huang, Lianghao Xia, Yuxuan Liang, Yanwei Yu, and Chenliang Li. 2022. Multi-Behavior Hypergraph-Enhanced Transformer for Sequential Recommendation. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (2022), 2263--2274.

Digital Library

[51]

Wenwen Ye, Shuaiqiang Wang, Xu Chen, Xuepeng Wang, Zheng Qin, and Dawei Yin. 2020. Time Matters: Sequential Recommendation with Complex Temporal Information. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (2020), 1459--1468.

Digital Library

[52]

Jiaxuan You, Yichen Wang, Aditya Pal, Pong Eksombatchai, Chuck Rosenberg, and Jure Leskovec. 2019. Hierarchical Temporal Convolutional Networks for Dynamic Recommender Systems. Proceedings of the 28th World Wide Web Conference (2019), 2236--2246.

Digital Library

[53]

Shuai Zhang, Yi Tay, Lina Yao, and Aixin Sun. 2019a. Next Item Recommendation with Self-Attentive Metric Learning. AAAI 2019 Workshop on Recommender Systems and Natural Language Processing (2019).

[54]

Tingting Zhang, Pengpeng Zhao, Yanchi Liu, Jiajie Xu, Junhua Fang, Lei Zhao, Victor S. Sheng, and Zhiming Cui. 2019b. AdaCML: Adaptive Collaborative Metric Learning for Recommendation. Proceedings of the 24th International Conference on Database Systems for Advanced Applications (2019), 301--316.

Digital Library

[55]

Yixin Zhang, Yong Liu, Yonghui Xu, Hao Xiong, Chenyi Lei, Wei He, Lizhen Cui, and Chunyan Miao. 2022. Enhancing Sequential Recommendation with Graph Contrastive Learning. Proceedings of the 31st International Joint Conference on Artificial Intelligence (2022), 2398--2405.

[56]

Chang Zhou, Jinze Bai, Junshuai Song, Xiaofei Liu, Zhengchao Zhao, Xiusi Chen, and Jun Gao. 2018. ATRank: An Attention-Based User Behavior Modeling Framework for Recommendation. Proceedings of the 32nd AAAI Conference on Artificial Intelligence (2018), 4564--4571.

Cited By

Schedl MMoscati MSguerra BHennequin RLex EAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)Psychology-informed Information Access Systems WorkshopProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635722(1216-1217)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635722
Zheng XLi XChen ZSun LYu QGuo LLuo Y(2024)Enhanced Self-Attention Mechanism for Long and Short Term Sequential Recommendation ModelsIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2024.33667718:3(2457-2466)Online publication date: Jun-2024
https://doi.org/10.1109/TETCI.2024.3366771

Recommendations

User Popularity Preference Aware Sequential Recommendation
Computational Science – ICCS 2023
Abstract
In recommender systems, users’ preferences for item popularity are diverse and dynamic, which reveals the different items that users prefer. Therefore, identifying user popularity preferences are significant for personalized recommendations. ...
Dynamic time-aware collaborative sequential recommendation with attention-based network
Abstract
A natural way of user modeling in sequential recommendation is to capture long-term and short-term preferences, respectively, given user historical behaviors and then fuse them together. Most existing approaches building on attention-based network ...
Is News Recommendation a Sequential Recommendation Task?
SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

News recommendation is often modeled as a sequential recommendation task, assuming there are rich short-term dependencies over historical clicked news. However, users usually have strong preferences on the temporal diversity of news information and may ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2023

3567 pages

ISBN:9781450394086

DOI:10.1145/3539618

General Chairs:
Hsin-Hsi Chen
National Taiwan University
,
Wei-Jou (Edward) Duh
National Taiwan University
,
Hen-Hsen Huang
Academia Sinica
,
Program Chairs:
Makoto P. Kato
Spotify
,
Josiane Mothe
Universite de Toulouse
,
Barbara Poblete
University of Chile and Amazon Visiting Academic

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

SIGIR '23

Sponsor:

SIGIR

SIGIR '23: The 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 23 - 27, 2023

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
366
Total Downloads

Downloads (Last 12 months)298
Downloads (Last 6 weeks)9

Reflects downloads up to 11 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Schedl MMoscati MSguerra BHennequin RLex EAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)Psychology-informed Information Access Systems WorkshopProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635722(1216-1217)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635722
Zheng XLi XChen ZSun LYu QGuo LLuo Y(2024)Enhanced Self-Attention Mechanism for Long and Short Term Sequential Recommendation ModelsIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2024.33667718:3(2457-2466)Online publication date: Jun-2024
https://doi.org/10.1109/TETCI.2024.3366771

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents