short-paper

Public Access

Modeling Sequences as Distributions with Uncertainty for Sequential Recommendation

Authors:

Philip S. YuAuthors Info & Claims

CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

Pages 3019 - 3023

https://doi.org/10.1145/3459637.3482145

Published: 30 October 2021 Publication History

Abstract

The sequential patterns within the user interactions are pivotal for representing the user's preference and capturing latent relationships among items. The recent advancements of sequence modeling by Transformers advocate the community to devise more effective encoders for the sequential recommendation. Most existing sequential methods assume users are deterministic. However, item-item transitions might fluctuate significantly in several item aspects and exhibit randomness of user interests. This stochastic characteristics brings up a solid demand to include uncertainties in representing sequences and items. Additionally, modeling sequences and items with uncertainties expands users' and items' interaction spaces, thus further alleviating cold-start problems.

In this work, we propose a Distribution-based Transformer for Sequential Recommendation (DT4SR), which injects uncertainties into sequential modeling. We use Elliptical Gaussian distributions to describe items and sequences with uncertainty. We describe the uncertainty in items and sequences as Elliptical Gaussian distribution. And we adopt Wasserstein distance to measure the similarity between distributions. We devise two novel Transformers for modeling mean and covariance, which guarantees the positive-definite property of distributions. The proposed method significantly outperforms the state-of-the-art methods. The experiments on three benchmark datasets also demonstrate its effectiveness in alleviating cold-start issues. The code is available in https://github.com/DyGRec/DT4SR.

References

[1]

Martin Arjovsky, Soumith Chintala, and Leon Bottou. 2017. Wasserstein generative adversarial networks. In International conference on machine learning. PMLR, 214--223.

Digital Library

[2]

Aleksandar Bojchevski and Stephan Günnemann. 2017. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. arXiv preprint arXiv:1707.03815 (2017).

[3]

Xu Chen, Hongteng Xu, Yongfeng Zhang, Jiaxi Tang, Yixin Cao, Zheng Qin, and Hongyuan Zha. 2018. Sequential recommendation with user memory networks. In Proceedings of the eleventh ACM international conference on web search and data mining. 108--116.

Digital Library

[4]

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 (2015).

[5]

Guglielmo Faggioli, Mirko Polato, and Fabio Aiolli. 2020. Recency aware collaborative filtering for next basket recommendation. In Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization. 80--87.

Digital Library

[6]

Ziwei Fan, Zhiwei Liu, Jiawei Zhang, Yun Xiong, Lei Zheng, and Philip S. Yu. 2021. Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management. ACM.

Digital Library

[7]

Sharon Givon and Victor Lavrenko. 2009. Predicting social-tags for cold start book recommendations. In Proceedings of the third ACM conf. on Recommender systems. 333--336.

Digital Library

[8]

Ruining He, Wang-Cheng Kang, and Julian McAuley. 2017. Translation-based recommendation. In Proceedings of the eleventh ACM conference on recommender systems. 161--169.

Digital Library

[9]

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 639--648.

Digital Library

[10]

Cheng-Kang Hsieh, Longqi Yang, Yin Cui, Tsung-Yi Lin, Serge Belongie, and Deborah Estrin. 2017. Collaborative metric learning. In Proceedings of the 26th international conference on world wide web. 193--201.

Digital Library

[11]

Wang-Cheng Kang, Chen Fang, Zhaowen Wang, and Julian McAuley. 2017. Visually-aware fashion recommendation and design with generative image models. In 2017 IEEE International Conference on Data Mining (ICDM). IEEE, 207--216.

[12]

Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recommendation. In 2018 IEEE International Conference on Data Mining (ICDM). IEEE, 197--206.

[13]

S Kullback and RA Leibler. 1951. On information and sufficiency Annals of Mathematical Statistics, 22 (1): 79--86.

[14]

Jiacheng Li, Yujie Wang, and Julian McAuley. 2020 a. Time interval aware self-attention for sequential recommendation. In Proceedings of the 13th International Conf. on Web Search and Data Mining. 322--330.

Digital Library

[15]

Mingming Li, Shuai Zhang, Fuqing Zhu, Wanhui Qian, Liangjun Zang, Jizhong Han, and Songlin Hu. 2020 b. Symmetric metric learning with adaptive margin for recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 4634--4641.

[16]

Ye Liu, Yao Wan, Jianguo Zhang, Wenting Zhao, and S Yu Philip. 2021 b. Enriching Non-Autoregressive Transformer with Syntactic and Semantic Structures for Neural Machine Translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 1235--1244.

[17]

Zhiwei Liu, Ziwei Fan, Yu Wang, and Philip S. Yu. 2021 a. Augmenting Sequential Recommendation with Pseudo-Prior Items via Reversely Pre-training Transformer. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval.

Digital Library

[18]

Zhiwei Liu, Xiaohan Li, Ziwei Fan, Stephen Guo, Kannan Achan, and Philip S. Yu. 2020 a. Basket Recommendation with Multi-Intent Translation Graph Neural Network. arXiv preprint arXiv:2010.11419 (2020).

[19]

Zhiwei Liu, Mengting Wan, Stephen Guo, Kannan Achan, and Philip S. Yu. 2020 b. BasConv: Aggregating Heterogeneous Interactions for Basket Recommendation with Graph Convolutional Neural Network. In Proceedings of the 2020 SIAM International Conf. on Data Mining. SIAM, 64--72.

[20]

Chen Ma, Peng Kang, and Xue Liu. 2019. Hierarchical gating networks for sequential recommendation. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 825--833.

Digital Library

[21]

Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. 43--52.

Digital Library

[22]

Seong Joon Oh, Kevin Murphy, Jiyan Pan, Joseph Roth, Florian Schroff, and Andrew Gallagher. 2018. Modeling uncertainty with hedged instance embedding. arXiv preprint arXiv:1810.00319 (2018).

[23]

Bo Peng, Zhiyun Ren, Srinivasan Parthasarathy, and Xia Ning. 2020. M2: Mixed Models with Preferences, Popularities and Transitions for Next-Basket Recommendation. arXiv preprint arXiv:2004.01646 (2020).

[24]

Bo Peng, Zhiyun Ren, Srinivasan Parthasarathy, and Xia Ning. 2021. HAM: hybrid associations models for sequential recommendation. IEEE Transactions on Knowledge and Data Engineering.

[25]

Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012. BPR: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618 (2012).

[26]

Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conf. on information and knowledge management. 1441--1450.

Digital Library

[27]

Jiaxi Tang and Ke Wang. 2018. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. 565--573.

Digital Library

[28]

Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. 2018. Latent relational metric learning via memory-based attention for collaborative ranking. In Proceedings of the 2018 World Wide Web Conference. 729--739.

Digital Library

[29]

A"aron Van Den Oord, Sander Dieleman, and Benjamin Schrauwen. 2013. Deep content-based music recommendation. In Neural Information Processing Systems Conference (NIPS 2013), Vol. 26. Neural Information Processing Systems Foundation (NIPS).

Digital Library

[30]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762 (2017).

[31]

Luke Vilnis and Andrew McCallum. 2014. Word representations via gaussian embedding. arXiv preprint arXiv:1412.6623 (2014).

[32]

Yu Wang, Zhiwei Liu, Ziwei Fan, Philip S. Yu, and Lichao Sun. 2021. DSKReG: Differentiable Sampling on Knowledge Graph forRecommendation with Relational GNN. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management. ACM.

Digital Library

[33]

Congying Xia, Caiming Xiong, and Philip Yu. 2021. Pseudo Siamese Network for Few-shot Intent Generation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval.

Digital Library

[34]

An Yan, Shuo Cheng, Wang-Cheng Kang, Mengting Wan, and Julian McAuley. 2019. CosRec: 2D convolutional neural networks for sequential recommendation. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2173--2176.

Digital Library

[35]

Jian-Guo Zhang, Kazuma Hashimoto, Yao Wan, Ye Liu, Caiming Xiong, and Philip S. Yu. 2021. Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detection. arXiv preprint arXiv:2106.04564 (2021).

[36]

Lei Zheng, Ziwei Fan, Chun-Ta Lu, Jiawei Zhang, and Philip S. Yu. 2019a. Gated Spectral Units: Modeling Co-evolving Patterns for Sequential Recommendation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1077--1080.

Digital Library

[37]

Lei Zheng, Chaozhuo Li, Chun-Ta Lu, Jiawei Zhang, and Philip S. Yu. 2019b. Deep Distribution Network: Addressing the Data Sparsity Issue for Top-N Recommendation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1081--1084.

Digital Library

Cited By

Ge MWang CQin XDai JHuang LQin QZhang W(2025)Personalized Dual Transformer Network for sequential recommendationNeurocomputing10.1016/j.neucom.2024.129244(129244)Online publication date: Jan-2025
https://doi.org/10.1016/j.neucom.2024.129244
Balsells-Rodas CYang FHuang ZGao YHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Explainable Uncertainty Attribution for Sequential RecommendationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657900(2401-2405)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657900
Shao ZWang SLu WZhang WGuan HZhao L(2024)Filter-Enhanced Hypergraph Transformer for Multi-Behavior Sequential RecommendationICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10446828(6575-6579)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10446828
Show More Cited By

Index Terms

Modeling Sequences as Distributions with Uncertainty for Sequential Recommendation
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems
    2. Users and interactive retrieval
      1. Personalization
  2. Information systems applications
    1. Data mining
      1. Collaborative filtering

Recommendations

Sequential Recommendation via Stochastic Self-Attention
WWW '22: Proceedings of the ACM Web Conference 2022

Sequential recommendation models the dynamics of a user’s previous behaviors in order to forecast the next item, and has drawn a lot of attention. Transformer-based approaches, which embed items as vectors and use dot-product self-attention to measure ...
Sequential Recommendation with Dual Side Neighbor-based Collaborative Relation Modeling
WSDM '20: Proceedings of the 13th International Conference on Web Search and Data Mining

Sequential recommendation task aims to predict user preference over items in the future given user historical behaviors. The order of user behaviors implies that there are resourceful sequential patterns embedded in the behavior history which reveal the ...
User Popularity Preference Aware Sequential Recommendation
Computational Science – ICCS 2023
Abstract
In recommender systems, users’ preferences for item popularity are diverse and dynamic, which reveals the different items that users prefer. Therefore, identifying user popularity preferences are significant for personalized recommendations. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

October 2021

4966 pages

ISBN:9781450384469

DOI:10.1145/3459637

General Chairs:
Gianluca Demartini
The University of Queensland, Australia
,
Guido Zuccon
The University of Queensland, Australia
,
Program Chairs:
J. Shane Culpepper
RMIT University, Australia
,
Zi Huang
The University of Queensland, Australia
,
Hanghang Tong
University of Illinois at Urbana-Champaign, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

NSF (National Science Foundation)

Conference

CIKM '21

Sponsor:

CIKM '21: The 30th ACM International Conference on Information and Knowledge Management

November 1 - 5, 2021

Queensland, Virtual Event, Australia

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
1,065
Total Downloads

Downloads (Last 12 months)372
Downloads (Last 6 weeks)50

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ge MWang CQin XDai JHuang LQin QZhang W(2025)Personalized Dual Transformer Network for sequential recommendationNeurocomputing10.1016/j.neucom.2024.129244(129244)Online publication date: Jan-2025
https://doi.org/10.1016/j.neucom.2024.129244
Balsells-Rodas CYang FHuang ZGao YHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Explainable Uncertainty Attribution for Sequential RecommendationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657900(2401-2405)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657900
Shao ZWang SLu WZhang WGuan HZhao L(2024)Filter-Enhanced Hypergraph Transformer for Multi-Behavior Sequential RecommendationICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10446828(6575-6579)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10446828
Wang CWang MWang XTan Y(2024)IPSRM: An intent perceived sequential recommendation modelJournal of King Saud University - Computer and Information Sciences10.1016/j.jksuci.2024.10220636:9(102206)Online publication date: Nov-2024
https://doi.org/10.1016/j.jksuci.2024.102206
Bai TWang XZhang ZSong WWu BNie J(2024)GPR-OPTInformation Processing and Management: an International Journal10.1016/j.ipm.2023.10352561:1Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.ipm.2023.103525
Wu JYang BMao RLi Q(2024)Popularity-aware sequential recommendation with user desireExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.121429237:PBOnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.121429
Zhang XLin DLiu MChen J(2024)Analysis of order-of-addition experimentsComputational Statistics & Data Analysis10.1016/j.csda.2024.108077(108077)Online publication date: Nov-2024
https://doi.org/10.1016/j.csda.2024.108077
Liu PZheng LChen JZhang GXu YFang J(2024)Enhancing Recommendation with Denoising Auxiliary TaskJournal of Computer Science and Technology10.1007/s11390-024-4069-539:5(1123-1137)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s11390-024-4069-5
Peng BParthasarathy SNing X(2024)Intention enhanced mixed attentive model for session-based recommendationData Mining and Knowledge Discovery10.1007/s10618-024-01023-038:4(2032-2061)Online publication date: 3-Jun-2024
https://doi.org/10.1007/s10618-024-01023-0
He TLiu XXing ZHuang XXu ZWang Y(2024)Adversarial Training and Contrastive Learning with Bidirectional Transformers for Sequence RecommendationPattern Recognition and Computer Vision10.1007/978-981-97-8490-5_34(477-489)Online publication date: 7-Nov-2024
https://doi.org/10.1007/978-981-97-8490-5_34
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents