Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3397271.3401467acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
research-article
Open access

Deep Reinforcement Learning for Information Retrieval: Fundamentals and Advances

Published: 25 July 2020 Publication History

Abstract

Information retrieval (IR) techniques, such as search, recommendation and online advertising, satisfying users' information needs by suggesting users personalized objects (information or services) at the appropriate time and place, play a crucial role in mitigating the information overload problem. Since the widely use of mobile applications, more and more information retrieval services have provided interactive functionality and products. Thus, learning from interaction becomes a crucial machine learning paradigm for interactive IR, which is based on reinforcement learning. With recent great advances in deep reinforcement learning (DRL), there have been increasing interests in developing DRL based information retrieval techniques, which could continuously update the information retrieval strategies according to users' real-time feedback, and optimize the expected cumulative long-term satisfaction from users. Our workshop aims to provide a venue, which can bring together academia researchers and industry practitioners (i) to discuss the principles, limitations and applications of DRL for information retrieval, and (ii) to foster research on innovative algorithms, novel techniques, and new applications of DRL to information retrieval.

Supplementary Material

MP4 File (3397271.3401467.mp4)
Information retrieval (IR) techniques satisfy users' information needs by suggesting users personalized objects at the appropriate time and place, play a crucial role in mitigating the information overload problem. With recent great advances in deep reinforcement learning (DRL), there have been increasing interests in developing DRL based information retrieval techniques, which could continuously update the information retrieval strategies according to users' real-time feedback, and optimize the cumulative long-term satisfaction from users. Our workshop aims to provide a venue, which can bring together academia researchers and industry practitioners (i) to discuss the principles, limitations and applications of DRL for information retrieval, and (ii) to foster research on innovative algorithms, novel techniques, and new applications of DRL to information retrieval.

References

[1]
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. 2002 a. Finite-time analysis of the multiarmed bandit problem. Machine learning, Vol. 47, 2--3 (2002), 235--256.
[2]
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. 2002 b. The nonstochastic multiarmed bandit problem. SIAM journal on computing, Vol. 32, 1 (2002), 48--77.
[3]
Han Cai, Kan Ren, Weinan Zhang, Kleanthis Malialis, Jun Wang, Yong Yu, and Defeng Guo. 2017. Real-time bidding by reinforcement learning in display advertising. In WSDM. 661--670.
[4]
Chia-Hui Chang, Mohammed Kayed, Moheb R Girgis, and Khaled F Shaalan. 2006. A survey of web information extraction systems. TKDE, Vol. 18, 10 (2006), 1411--1428.
[5]
Haokun Chen, Xinyi Dai, Han Cai, Weinan Zhang, Xuejian Wang, Ruiming Tang, Yuzhou Zhang, and Yong Yu. 2019 b. Large-scale interactive recommendation with tree-structured policy gradient. In AAAI, Vol. 33. 3312--3320.
[6]
Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. 2019 a. Top-K Off-Policy Correction for a REINFORCE Recommender System. In WSDM. ACM, 456--464.
[7]
W Bruce Croft, Michael Bendersky, Hang Li, and Gu Xu. 2011. Query representation and understanding workshop. In ACM SIGIR Forum, Vol. 44. ACM New York, NY, USA, 48--53.
[8]
Marina Drosou and Evaggelia Pitoura. 2010. Search result diversification. ACM SIGMOD Record, Vol. 39, 1 (2010), 41--47.
[9]
Jun Feng, Minlie Huang, Li Zhao, Yang Yang, and Xiaoyan Zhu. 2018. Reinforcement learning for relation classification from noisy data. In Thirty-Second AAAI Conference on Artificial Intelligence.
[10]
Hector Garcia-Molina, Georgia Koutrika, and Aditya Parameswaran. 2011. Information seeking: convergence of search, recommendations, and advertising. Commun. ACM, Vol. 54, 11 (2011), 121--130.
[11]
Li He, Liang Wang, Kaipeng Liu, Bo Wu, and Weinan Zhang. 2018. Optimizing Sponsored Search Ranking Strategy by Deep Reinforcement Learning. arXiv preprint arXiv:1803.07347 (2018).
[12]
Junqi Jin, Chengru Song, Han Li, Kun Gai, Jun Wang, and Weinan Zhang. 2018. Real-time bidding with multi-agent reinforcement learning in display advertising. In CIKM. 2193--2201.
[13]
Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Reinforcement learning: A survey. Journal of artificial intelligence research, Vol. 4 (1996), 237--285.
[14]
Henrik Kretzschmar, Markus Spies, Christoph Sprunk, and Wolfram Burgard. 2016. Socially compliant mobile robot navigation via inverse reinforcement learning. The International Journal of Robotics Research, Vol. 35, 11 (2016), 1289--1307.
[15]
Feng Liu, Ruiming Tang, Xutao Li, Weinan Zhang, Yunming Ye, Haokun Chen, Huifeng Guo, and Yuzhou Zhang. 2018. Deep reinforcement learning based recommendation with explicit user-item interactions modeling. arXiv preprint arXiv:1810.12027 (2018).
[16]
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In ICML. 1928--1937.
[17]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013).
[18]
Rodrigo Nogueira, Jannis Bulian, and Massimiliano Ciaramita. 2018. Learning to coordinate multiple reinforcement learning agents for diverse query reformulation. arXiv preprint arXiv:1809.10658 (2018).
[19]
Rodrigo Nogueira and Kyunghyun Cho. 2017. Task-oriented query reformulation with reinforcement learning. arXiv preprint arXiv:1704.04572 (2017).
[20]
Alessandro Nuara, Francesco Trovo, Nicola Gatti, and Marcello Restelli. 2018. A combinatorial-bandit algorithm for the online joint bid/budget optimization of pay-per-click advertising campaigns. In AAAI.
[21]
Yanru Qu, Bohui Fang, Weinan Zhang, Ruiming Tang, Minzhe Niu, Huifeng Guo, Yong Yu, and Xiuqiang He. 2018. Product-based neural networks for user response prediction over multi-field categorical data. TOIS, Vol. 37, 1 (2018), 1--35.
[22]
Razieh Rahimi and Grace Hui Yang. [n. d.]. Modeling Exploration of Intrinsically Diverse Search Tasks as Markov Decision Processes. ([n. d.]).
[23]
Konstantin Salomatin, Tie-Yan Liu, and Yiming Yang. 2012. A unified optimization framework for auction and guaranteed delivery in online advertising. In CIKM. 2005--2009.
[24]
Guy Shani, David Heckerman, and Ronen I Brafman. 2005. An MDP-based recommender system. JMLR, Vol. 6, Sep (2005), 1265--1295.
[25]
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of Go with deep neural networks and tree search. nature, Vol. 529, 7587 (2016), 484.
[26]
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. 2017. Mastering the game of Go without human knowledge. Nature, Vol. 550, 7676 (2017), 354.
[27]
Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction .MIT press.
[28]
Liang Tang, Romer Rosales, Ajit Singh, and Deepak Agarwal. 2013. Automatic ad format selection via contextual bandits. In CIKM. 1587--1594.
[29]
Zhiwen Tang and Grace Hui Yang. 2017. A Reinforcement Learning Approach for Dynamic Search. In TREC.
[30]
Christopher John Cornish Hellaby Watkins. 1989. Learning from delayed rewards. Ph.D. Dissertation. King's College, Cambridge.
[31]
Zeng Wei, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. 2017. Reinforcement learning to rank with Markov decision process. In SIGIR. 945--948.
[32]
Di Wu, Cheng Chen, Xun Yang, Xiujun Chen, Qing Tan, Jian Xu, and Kun Gai. 2018a. A multi-agent reinforcement learning method for impression allocation in online display advertising. arXiv preprint arXiv:1809.03152 (2018).
[33]
Qingyun Wu, Naveen Iyer, and Hongning Wang. 2018b. Learning contextual bandits in a non-stationary environment. In SIGIR. 495--504.
[34]
Long Xia, Jun Xu, Yanyan Lan, Jiafeng Guo, Wei Zeng, and Xueqi Cheng. 2017. Adapting Markov decision process for search result diversification. In SIGIR. 535--544.
[35]
Min Xu, Tao Qin, and Tie-Yan Liu. 2013. Estimation bias in multi-armed bandit algorithms for search advertising. In NIPS. 2400--2408.
[36]
Hongxia Yang and Quan Lu. 2016. Dynamic contextual multi arm bandits in display advertisement. In ICDM. IEEE, 1305--1310.
[37]
Chunqiu Zeng, Qing Wang, Shekoofeh Mokhtari, and Tao Li. 2016. Online context-aware recommendation with time varying multi-armed bandit. In KDD. 2025--2034.
[38]
Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. 2018a. Deep Reinforcement Learning for Page-wise Recommendations. In Proceedings of the 12th ACM Recommender Systems Conference. ACM, 95--103.
[39]
Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. 2018b. Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning. In KDD. ACM, 1040--1048.
[40]
Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Dawei Yin, Yihong Zhao, and Jiliang Tang. 2017. Deep Reinforcement Learning for List-wise Recommendations. arXiv preprint arXiv:1801.00209 (2017).
[41]
Xiaoxue Zhao, Weinan Zhang, and Jun Wang. 2013. Interactive collaborative filtering. In CIKM. 1411--1420.
[42]
Xiangyu Zhao, Xudong Zheng, Xiwang Yang, Xiaobing Liu, and Jiliang Tang. 2020. Jointly Learning to Recommend and Advertise. arXiv preprint arXiv:2003.00097 (2020).

Cited By

View all
  • (2024)AgentIR: 1st Workshop on Agent-based Information RetrievalProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657989(3025-3028)Online publication date: 10-Jul-2024
  • (2024)A novel ensemble deep reinforcement learning model for short‐term load forecasting based on Q‐learning dynamic model selectionThe Journal of Engineering10.1049/tje2.124092024:7Online publication date: 2-Jul-2024
  • (2023)TGRLProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619695(31077-31093)Online publication date: 23-Jul-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval
July 2020
2548 pages
ISBN:9781450380164
DOI:10.1145/3397271
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. deep reinforcement learning
  2. information retrieval

Qualifiers

  • Research-article

Funding Sources

  • NSFC

Conference

SIGIR '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)520
  • Downloads (Last 6 weeks)45
Reflects downloads up to 12 Sep 2024

Other Metrics

Citations

Cited By

View all
  • (2024)AgentIR: 1st Workshop on Agent-based Information RetrievalProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657989(3025-3028)Online publication date: 10-Jul-2024
  • (2024)A novel ensemble deep reinforcement learning model for short‐term load forecasting based on Q‐learning dynamic model selectionThe Journal of Engineering10.1049/tje2.124092024:7Online publication date: 2-Jul-2024
  • (2023)TGRLProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619695(31077-31093)Online publication date: 23-Jul-2023
  • (2023)Information Retrieval and Optimization in Distribution and Logistics Management Using Deep Reinforcement LearningInternational Journal of Information Systems and Supply Chain Management10.4018/IJISSCM.31616616:1(1-19)Online publication date: 13-Jan-2023
  • (2023)Adversarially Trained Environment Models Are Effective Policy Evaluators and Improvers - An Application to Information RetrievalProceedings of the Fifth International Conference on Distributed Artificial Intelligence10.1145/3627676.3627680(1-12)Online publication date: 30-Nov-2023
  • (2023)User Retention-oriented Recommendation with Decision TransformerProceedings of the ACM Web Conference 202310.1145/3543507.3583418(1141-1149)Online publication date: 30-Apr-2023
  • (2023)LinRec: Linear Attention Mechanism for Long-term Sequential Recommender SystemsProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591717(289-299)Online publication date: 19-Jul-2023
  • (2023)Information Retrieval: Recent Advances and BeyondIEEE Access10.1109/ACCESS.2023.329577611(76581-76604)Online publication date: 2023
  • (2023)From Rational Agent to Human with Bounded RationalityA Behavioral Economics Approach to Interactive Information Retrieval10.1007/978-3-031-23229-9_3(65-89)Online publication date: 18-Feb-2023
  • (2022)Machine Reading at Scale: A Search Engine for Scientific and Academic ResearchSystems10.3390/systems1002004310:2(43)Online publication date: 5-Apr-2022
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media