short-paper

On Including the User Dynamic in Learning to Rank

Authors:

Claudio Lucchese,

Raffaele PeregoAuthors Info & Claims

SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 1041 - 1044

https://doi.org/10.1145/3077136.3080714

Published: 07 August 2017 Publication History

Abstract

Ranking query results effectively by considering user past behaviour and preferences is a primary concern for IR researchers both in academia and industry. In this context, LtR is widely believed to be the most effective solution to design ranking models that account for user-interaction features that have proved to remarkably impact on IR effectiveness. In this paper, we explore the possibility of integrating the user dynamic directly into the LtR algorithms. Specifically, we model with Markov chains the behaviour of users in scanning a ranked result list and we modify Lambdamart, a state-of-the-art LtR algorithm, to exploit a new discount loss function calibrated on the proposed Markovian model of user dynamic. We evaluate the performance of the proposed approach on publicly available LtR datasets, finding that the improvements measured over the standard algorithm are statistically significant.

References

[1]

E. Agichtein, E. Brill, and S. Dumais. Improving Web Search Ranking by Incorporating User Behavior Information. In SIGIR, pages 19--26, ACM, 2006.

Digital Library

[2]

A. Broder. A Taxonomy of Web Search. SIGIR Forum, 36(2):3--10, 2002.

Digital Library

[3]

C. J. C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to Rank Using Gradient Descent. In ICML, pages 89--96, ACM, 2005.

Digital Library

[4]

C. J. C. Burges. From RankNet to LambdaRank to LambdaMART: An Overview. Technical Report, 2010.

[5]

C. J. C. Burges, R. Ragno, and Q. V. Le. Learning to Rank with Nonsmooth Cost Functions. In NIPS, Vol. 6, pages 193--200, 2006.

[6]

G. Capannini, C. Lucchese, F. M. Nardini, S. Orlando, R. Perego and N. Tonellotto. Quality versus Efficiency in Document Scoring with Learning-to-Rank Models. In IPM, 52(6):1161--1177, 2016.

Digital Library

[7]

O. Chapelle, T. Joachims, F. Radlinski, and Y. Yue. Large-scale Validation and Analysis of Interleaved Search Evaluation. In TOIS, 30(1):6:1--6:41, 2012.

Digital Library

[8]

D. Dato, C. Lucchese, F. M. Nardini, S. Orlando, R. Perego, N. Tonellotto, and R. Venturini. Fast Ranking with Additive Ensembles of Oblivious and Non-Oblivious Regression Trees. In TOIS, 35(2):15:1--15:31, 2016.

Digital Library

[9]

P. Donmez, K. M. Svore, and C. J. C. Burges. On the Local Optimality of LambdaRank. In SIGIR, pages 460--467, ACM, 2009.

Digital Library

[10]

M. Ferrante, N. Ferro, and M. Maistro. Injecting User Models and Time into Precision via Markov Chains. In SIGIR, pages 597--606, ACM, 2014.

Digital Library

[11]

K. Hofmann, A. Schuth, S. Whiteson, and M. de Rijke. Reusing Historical Interaction Data for Faster Online Learning to Rank for IR. In WSDM, pages 183--192, ACM, 2013.

Digital Library

[12]

T. Liu. Learning to Rank for Information Retrieval. IN FnTIR, 3(3):225--331, 2009.

[13]

C. Lucchese, S. Orlando, R. Perego, F. Silvestri and G. Tolomei. Discovering tasks from search engine query logs. In TOIS, 31(3):14:1--14:43, 2013.

Digital Library

[14]

J. R. Norris. Markov Chains. Cambridge University Press, 1998.

[15]

T. Qin, and T. Liu. Introducing LETOR 4.0 Datasets. In CoRR, 2013.

[16]

A. Schuth, K. Hofmann, S. Whiteson, and M. de Rijke. Lerot: An Online Learning to Rank Framework. In LivingLab, pages 23--26, ACM, 2013.

Digital Library

[17]

P. Serdyukov, N. Craswell, G. Dupret. WSCD2012: Workshop on Web Search Click Data 2012. In WSDM, pages 771--772, ACM, 2012.

[18]

I. Teodorescu. Maximum Likelihood Estimation for Markov Chains. In arXiv preprint arXiv:0905.4131, 2009.

[19]

Q. Wu, Burges, C. J. C. Christopher K. M. Svore, J. Gao. Adapting boosting for information retrieval measures. In Information Retrieval, 13(3):254--270, 2010.

Digital Library

[20]

E. Yilmaz, M. Shokouhi, N. Craswell, and S. Robertson. Expected Browsing Utility for Web Search Evaluation. In CIKM, pages 1561--1565, ACM, 2010.

Digital Library

[21]

Y. Zhang, L. A. F. Park, and A. Moffat. Click-based Evidence for Decaying Weight Distributions in Search Effectiveness Metrics. In Information Retrieval, 13(1):46--69, 2010.

Digital Library

Cited By

Zhang CChen WZhang WXu M(2024)Mitigating the Impact of Inaccurate Feedback in Dynamic Learning-to-Rank: A Study of Overlooked Interesting ItemsACM Transactions on Intelligent Systems and Technology10.1145/3653983Online publication date: 26-Mar-2024
https://doi.org/10.1145/3653983
Yang TLuo CLu HGupta PYin BAi QAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Can Clicks Be Both Labels and Features?Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531948(6-17)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3531948
Maistro M(2019)Exploiting User Signals and Stochastic Models to Improve Information Retrieval Systems and EvaluationACM SIGIR Forum10.1145/3308774.330880552:2(174-175)Online publication date: 17-Jan-2019
https://dl.acm.org/doi/10.1145/3308774.3308805
Show More Cited By

Index Terms

On Including the User Dynamic in Learning to Rank
1. Information systems
  1. Information retrieval

Recommendations

Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm
WWW '19: The World Wide Web Conference

Recently a number of algorithms under the theme of 'unbiased learning-to-rank' have been proposed, which can reduce position bias, the major type of bias in click data, and train a high-performance ranker with click data. Most of the existing algorithms,...
Learning to rank code examples for code search engines

Source code examples are used by developers to implement unfamiliar tasks by learning from existing solutions. To better support developers in finding existing solutions, code search engines are designed to locate and rank code examples relevant to user'...
Bridging the Gap between Click and Relevance for Learning-to-Rank with Minimal Supervision
CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge Management

Recently, unbiased learning-to-rank models have been widely studied to learn a better ranker by eliminating the biases from click data. Toward this goal, existing work mainly focused on estimating the propensity weight to design a specific bias type ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

August 2017

1476 pages

ISBN:9781450350228

DOI:10.1145/3077136

General Chairs:
Noriko Kando
National Institute of Informatics
,
Tetsuya Sakai
Waseda University
,
Hideo Joho
University of Tsukuba
,
Program Chairs:
Hang Li
Huawei Noah's Ark Lab
,
Arjen P. de Vries
Radboud University
,
Ryen W. White
Microsoft Cortana

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 August 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

EC H2020 Program
SID 2016

Conference

SIGIR '17

Sponsor:

SIGIR

SIGIR '17: The 40th International ACM SIGIR conference on research and development in Information Retrieval

August 7 - 11, 2017

Tokyo, Shinjuku, Japan

Acceptance Rates

SIGIR '17 Paper Acceptance Rate 78 of 362 submissions, 22%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
218
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang CChen WZhang WXu M(2024)Mitigating the Impact of Inaccurate Feedback in Dynamic Learning-to-Rank: A Study of Overlooked Interesting ItemsACM Transactions on Intelligent Systems and Technology10.1145/3653983Online publication date: 26-Mar-2024
https://doi.org/10.1145/3653983
Yang TLuo CLu HGupta PYin BAi QAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Can Clicks Be Both Labels and Features?Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531948(6-17)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3531948
Maistro M(2019)Exploiting User Signals and Stochastic Models to Improve Information Retrieval Systems and EvaluationACM SIGIR Forum10.1145/3308774.330880552:2(174-175)Online publication date: 17-Jan-2019
https://dl.acm.org/doi/10.1145/3308774.3308805
Ferro NLucchese CMaistro MPerego R(2019)Boosting learning to rank with user dynamics and continuation methodsInformation Retrieval Journal10.1007/s10791-019-09366-9Online publication date: 5-Nov-2019
https://doi.org/10.1007/s10791-019-09366-9
Perna DInterdonato RTagarelli A(2018)Learning to lurker rank: an evaluation of learning-to-rank methods for lurking behavior analysisSocial Network Analysis and Mining10.1007/s13278-018-0516-z8:1Online publication date: 1-Jun-2018
https://doi.org/10.1007/s13278-018-0516-z
Ferro NLucchese CMaistro MPerego RKamps JKanoulas Ede Rijke MFang HYilmaz E(2017)LEARning Next gEneration Rankers (LEARNER 2017)Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3121050.3121110(331-332)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1145/3121050.3121110
Agosti MDi Nunzio GFerro NMaistro MMarchesin SOrio NPonchia CSilvello G(2017)Thirty Years of Digital Libraries Research at the University of Padua: The User SideDigital Libraries and Multimedia Archives10.1007/978-3-319-73165-0_5(42-54)Online publication date: 21-Dec-2017
https://doi.org/10.1007/978-3-319-73165-0_5

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents