research-article

Joint Optimization of Cascade Ranking Models

Authors:

Luke Gallagher,

Ruey-Cheng Chen,

J. Shane CulpepperAuthors Info & Claims

WSDM '19: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining

Pages 15 - 23

https://doi.org/10.1145/3289600.3290986

Published: 30 January 2019 Publication History

Abstract

Reducing excessive costs in feature acquisition and model evaluation has been a long-standing challenge in learning-to-rank systems. A cascaded ranking architecture turns ranking into a pipeline of multiple stages, and has been shown to be a powerful approach to balancing efficiency and effectiveness trade-offs in large-scale search systems. However, learning a cascade model is often complex, and usually performed stagewise independently across the entire ranking pipeline. In this work we show that learning a cascade ranking model in this manner is often suboptimal in terms of both effectiveness and efficiency. We present a new general framework for learning an end-to-end cascade of rankers using backpropagation. We show that stagewise objectives can be chained together and optimized jointly to achieve significantly better trade-offs globally. This novel approach is generalizable to not only differentiable models but also state-of-the-art tree-based algorithms such as LambdaMART and cost-efficient gradient boosted trees, and it opens up new opportunities for exploring additional efficiency-effectiveness trade-offs in large-scale search systems.

References

[1]

N. Asadi and J. Lin. 2013a. Document Vector Representations for Feature Extraction in Multi-Stage Document Ranking. Inf. Retr., Vol. 16, 6 (2013), 747--768.

Digital Library

[2]

N. Asadi and J. Lin. 2013b. Effectiveness/Efficiency Tradeoffs for Candidate Generation in Multi-Stage Retrieval Architectures. In Proc. SIGIR. 997--1000.

Digital Library

[3]

A. Broder, E. Gabrilovich, V. Josifovski, G. Mavromatis, D. Metzler, and J. Wang. 2010. Exploiting Site-level Information to Improve Web Search. In Proc. CIKM . 1393--1396.

Digital Library

[4]

C. Burges. 2010. From RankNet to LambdaRank to LambdaMart: An overview. Learning, Vol. 11, 23--581 (2010), 81.

[5]

B. B. Cambazoglu, H. Zaragoza, O. Chapelle, J. Chen, C. Liao, Z. Zheng, and J. Degenhardt. 2010. Early Exit Optimizations for Additive Machine Learned Ranking Systems. In Proc. WSDM . 411--420.

Digital Library

[6]

G. Capannini, C. Lucchese, F. M. Nardini, S. Orlando, R. Perego, and N. Tonellotto. 2016. Quality versus efficiency in document scoring with learning-to-rank models. Inf. Proc. & Man., Vol. 52, 6 (2016), 1161--1177.

Digital Library

[7]

O. Chapelle and Y. Chang. 2011. Yahoo! Learning to Rank Challenge Overview. J. Mach. Learn. Res., Vol. 14 (2011), 1--24.

Digital Library

[8]

R-C. Chen, L. Gallagher, R. Blanco, and J. S. Culpepper. 2017. Efficient Cost-Aware Cascade Ranking in Multi-Stage Retrieval. In Proc. SIGIR . 445--454.

Digital Library

[9]

T. Chen and C. Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proc. KDD. 785--794.

Digital Library

[10]

J. S. Culpepper, C. L. A. Clarke, and J. Lin. 2016. Dynamic Cutoff Prediction in Multi-Stage Retrieval Systems. In Proc. ADCS. 17--24.

Digital Library

[11]

D. Dato, C. Lucchese, F. M. Nardini, S. Orlando, R. Perego, N. Tonellotto, and R. Venturini. 2016. Fast ranking with additive ensembles of oblivious and non-oblivious regression trees. ACM Trans. Information Systems, Vol. 35, 2 (2016), 15.1--15.31.

Digital Library

[12]

B. T. Dinccer, C. Macdonald, and I. Ounis. 2016. Risk-sensitive evaluation and learning to rank using multiple baselines. In Proc. SIGIR . 483--492.

Digital Library

[13]

B. T. Dincc er, C. Macdonald, and I. Ounis. 2014. Hypothesis testing for the risk-sensitive evaluation of retrieval systems. In Proc. SIGIR. 23--32.

Digital Library

[14]

M. M. Dundar and J. Bi. 2007. Joint Optimization of Cascaded Classifiers for Computer Aided Detection. In IEEE Conf. on Comp. Vis. and Pat. Recog. 1--8.

[15]

J. Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics (2001), 1189--1232.

[16]

X. Jin, T. Yang, and X. Tang. 2016. A Comparison of Cache Blocking Methods for Fast Execution of Ensemble-based Score Computation. In Proc. SIGIR. 629--638.

Digital Library

[17]

G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T-Y. Liu. 2017. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In Proc. NeurIPS . 3149--3157.

Digital Library

[18]

R. Kohavi, A. Deng, B. Frasca, T. Walker, Y. Xu, and N. Pohlmann. 2013. Online controlled experiments at large scale. In Proc. KDD. 1168--1176.

Digital Library

[19]

Y. LeCun, L. Bottou, G. B. Orr, and K-R. Müller. 1998. Efficient backprop. In Neural networks: Tricks of the trade . Springer, 9--50.

Digital Library

[20]

S. Liu, F. Xiao, W. Ou, and L. Si. 2017. Cascade Ranking for Operational E-commerce Search. In Proc. KDD. 1557--1565.

Digital Library

[21]

T.-Y. Liu. 2009. Learning to Rank for Information Retrieval. Foundations and Trends in Information Retrieval, Vol. 3, 3 (2009), 225--331.

Digital Library

[22]

C. Lucchese, F. M. Nardini, S. Orlando, R. Perego, F. Silvestri, and S. Trani. 2016a. Post-learning optimization of tree ensembles for efficient ranking. In Proc. SIGIR . 949--952.

Digital Library

[23]

C. Lucchese, F. M. Nardini, S. Orlando, R. Perego, N. Tonellotto, and R. Venturini. 2015. QuickScorer: A Fast Algorithm to Rank Documents with Additive Ensembles of Regression Trees. In Proc. SIGIR. 73--82.

Digital Library

[24]

C. Lucchese, F. M. Nardini, S. Orlando, R. Perego, N. Tonellotto, and R. Venturini. 2016b. Exploiting CPU SIMD extensions to speed-up document scoring with tree ensembles. In Proc. SIGIR. 833--836.

Digital Library

[25]

C. Lucchese, F. M. Nardini, S. Orlando, R. Perego, and S. Trani. 2017. X-DART: Blending Dropout and Pruning for Efficient Learning to Rank. In Proc. SIGIR . 1077--1080.

Digital Library

[26]

C. Lucchese, F. M. Nardini, R. Perego, S. Orlando, and S. Trani. 2018. Selective Gradient Boosting for Effective Learning to Rank. In Proc. SIGIR . 155--164.

Digital Library

[27]

C. Macdonald, R. L. T. Santos, and I. Ounis. 2013a. The whens and hows of learning to rank for web search. Inf. Retr., Vol. 16, 5 (2013), 584--628.

Digital Library

[28]

C. Macdonald, R. L. T. Santos, I. Ounis, and B. He. 2013b. About learning models with multiple query-dependent features. ACM Trans. Information Systems, Vol. 31, 3 (2013), 11:1--11:39.

Digital Library

[29]

J. Mackenzie, J. S. Culpepper, R. Blanco, M. Crane, C. L. A. Clarke, and J. Lin. 2018. Query Driven Algorithm Selection in Early Stage Retrieval. In Proc. WSDM . 396--404.

Digital Library

[30]

I. Matveeva, C. Burges, T. Burkard, A. Laucius, and L. Wong. 2006. High Accuracy Retrieval with Multiple Nested Ranker. In Proc. SIGIR . 437--444.

Digital Library

[31]

H. R. Mohammad, K. Xu, J. Callan, and J. S. Culpepper. 2018. Dynamic Shard Cutoff Prediction for Selective Search. In Proc. SIGIR. 85--94.

Digital Library

[32]

V. Nair and G. E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proc. ICML . 807--814.

Digital Library

[33]

S. Peter, F. Diego, F. A. Hamprecht, and B. Nadler. 2017. Cost Efficient Gradient Boosting. In Proc. NeurIPS . 1550--1560.

Digital Library

[34]

C. Pölitz and R. Schenkel. 2011. Learning to rank under tight budget constraints. In Proc. SIGIR. 1173--1174.

Digital Library

[35]

T. Qin and T-Y. Liu. 2013. Introducing LETOR 4.0 Datasets. CoRR (2013). http://arxiv.org/abs/1306.2597

[36]

K. V. Rashmi and R. Gilad-Bachrach. 2015. DART: Dropouts meet Multiple Additive Regression Trees. J. Mach. Learn. Res., Vol. 38 (2015), 489--497.

[37]

V. C. Raykar, B. Krishnapuram, and S. Yu. 2010. Designing efficient cascaded classifiers: tradeoff between accuracy and cost. In Proc. KDD . 853--860.

Digital Library

[38]

V. N. Vapnik. 1995. The Nature of Statistical Learning Theory .Springer-Verlag New York, Inc., New York, NY, USA.

Digital Library

[39]

L. Wang, P. N. Bennett, and K. Collins-Thompson. 2012. Robust ranking models via risk-sensitive optimization. In Proc. SIGIR . 761--770.

Digital Library

[40]

L. Wang, J. Lin, and D. Metzler. 2010. Learning to efficiently rank. In Proc. SIGIR. 138--145.

Digital Library

[41]

L. Wang, J. Lin, and D. Metzler. 2011. A Cascade Ranking Model for Efficient Ranked Retrieval. In Proc. SIGIR . 105--114.

Digital Library

[42]

Z. Xu, M. J. Kusner, K. Q. Weinberger, and M. Chen. 2013. Cost-Sensitive Tree of Classifiers. In Proc. ICML. 133--141.

Digital Library

[43]

Z. Xu, M. J. Kusner, K. Q. Weinberger, M. Chen, and O. Chapelle. 2014. Classifier Cascades and Trees for Minimizing Feature Evaluation Cost. J. Mach. Learn. Res., Vol. 15 (2014), 2113--2144.

Digital Library

[44]

Z. Xu, K. Q. Weinberger, and O. Chapelle. 2012. The Greedy Miser: Learning Under Test-time Budgets. In Proc. ICML . 1175--1182.

Digital Library

Cited By

Wei JZhou YWu ZLiu ZBaeza-Yates RBonchi F(2024)Enhancing Pre-Ranking Performance: Tackling Intermediary Challenges in Multi-Stage Cascading Recommendation SystemsProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671580(5950-5958)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671580
Fu CWang KWu JChen YHuzhang GNi YZeng AZhou ZBaeza-Yates RBonchi F(2024)Residual Multi-Task Learner for Applied RankingProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671523(4974-4985)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671523
Zheng KZhao HHuang RZhang BMou NNiu YSong YWang HGai KChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Full Stage Learning to Rank: A Unified Framework for Multi-Stage SystemsProceedings of the ACM Web Conference 202410.1145/3589334.3645523(3621-3631)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645523
Show More Cited By

Index Terms

Joint Optimization of Cascade Ranking Models
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking
      1. Learning to rank

Recommendations

Efficient Cost-Aware Cascade Ranking in Multi-Stage Retrieval
SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

Complex machine learning models are now an integral part of modern, large-scale retrieval systems. However, collection size growth continues to outpace advances in efficiency improvements in the learning models which achieve the highest effectiveness. ...
A cascade ranking model for efficient ranked retrieval
SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval

There is a fundamental tradeoff between effectiveness and efficiency when designing retrieval models for large-scale document collections. Effectiveness tends to derive from sophisticated ranking functions, such as those constructed using learning to ...
Cascade Ranking for Operational E-commerce Search
KDD '17: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

In the 'Big Data' era, many real-world applications like search involve the ranking problem for a large number of items. It is important to obtain effective ranking results and at the same time obtain the results efficiently in a timely manner for ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WSDM '19: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining

January 2019

874 pages

ISBN:9781450359405

DOI:10.1145/3289600

General Chairs:
J. Shane Culpepper
RMIT University
,
Alistair Moffat
The University of Melbourne
,
Program Chairs:
Paul N. Bennett
Microsoft
,
Kristina Lerman
University of Southern California

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 January 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

WSDM '19

Sponsor:

WSDM '19: The Twelfth ACM International Conference on Web Search and Data Mining

February 11 - 15, 2019

Melbourne VIC, Australia

Acceptance Rates

WSDM '19 Paper Acceptance Rate 84 of 511 submissions, 16%;

Overall Acceptance Rate 498 of 2,863 submissions, 17%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
507
Total Downloads

Downloads (Last 12 months)33
Downloads (Last 6 weeks)2

Reflects downloads up to 11 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wei JZhou YWu ZLiu ZBaeza-Yates RBonchi F(2024)Enhancing Pre-Ranking Performance: Tackling Intermediary Challenges in Multi-Stage Cascading Recommendation SystemsProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671580(5950-5958)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671580
Fu CWang KWu JChen YHuzhang GNi YZeng AZhou ZBaeza-Yates RBonchi F(2024)Residual Multi-Task Learner for Applied RankingProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671523(4974-4985)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671523
Zheng KZhao HHuang RZhang BMou NNiu YSong YWang HGai KChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Full Stage Learning to Rank: A Unified Framework for Multi-Stage SystemsProceedings of the ACM Web Conference 202410.1145/3589334.3645523(3621-3631)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645523
Lyu LRoy NOosterhuis HAnand A(2024)Is Interpretable Machine Learning Effective at Feature Selection for Neural Learning-to-Rank?Advances in Information Retrieval10.1007/978-3-031-56066-8_29(384-402)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56066-8_29
Huang XLian DChen JZheng LXie XChen E(2023)Cooperative Retriever and Ranker in Deep RecommendersProceedings of the ACM Web Conference 202310.1145/3543507.3583422(1150-1161)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583422
Lian DHuang XChen XChen JWang XWang YJin HFan RLiu ZWu LChen EChen HDuh WHuang HKato MMothe JPoblete B(2023)RecStudio: Towards a Highly-Modularized Recommender SystemProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591894(2890-2900)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591894
Busolin FLucchese CNardini FOrlando SPerego RTrani S(2023)Early Exit Strategies for Learning-to-Rank CascadesIEEE Access10.1109/ACCESS.2023.333108811(126691-126704)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3331088
Salamat SArabzadeh NZarrinkalam FZihayat MBagheri E(2023)Learning Query-Space Document Representations for High-Recall RetrievalAdvances in Information Retrieval10.1007/978-3-031-28238-6_51(599-607)Online publication date: 2-Apr-2023
https://dl.acm.org/doi/10.1007/978-3-031-28238-6_51
Zamani HBendersky MMetzler DZhuang HWang XCrestani FPasi GGaussier E(2022)Stochastic Retrieval-Conditioned RerankingProceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3539813.3545141(81-91)Online publication date: 23-Aug-2022
https://dl.acm.org/doi/10.1145/3539813.3545141
Shehata DArabzadeh NClarke CAl Hasan MXiong L(2022)Early Stage Sparse Retrieval with Entity LinkingProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557588(4464-4469)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557588
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents