research-article

Open access

Multi-Label Learning to Rank through Multi-Objective Optimization

Authors:

Debabrata Mahapatra,

Chaosheng Dong,

Michinari MommaAuthors Info & Claims

KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Pages 4605 - 4616

https://doi.org/10.1145/3580305.3599870

Published: 04 August 2023 Publication History

Abstract

Learning to Rank (LTR) technique is ubiquitous in Information Retrieval systems, especially in search ranking applications. The relevance labels used to train ranking models are often noisy measurements of human behavior, such as product ratings in product searches. This results in non-unique ground truth rankings and ambiguity. To address this, Multi-Label LTR (MLLTR) is used to train models using multiple relevance criteria, capturing conflicting but important goals, such as product quality and purchase likelihood for improved revenue in product searches. This research leverages Multi-Objective Optimization (MOO) in MLLTR and employs modern MOO algorithms to solve the problem. A general framework is proposed to combine label information to characterize trade-offs among goals, and allows for the use of gradient-based MOO algorithms. We test the proposed framework on four publicly available LTR datasets and one E-commerce dataset to show its efficacy.

Supplementary Material

MOV File (adfp227-2min-promo.mov)

This video provides a brief overview of our work on "Multi-Label Learning to Rank through Multi-Objective Optimization", where we develop a general framework to train and update ML models for ranking a list of items based on multiple criteria.

Download
43.17 MB

References

[1]

Anne Auger, Johannes Bader, Dimo Brockhoff, and Eckart Zitzler. 2009. Theory of the hypervolume indicator: optimal ??-distributions and the choice of the reference point. In Proceedings of the Tenth ACM SIGEVO Workshop on Founda- tions of Genetic Algorithms (FOGA '09). Association for Computing Machinery, Orlando, Florida, USA, 87--102.

Digital Library

[2]

Stephen Boyd and Lieven Vandenberghe. 2004. Convex Optimization. Cam- bridge University Press.

[3]

Sebastian Bruch. 2021. An alternative cross entropy loss for learning-to-rank. In Proceedings of the Web Conference 2021 (WWW '21). Association for Computing Machinery, Ljubljana, Slovenia, 118--126.

Digital Library

[4]

C. Burges. 2010. From ranknet to lambdarank to lambdamart: an overview. In.

[5]

Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamil- ton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, 89--96.

Digital Library

[6]

David Carmel, Elad Haramaty, Arnon Lazerson, and Liane Lewin-Eytan. 2020. Multi-objective ranking optimization for product search using stochastic label aggregation. In Proceedings of The Web Conference 2020 (WWW '20). Associa- tion for Computing Machinery, Taipei, Taiwan, 373--383.

Digital Library

[7]

Olivier Chapelle and Yi Chang. 2011. Yahoo! learning to rank challenge overview. In Proceedings of the Learning to Rank Challenge (Proceedings of Machine Learn- ing Research). Olivier Chapelle, Yi Chang, and Tie-Yan Liu, (Eds.) Vol. 14. PMLR, Haifa, Israel, 1--24.

[8]

Na Dai, Milad Shokouhi, and Brian D. Davison. 2011. Learning to rank for freshness and relevance. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '11). Association for Computing Machinery, Beijing, China, 95--104.

[9]

Onkar Dalal, Srinivasan H. Sengemedu, and Subhajit Sanyal. 2012. Multi- objective ranking of comments on web. In Proceedings of the 21st Interna- tional Conference on World Wide Web (WWW '12). Association for Computing Machinery, Lyon, France, 419--428.

Digital Library

[10]

Domenico Dato, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. 2016. Fast ranking with additive ensembles of oblivious and non-oblivious regression trees. ACM Transactions on Information Systems (TOIS).

[11]

Chaosheng Dong and Bo Zeng. 2020. Expert learning through generalized inverse multiobjective optimization: models, insights, and algorithms. In Inter- national Conference on Machine Learning.

[12]

Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics, 1189--1232.

[13]

Chengyue Gong, Xingchao Liu, and qiang liu. 2021. Automatic and harmless regularization with constrained and lexicographic optimization: a dynamic barrier approach. In Advances in Neural Information Processing Systems.

[14]

Changsung Kang, Xuanhui Wang, Yi Chang, and Belle Tseng. 2012. Learning to rank with multi-aspect relevance for vertical search. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining (WSDM '12). Association for Computing Machinery, Seattle, Washington, USA, 453--462.

Digital Library

[15]

Shubhra Kanti Karmaker Santu, Parikshit Sondhi, and ChengXiang Zhai. 2017. On application of learning to rank for e-commerce search. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '17). Association for Computing Machinery, Shinjuku, Tokyo, Japan, 475--484.

Digital Library

[16]

Chang Li, Haoyun Feng, and Maarten de Rijke. 2020. Cascading hybrid ban- dits: online learning to rank for relevance and diversity. In Fourteenth ACM Conference on Recommender Systems.

Digital Library

[17]

Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. 2019. Pareto multi-task learning. In Advances in Neural Information Processing Sys- tems. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, (Eds.) Vol. 32. Curran Associates, Inc.

[18]

Xiao Lin, Hongjie Chen, Changhua Pei, Fei Sun, Xuanji Xiao, Hanxiao Sun, Yongfeng Zhang, Wenwu Ou, and Peng Jiang. 2019. A pareto-efficient algo- rithm for multiple objective optimization in e-commerce recommendation. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys '19). Association for Computing Machinery, Copenhagen, Denmark, 20--28.

Digital Library

[19]

Bo Long, Jiang Bian, Anlei Dong, and Yi Chang. 2012. Enhancing product search by best-selling prediction in e-commerce. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management (CIKM '12). Association for Computing Machinery, Maui, Hawaii, USA, 2479--2482.

Digital Library

[20]

Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Fabrizio Silvestri, and Salvatore Trani. 2016. Post-learning optimization of tree ensembles for efficient ranking. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval.

Digital Library

[21]

Debabrata Mahapatra, Chaosheng Dong, and Michinari Momma. 2023. Query- wise fair learning to rank through multi-objective optimization. In Proceedings of the 29th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.

[22]

Debabrata Mahapatra and Vaibhav Rajan. 2021. Exact pareto optimal search for multi-task learning: touring the pareto front. (2021). arXiv: 2108.00597

[23]

.

[24]

Debabrata Mahapatra and Vaibhav Rajan. 2020. Multi-task learning with user preferences: gradient descent with controlled ascent in pareto optimization. In Proceedings of the 37th International Conference on Machine Learning.

[25]

Kaisa Miettinen. 1998. Nonlinear multiobjective optimization. International series in operations research and management science. Vol. 12. Kluwer.

[26]

Michinari Momma, Chaosheng Dong, and Jia Liu. 2022. A multi-objective / multi-task learning framework induced by pareto stationarity. In Proceedings of the 39th International Conference on Machine Learning.

[27]

Michinari Momma, Ali Bagheri Garakani, Nanxun Ma, and Yi Sun. 2020. Multi- objective relevance ranking via constrained optimization. In The Web Confer- ence 2020.

[28]

Michinari Momma, Alireza Bagheri Garakani, and Yi Sun. 2019. Multi-objective relevance ranking. In eCOM@ SIGIR.

[29]

Marco Morik, Ashudeep Singh, Jessica Hong, and Thorsten Joachims. 2020. Controlling fairness and bias in dynamic learning-to-rank. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval.

Digital Library

[30]

Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 datasets. CoRR, abs/ 1306.2597.

[31]

Sebastian Ruder. 2016. An overview of gradient descent optimization algo- rithms. CoRR, abs/1609.04747. arXiv: 1609.04747.

[32]

Ozan Sener and Vladlen Koltun. 2018. Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, (Eds.) Vol. 31. Curran Associates, Inc.

[33]

Ashudeep Singh and Thorsten Joachims. 2019. Policy learning for fairness in ranking. Advances in Neural Information Processing Systems, 32.

[34]

Daria Sorokina and Erick Cantu-Paz. 2016. Amazon search: the joy of ranking products. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '16). Association for Computing Machinery, Pisa, Italy, 459--460.

Digital Library

[35]

Bogdana Stanojevi? and Fred Glover. 2020. A new approach to generate pattern- efficient sets of non-dominated vectors for multi-objective optimization. Infor- mation Sciences, 530, 22--42.

[36]

Krysta M. Svore, Maksims N. Volkovs, and Christopher J.C. Burges. 2011. Learning to rank with multiple objective functions. In Proceedings of the 20th International Conference on World Wide Web (WWW '11). Association for Computing Machinery, Hyderabad, India, 367--376.

[37]

Joost van Doorn, Daan Odijk, Diederik M Roijers, and Maarten de Rijke. 2016. Balancing relevance criteria through multi-objective optimization. In Proceed- ings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, 769--772.

[38]

2019. Painless stochastic gradient: interpolation, line-search, and convergence rates. Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA, 14 pages.

[39]

Lidan Wang, Paul N. Bennett, and Kevyn Collins-Thompson. 2012. Robust ranking models via risk-sensitive optimization. In Proceedings of the 35th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '12). Association for Computing Machinery, Portland, Oregon, USA, 761--770.

Digital Library

[40]

Xiaojie Wang, Ruoyuan Gao, Anoop Jain, Graham Edge, and Sachin Ahuja. 2023. How well do offline metrics predict online performance of product rank- ing models? In Proceedings of the 46th International ACM SIGIR conference on Research and Development in Information Retrieval.

Digital Library

[41]

Le Yan, Zhen Qin, Xuanhui Wang, Mike Bendersky, and Marc Najork. 2022. Scale calibration of deep ranking models. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Digital Library

Cited By

Xiang ZZhao HZhao CHe MFan JBaeza-Yates RBonchi F(2024)Performative Debias with Fair-exposure Optimization Driven by Strategic Agents in Recommender SystemsProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671786(3507-3517)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671786
Tang JGao HHe LKatariya SBaeza-Yates RBonchi F(2024)Multi-objective Learning to Rank by Model DistillationProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671597(5783-5792)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671597
Ismail WIbrahim OAlsalamah HMohamed E(2023)Multiobjective Learning to Rank Based on the (1 + 1) Evolutionary Strategy: An Evaluation of Three Novel Pareto Optimal MethodsElectronics10.3390/electronics1217372412:17(3724)Online publication date: 4-Sep-2023
https://doi.org/10.3390/electronics12173724

Index Terms

Multi-Label Learning to Rank through Multi-Objective Optimization
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Multi-task learning
2. Information systems
  1. Information retrieval
    1. Retrieval models and ranking
      1. Learning to rank

Recommendations

Querywise Fair Learning to Rank through Multi-Objective Optimization
KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

In Learning-to-Rank (LTR) problems, the task of delivering relevant search results and allocating fair exposure to items of a protected group can conflict. Previous works in Fair LTR have attempted to resolve this by combining the objectives of relevant ...
Multi-objective optimization in learning to rank
SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval

Supervised learning to rank algorithms typically optimize for high relevance and ignore other facets of search quality, such as freshness and diversity. Prior work on multi-objective ranking trained rankers focused on using hybrid labels that combine ...
Multi-Label Classification Based on Multi-Objective Optimization
Special Issue on Linking Social Granularity and Functions

Multi-label classification refers to the task of predicting potentially multiple labels for a given instance. Conventional multi-label classification approaches focus on single objective setting, where the learning algorithm optimizes over a single ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 2023

5996 pages

ISBN:9798400701030

DOI:10.1145/3580305

General Chairs:
Ambuj Singh
UC Santa Barbara, USA
,
Yizhou Sun
UC Los Angeles, USA
,
Program Chairs:
Leman Akoglu
Carnegie Mellon University, USA
,
Dimitrios Gunopulos
University of Athens, Greece
,
Xifeng Yan
UC Santa Barbara, USA
,
Ravi Kumar
Google, USA
,
Fatma Ozcan
Google, USA
,
Jieping Ye
Alibaba DAMO Academy

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 August 2023

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Research Foundation Singapore

Conference

KDD '23

Sponsor:

KDD '23: The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 6 - 10, 2023

CA, Long Beach, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
969
Total Downloads

Downloads (Last 12 months)969
Downloads (Last 6 weeks)62

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xiang ZZhao HZhao CHe MFan JBaeza-Yates RBonchi F(2024)Performative Debias with Fair-exposure Optimization Driven by Strategic Agents in Recommender SystemsProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671786(3507-3517)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671786
Tang JGao HHe LKatariya SBaeza-Yates RBonchi F(2024)Multi-objective Learning to Rank by Model DistillationProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671597(5783-5792)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671597
Ismail WIbrahim OAlsalamah HMohamed E(2023)Multiobjective Learning to Rank Based on the (1 + 1) Evolutionary Strategy: An Evaluation of Three Novel Pareto Optimal MethodsElectronics10.3390/electronics1217372412:17(3724)Online publication date: 4-Sep-2023
https://doi.org/10.3390/electronics12173724

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents