article

Learning to order things

Authors:

William W. Cohen,

Robert E. Schapire, and

Yoram SingerAuthors Info & Claims

Journal of Artificial Intelligence Research, Volume 10, Issue 1

Pages 243 - 270

Published: 01 May 1999 Publication History

Abstract

There are many applications in which it is desirable to order rather than classify instances. Here we consider the problem of learning how to order instances given feedback in the form of preference judgments, i.e., statements to the effect that one instance should be ranked ahead of another. We outline a two-stage approach in which one first learns by conventional means a binary preference function indicating whether it is advisable to rank one instance before another. Here we consider an on-line algorithm for learning preference functions that is based on Freund and Schapire's "Hedge" algorithm. In the second stage, new instances are ordered so as to maximize agreement with the learned preference function. We show that the problem of finding the ordering that agrees best with a learned preference function is NP-complete. Nevertheless, we describe simple greedy algorithms that are guaranteed to find a good approximation. Finally, we show how metasearch can be formulated as an ordering problem, and present experimental results on learning a combination of "search experts," each of which is a domain-specific query expansion strategy for a web search engine.

References

[1]

Balabanovíc, M., & Shoham, Y. (1997). FAB: Content-based, collaborative recommendation. Communications of the ACM, 40(3), 66-72.

Digital Library

[2]

Bartell, B., Cottrell, G., & Belew, R. (1994). Automatic combination of multiple ranked retrieval systems. In Seventeenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.

[3]

Bartholdi, J., Tovey, C., & Trick, M. (1989). Voting schemes for which it can be difficult to tell who won the elections. Social Choice and Welfare, 6, 157-165.

[4]

Berger, B., & Shor, P. (1997). Tight bounds for the acyclic subgraph problem. Journal of Algorithms, 25, 1-18.

Digital Library

[5]

Black, D. (1958). Theory of Committees and Elections. Cambridge University Press.

[6]

Boyan, J., Freitag, D., & Joachims, T. (1994). A machine learning architecture for optimizing web search engines. Tech. rep. WS-96-05, American Association of Artificial Intelligence.

[7]

Caruana, R., Baluja, S., & Mitchell, T. (1996). Using the future to 'Sort Out' the present: Rankprop and multitask learning for medical risk evaluation. In Advances in Neural Information Processing Systems (NIPS) 8.

[8]

Cooper, W. (1968). Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems. American Documentation, 19, 30-41.

[9]

Dodgson, C. (1876). A method for taking votes on more than two issues. Clarendon Press, Oxford. Reprinted with discussion in (Black, 1958).

[10]

Etzioni, O., Hanks, S., Jiang, T., Karp, R. M., Madani, O., & Waarts, O. (1996). Efficient information gathering on the internet. In Proceedings of the 37th Annual Symposium on Foundations of Computer Science (FOCS-96) Burlington, Vermont. IEEE Computer Society Press.

[11]

Even, G., Naor, J., Rao, S., & Schieber, B. (1996). Divide-and-conquer approximation algorithms via spreading metrics. In 36th Annual Symposium on Foundations of Computer Science (FOCS-96), pp. 62-71 Burlington, Vermont. IEEE Computer Society Press.

Digital Library

[12]

Even, G., Naor, J., Schieber, B., & Sudan, M. (1998). Approximating minimum feedback sets and multicuts in directed graphs. Algorithmica, 20(2), 151-174.

[13]

Fishburn, F. (1970). Utility Theory for Decision Making. Wiley, New York.

[14]

French, S. (1989). Decision Theory: An Introduction to the Mathematics of Rationality. Ellis Horwood Series in Mathematics and Its Applications.

[15]

Freund, Y., Iyer, R., Schapire, R., & Singer, Y. (1998). An efficient boosting algorithm for combining preferences. In Machine Learning: Proceedings of the Fifteenth International Conference.

[16]

Freund, Y., & Schapire, R. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1), 119-139.

Digital Library

[17]

Galil, Z., & Megido, N. (1977). Cyclic ordering is NP-complete. Theoretical Computer Science, 5, 179-182.

[18]

Gary, M., & Johnson, D. (1979). Computers and Intractability: A Guide to the Theory of NP-completeness. W. H. Freeman and Company, New York.

[19]

Kantor, P. (1994). Decision level data fusion for routing of documents in the TREC3 context: a best case analysis of worst case results. In Proceedings of the third text retrieval conference (TREC-3).

[20]

Kemeny, J. (1959). Mathematics without numbers. Daedalus, 88, 571-591.

[21]

Kemeny, J., & Snell, J. (1962). Mathematical Models in the Social Sciences. Blaisdell, New York.

[22]

Littlestone, N. (1988). Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine Learning, 2(4).

[23]

Littlestone, N., & Warmuth, M. (1994). The weighted majority algorithm. Information and Computation, 108(2), 212-261.

Digital Library

[24]

Lochbaum, K., & Streeter, L. (1989). Comparing and combining the effectiveness of latent semantic indexing and the ordinary vector space model for information retrieval. Information processing and management, 25(6), 665-676.

[25]

Resnick, P., & Varian, H. (1997). Introduction to special section on Recommender Systems. Communication of the ACM, 40(3).

Digital Library

[26]

Roberts, F. (1979). Measurement theory with applications to decision making, utility, and social sciences. Addison Wesley, Reading, MA.

[27]

Salton, G., & McGill, M. (1983). Introduction to Modern Information Retrieval. McGraw-Hill.

[28]

Seymour, P. (1995). Packing directed circuits fractionally. Combinatorica, 15, 281-288.

[29]

Shakes, J., Langheinrich, M., & Etzioni, O. (1997). Dynamic reference sifting: a case study in the homepage domain. In Proceedings of WWW6.

[30]

Shmoys, D. (1997). Cut problems and their application to divide-and-conquer. In Hochbaum, D. (Ed.), Approximation algorithms for NP-Hard Problems. PWS Publishing Company, New York.

[31]

Towell, G., Voorhees, E., Gupta, N., & Johnson-Laird, B. (1995). Learning collection fusion strategies for information retrieval. In Machine Learning: Proceedings of the Twelfth International Conference Lake Taho, California. Morgan Kaufmann.

[32]

Utgoff, P., & Clouse, J. (1991). Two kinds of training information for evaluation function learning. In Proceedings of the Ninth National Conference on Artificial Intelligence (AAAI-91), pp. 596-600 Cambridge, MA. AAAI Press/MIT PRess.

[33]

Utgoff, P., & Saxena, S. (1987). Learning a preference predicate. In Proceedings of the Fourth International Workshop on Machine Learning, pp. 115-121 San Francisco, CA. Morgan Kaufmann.

[34]

Voorhees, E., Gupta, N., & Johnson-Laird, B. (1994). The collection fusion problem. In Seventeenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.

[35]

Yao, Y. (1995). Measuring retrieval effectiveness based on user preference of documents. Journal of the American Society for Information Science, 46(2), 133-145.

Cited By

Zhou YHao JLi Z(2024)Heuristic Search for Rank Aggregation with Application to Label RankingINFORMS Journal on Computing10.1287/ijoc.2022.001936:2(308-326)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1287/ijoc.2022.0019
Pérez-Rodríguez RHernández-Aguirre A(2019)A hybrid estimation of distribution algorithm for the vehicle routing problem with time windowsComputers and Industrial Engineering10.1016/j.cie.2019.02.017130:C(75-96)Online publication date: 1-Apr-2019
https://dl.acm.org/doi/10.1016/j.cie.2019.02.017
Ahmadi Fahandar MHüllermeier E(2019)Analogy-Based Preference Learning with KernelsKI 2019: Advances in Artificial Intelligence10.1007/978-3-030-30179-8_3(34-47)Online publication date: 23-Sep-2019
https://dl.acm.org/doi/10.1007/978-3-030-30179-8_3
Show More Cited By

Recommendations

Learning to order things
NIPS'97: Proceedings of the 10th International Conference on Neural Information Processing Systems

There are many applications in which it is desirable to order rather than classify instances. Here we consider the problem of learning how to order, given feedback in the form of preference judgments, i.e., statements to the effect that one instance ...
Read More
Problem order implications for learning transfer
ITS'12: Proceedings of the 11th international conference on Intelligent Tutoring Systems

The order of problems presented to students is an important variable that affects learning effectiveness. Previous studies have shown that solving problems in a blocked order, in which all problems of one type are completed before the student is ...
Read More
Learning in order to reason
Read More

Comments

Information & Contributors

Information

Published In

cover image Journal of Artificial Intelligence Research

Journal of Artificial Intelligence Research Volume 10, Issue 1

January 1999

469 pages

ISSN:1076-9757

Issue’s Table of Contents

Publisher

AI Access Foundation

El Segundo, CA, United States

Publication History

Published: 01 May 1999

Received: 01 October 1998

Published in JAIR Volume 10, Issue 1

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

205
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Other Metrics

View Author Metrics

Citations

Cited By

Zhou YHao JLi Z(2024)Heuristic Search for Rank Aggregation with Application to Label RankingINFORMS Journal on Computing10.1287/ijoc.2022.001936:2(308-326)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1287/ijoc.2022.0019
Pérez-Rodríguez RHernández-Aguirre A(2019)A hybrid estimation of distribution algorithm for the vehicle routing problem with time windowsComputers and Industrial Engineering10.1016/j.cie.2019.02.017130:C(75-96)Online publication date: 1-Apr-2019
https://dl.acm.org/doi/10.1016/j.cie.2019.02.017
Ahmadi Fahandar MHüllermeier E(2019)Analogy-Based Preference Learning with KernelsKI 2019: Advances in Artificial Intelligence10.1007/978-3-030-30179-8_3(34-47)Online publication date: 23-Sep-2019
https://dl.acm.org/doi/10.1007/978-3-030-30179-8_3
Pandey GWang SRen ZChang Y(2019)Vectors of Pairwise Item PreferencesAdvances in Information Retrieval10.1007/978-3-030-15712-8_21(323-336)Online publication date: 14-Apr-2019
https://dl.acm.org/doi/10.1007/978-3-030-15712-8_21
Fahandar MHüllermeier EMcIlraith SWeinberger K(2018)Learning to rank based on analogical reasoningProceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence10.5555/3504035.3504395(2951-2958)Online publication date: 2-Feb-2018
https://dl.acm.org/doi/10.5555/3504035.3504395
Schäfer DHüllermeier E(2018)Dyad ranking using Plackett---Luce models based on joint feature representationsMachine Language10.1007/s10994-017-5694-9107:5(903-941)Online publication date: 1-May-2018
https://dl.acm.org/doi/10.1007/s10994-017-5694-9
Pérez-Rodríguez RHernández-Aguirre A(2018)A hybrid estimation of distribution algorithm for flexible job-shop scheduling problems with process plan flexibilityApplied Intelligence10.1007/s10489-018-1160-z48:10(3707-3734)Online publication date: 1-Oct-2018
https://dl.acm.org/doi/10.1007/s10489-018-1160-z
Dey PMisra NNarahari YLarson KWinikoff MDas SDurfee E(2017)Parameterized Dichotomy of Choosing Committees Based on Approval Votes in the Presence of OutliersProceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems10.5555/3091125.3091137(42-50)Online publication date: 8-May-2017
https://dl.acm.org/doi/10.5555/3091125.3091137
Joachims TGranka LPan BHembrooke HGay G(2017)Accurately Interpreting Clickthrough Data as Implicit FeedbackACM SIGIR Forum10.1145/3130332.313033451:1(4-11)Online publication date: 2-Aug-2017
https://dl.acm.org/doi/10.1145/3130332.3130334
Huang JWang JZhong NKando NSakai TJoho HLi Hde Vries AWhite R(2017)A Poisson Regression Method for Top-N RecommendationProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3077136.3080670(885-888)Online publication date: 7-Aug-2017
https://dl.acm.org/doi/10.1145/3077136.3080670
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents