Article

High accuracy retrieval with multiple nested ranker

Authors:

Irina Matveeva,

Leon WongAuthors Info & Claims

SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 437 - 444

https://doi.org/10.1145/1148170.1148246

Published: 06 August 2006 Publication History

Abstract

High precision at the top ranks has become a new focus of research in information retrieval. This paper presents the multiple nested ranker approach that improves the accuracy at the top ranks by iteratively re-ranking the top scoring documents. At each iteration, this approach uses the RankNet learning algorithm to re-rank a subset of the results. This splits the problem into smaller and easier tasks and generates a new distribution of the results to be learned by the algorithm. We evaluate this approach using different settings on a data set labeled with several degrees of relevance. We use the normalized discounted cumulative gain (NDCG) to measure the performance because it depends not only on the position but also on the relevance score of the document in the ranked list. Our experiments show that making the learning algorithm concentrate on the top scoring results improves precision at the top ten documents in terms of the NDCG score.

References

[1]

E. B. Baum and F. Wilczek. Supervised learning of probability distributions by neural networks. Neural Information Processing Systems, pages 52--61, 1987.

[2]

C. J. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In Proceedings of ICML, 2005.

Digital Library

[3]

Y. Freund. Boosting a weak learning algorithm by majority. Information and Computation, pages 121(2):256--285, 1995.

Digital Library

[4]

Y. Freund, R. Iyer, and R. Shapire. An efficient boosting algorithm for combining preferences. In Journal of Machine Learning Research, volume 4, pages 933--969, 2003.

Digital Library

[5]

D. He and D. Demner-Fushman. HARD experiment at Maryland: From need negotiation to automated HARD process. In Proceedings of TREC, 2003.

[6]

X. He and P. Niyogi. Locality preserving projections. In Proceedings of NIPS, 2003.

[7]

A. James. HARD track overview in TREC 2003. In Proceedings of TREC, 2003.

[8]

A. James. HARD track overview in TREC 2004. In Proceedings of TREC, 2004.

[9]

B. J. Jansen and A. Spink. An analysis of web documents retrieved and viewed. In Proceedings of the International Conference on Internet Computing, pages 65--69, 2003.

[10]

B. J. Jansen, A. Spink, J. Bateman, and T. Saracevic. Real life information retrieval: A study of user queries on the web. In Proceedings of SIGIR, pages 5--17, 1998.

Digital Library

[11]

K. Jarvelin and J. Kekalainen. IR evaluation methods for retrieving highly relevant documents. In Proceedings of SIGIR, pages 41--48, 2000.

Digital Library

[12]

T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In Proceeding of SIGIR, pages 154--161, 2005.

Digital Library

[13]

Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Mueller. Efficient backprop. In Neural Networks: Tricks of the Trade, pages 9--50. Springer, 1998.

[14]

G. Levow and I. Matveeva. University of Chicago at CLEF2004: Cross-language text and spoken document retrieval. In Proceedings of CLEF, 2004.

Digital Library

[15]

W. Lin, A. Hauptmann, and R. Jin. Web image retrieval reranking with a relevance model. In Proceedings IEEE/WIC, 2003.

Digital Library

[16]

J. Ponte. Language models for relevance feedback. In W. Croft, editor, Advances in Information Retrieval, pages 73--96, 2000.

[17]

J. J. Roccio. Relevance feedback in information retrieval. In The SMART Retrieval System: Experiments in Automatic Document Processing, pages 313--323. Prentice Hall, 1971.

Digital Library

[18]

C. Shah and W. B. Croft. Evaluating high accuracy retrieval techniques. In Proceedings of SIGIR, pages 2--9, 2004.

Digital Library

[19]

L. Shen and A. K. Joshi. Ranking and reranking with perceptron. Machine Learning, 60(1-3):73--96, 2005.

Digital Library

[20]

E. M. Voorhees. Evaluation by highly relevant documents. In Proceedings of SIGIR, pages 74--82, 2001.

Digital Library

[21]

Y. Xiao, R. Luk, K. Wong, and K. Kwok. Some experiments with blind feedback and re-ranking for chinese information retrieval. In Proceedings NTCIR, 2005.

Cited By

Kumarasinghe ULekssays ASencar HBoughorbel SElvitigala CNakov PQuek TGao DZhou JCardenas A(2024)Semantic Ranking for Automated Adversarial Technique Annotation in Security TextProceedings of the 19th ACM Asia Conference on Computer and Communications Security10.1145/3634737.3645000(49-62)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3634737.3645000
McKechnie JMcDonald GMacdonald CHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Bi-Objective Negative Sampling for Sensitivity-Aware SearchProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657895(2296-2300)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657895
Rateria SSingh S(2024)Transparent, Low Resource, and Context-Aware Information Retrieval From a Closed Domain Knowledge BaseIEEE Access10.1109/ACCESS.2024.338000612(44233-44243)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3380006
Show More Cited By

Index Terms

High accuracy retrieval with multiple nested ranker
1. Information systems
  1. Information retrieval

Recommendations

Evaluating high accuracy retrieval techniques
SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval

Although information retrieval research has always been concerned with improving the effectiveness of search, in some applications, such as information analysis, a more specific requirement exists for high accuracy retrieval. This means that achieving ...
Improving Ranking Consistency for Web Search by Leveraging a Knowledge Base and Search Logs
CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

In this paper, we propose a new idea called ranking consistency in web search. Relevance ranking is one of the biggest problems in creating an effective web search system. Given some queries with similar search intents, conventional approaches typically ...
On using inter-document relations in microblog retrieval
WWW '13 Companion: Proceedings of the 22nd International Conference on World Wide Web

Microblog Ad-hoc retrieval has received much attention in recent years. As a result of the high vocabulary diversity of the publishing users, a mismatch is formed between the queries being formulated and the tweets representing the actual topics. In ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval

August 2006

768 pages

ISBN:1595933697

DOI:10.1145/1148170

General Chair:
Efthimis N. Efthimiadis
University of Washington
,
Program Chairs:
Susan Dumais
Microsoft Research, Redmond
,
David Hawking
CSIRO ICT Centre, Canberra, Australia
,
Kalervo Järvelin,
University of Tampere, Finland

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 August 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR06

Sponsor:

SIGIR06: The 29th Annual International SIGIR Conference

August 6 - 11, 2006

Washington, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

68
Total Citations
View Citations
826
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)3

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kumarasinghe ULekssays ASencar HBoughorbel SElvitigala CNakov PQuek TGao DZhou JCardenas A(2024)Semantic Ranking for Automated Adversarial Technique Annotation in Security TextProceedings of the 19th ACM Asia Conference on Computer and Communications Security10.1145/3634737.3645000(49-62)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3634737.3645000
McKechnie JMcDonald GMacdonald CHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Bi-Objective Negative Sampling for Sensitivity-Aware SearchProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657895(2296-2300)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657895
Rateria SSingh S(2024)Transparent, Low Resource, and Context-Aware Information Retrieval From a Closed Domain Knowledge BaseIEEE Access10.1109/ACCESS.2024.338000612(44233-44243)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3380006
Hina NBoughanem MDkaki T(2024)Token Pruning by Dimensionality Reduction Methods on TCT-ColBERT for RerankingFoundations of Intelligent Systems10.1007/978-3-031-62700-2_7(65-74)Online publication date: 17-Jun-2024
https://doi.org/10.1007/978-3-031-62700-2_7
Acquavia AMacdonald CTonellotto N(2023)Static Pruning for Multi-Representation Dense RetrievalProceedings of the ACM Symposium on Document Engineering 202310.1145/3573128.3604896(1-10)Online publication date: 22-Aug-2023
https://dl.acm.org/doi/10.1145/3573128.3604896
Rudra KFernando ZAnand A(2023)An in-depth analysis of passage-level label transfer for contextual document rankingInformation Retrieval10.1007/s10791-023-09430-526:1-2Online publication date: 8-Dec-2023
https://dl.acm.org/doi/10.1007/s10791-023-09430-5
Qiao YJi SWang CShao JYang T(2023)Privacy-aware document retrieval with two-level inverted indexingInformation Retrieval Journal10.1007/s10791-023-09428-z26:1-2Online publication date: 17-Nov-2023
https://doi.org/10.1007/s10791-023-09428-z
Ravanmehr RMohamadrezaei RRavanmehr RMohamadrezaei R(2023)Learning to Rank in Session-Based Recommender SystemsSession-Based Recommender Systems Using Deep Learning10.1007/978-3-031-42559-2_6(245-292)Online publication date: 21-Dec-2023
https://doi.org/10.1007/978-3-031-42559-2_6
Lin J(2022)A proposed conceptual framework for a representational approach to information retrievalACM SIGIR Forum10.1145/3527546.352755255:2(1-29)Online publication date: 17-Mar-2022
https://dl.acm.org/doi/10.1145/3527546.3527552
Cai YGuo JFan YAi QZhang RCheng XAl Hasan MXiong L(2022)Hard Negatives or False NegativesProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557343(118-127)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557343
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents