article

Comparing rankings of search results on the Web

Author:

Judit Bar-IlanAuthors Info & Claims

Information Processing and Management: an International Journal, Volume 41, Issue 6

Pages 1511 - 1519

Published: 01 December 2005 Publication History

Abstract

The Web has become an information source for professional data gathering. Because of the vast amounts of information on almost all topics, one cannot systematically go over the whole set of results, and therefore must rely on the ordering of the results by the search engine. It is well known that search engines on the Web have low overlap in terms of coverage. In this study we measure how similar are the rankings of search engines on the overlapping results. We compare rankings of results for identical queries retrieved from several search engines. The method is based only on the set of URLs that appear in the answer sets of the engines being compared. For comparing the similarity of rankings of two search engines, the Spearman correlation coefficient is computed. When comparing more than two sets Kendall's W is used. These are well-known measures and the statistical significance of the results can be computed. The methods are demonstrated on a set of 15 queries that were submitted to four large Web search engines. The findings indicate that the large public search engines on the Web employ considerably different ranking algorithms.

References

[1]

Bar-Ilan, J., Levene, M., & Mat-Hassan, M. (2004). Dynamics of search engine rankings-a case study. In Proceedings of the 3rd international workshop on web dynamics, New York, May 2004. Available: <http://www.dcs.bbk.ac.uk/webDyn3/webdyn3_proceedings.pdf>

[2]

Bharat, K., & Broder, A. (1998). A technique for measuring the relative size and overlap of public Web search engines. In Proceedings of the 7th international world wide web conference, April 1998, computer networks and ISDN systems (Vol. 30, pp. 379-388). Available: <http://decweb.ethz.ch/WWW7/1937/com1937.htm>

Digital Library

[3]

Bove, R. E. (2002). Correlation. Available: <http://courses.wcupa.edu/rbove/eco252/252corr.doc>

[4]

Statistical power analysis for behavioral sciences. Erlbaum, Hilldale, NJ.

[5]

Comparing top k lists. SIAM Journal on Discrete Mathematics. v17 i1. 134-160.

[6]

Garson, D. (2004) Correlation. In Qualitative methods in public administration. Available: <http://www2.chass.ncsu.edu/garson/pa765/correl.htm>

[7]

Google (2004). Information for Webmasters. Available: <http://www.google.com/webmasters/4.html>

[8]

Measuring search engine quality. Information Retrieval. v4. 33-59.

[9]

Use of electronic resources in scholarly electronic journals: A citation analysis. College & Research Libraries. v63 i4. 334-340.

[10]

Free online availability substantially increases a paper's impact. Nature. v411. 521

[11]

Accessibility of information on the Web. Nature. v400. 107-109.

[12]

Lowry, R. (2004). Rank-order correlation. In Concepts and applications of inferential statistics. Available: <http://faculty.vassar.edu/lowry/ch3b.html>

[13]

Nielsen/NetRatings. (2004). NetView usage metrics. Available: <http://www.netratings.com/news.jsp?section=dat_to>

[14]

Statistics without tears: A primer for non-mathematicians. Penguin.

[15]

Analysis of a very large Web search engine query log. ACM SIGIR Forum. v33 i1.

[16]

The referencing of internet web sites in medical and scientific publications. Brain and Cognition. v50. 335-337.

[17]

US versus European Web searching trends. SIGIR Forum Fall.

[18]

Su, L. T., Chen, H. L. & Dong, X. Y. (1998). Evaluation of Web-based search engines from the end-user's perspective: A pilot study. In Proceedings of the ASIS Annual Meeting (Vol. 35, pp. 348-361)

[19]

Sullivan, D. (2003). Florida Google dance resources. Available: <http://www.searchenginewatch.com/searchday/article.php/3285661>

[20]

Sullivan, D. (2004a). Nielsen NetRatings search engine rankings. In Searchenginewatch reports. Available: <http://searchenginewatch.com/reports/article.php/2156451>

[21]

Sullivan, D., (2004b). Who powers whom? Search providers chart. In Searchenginewatch reports. Retrieved October 15, 2004, from Available: <http://searchenginewatch.com/reports/article.php/2156451>

[22]

New measurements for search engine evaluation proposed and tested. Information Processing & Management. v40 i4. 677-691.

[23]

Scholarly use of Internet-based electronic resources. Journal of the American Society for Information Science and Technology. v52 i8. 628-650.

Cited By

Criado RGarcía EPedroche FRomance M(2016)On graphs associated to sets of rankingsJournal of Computational and Applied Mathematics10.1016/j.cam.2015.03.009291:C(497-508)Online publication date: 1-Jan-2016
https://dl.acm.org/doi/10.1016/j.cam.2015.03.009
Kumar PSingh AMohan A(2015)Efficient methodologies to overcome the effects of hanging pages in search engine optimisationInternational Journal of Web Engineering and Technology10.1504/IJWET.2015.07233510:2(129-151)Online publication date: 1-Oct-2015
https://dl.acm.org/doi/10.1504/IJWET.2015.072335
Zhang QYoussef A(2015)Performance Evaluation and Optimization of Math-Similarity SearchProceedings of the International Conference on Intelligent Computer Mathematics - Volume 915010.1007/978-3-319-20615-8_16(243-257)Online publication date: 13-Jul-2015
https://dl.acm.org/doi/10.1007/978-3-319-20615-8_16
Show More Cited By

Index Terms

Comparing rankings of search results on the Web
1. Applied computing
  1. Document management and text processing
2. Information systems
  1. Information retrieval
    1. Document representation
  2. Information storage systems
    1. Storage architectures
      1. Storage network architectures

Recommendations

Methods for comparing rankings of search engine results
Web dynamics

In this paper we present a number of measures that compare rankings of search engine results. We apply these measures to five queries that were monitored daily for two periods of 14 or 21 days each. Rankings of the different search engines (Google, ...
A study of results overlap and uniqueness among major web search engines

The performance and capabilities of Web search engines is an important and significant area of research. Millions of people world wide use Web search engines very day. This paper reports the results of a major study examining the overlap among results ...
Re-ranking search results using query logs
CIKM '06: Proceedings of the 15th ACM international conference on Information and knowledge management

This work addresses two common problems in search, frequently occurring with underspecified user queries: the top-ranked results for such queries may not contain documents relevant to the user's search intent, and fresh and relevant pages may not get ...

Comments

Information & Contributors

Information

Published In

cover image Information Processing and Management: an International Journal

Information Processing and Management: an International Journal Volume 41, Issue 6

Special issue: Infometrics

December 2005

316 pages

ISSN:0306-4573

Issue’s Table of Contents

Copyright © Elsevier Ltd © 2005.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 December 2005

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Criado RGarcía EPedroche FRomance M(2016)On graphs associated to sets of rankingsJournal of Computational and Applied Mathematics10.1016/j.cam.2015.03.009291:C(497-508)Online publication date: 1-Jan-2016
https://dl.acm.org/doi/10.1016/j.cam.2015.03.009
Kumar PSingh AMohan A(2015)Efficient methodologies to overcome the effects of hanging pages in search engine optimisationInternational Journal of Web Engineering and Technology10.1504/IJWET.2015.07233510:2(129-151)Online publication date: 1-Oct-2015
https://dl.acm.org/doi/10.1504/IJWET.2015.072335
Zhang QYoussef A(2015)Performance Evaluation and Optimization of Math-Similarity SearchProceedings of the International Conference on Intelligent Computer Mathematics - Volume 915010.1007/978-3-319-20615-8_16(243-257)Online publication date: 13-Jul-2015
https://dl.acm.org/doi/10.1007/978-3-319-20615-8_16
Thorleuchter DVan Den Poel D(2013)Web mining based extraction of problem solution ideasExpert Systems with Applications: An International Journal10.1016/j.eswa.2013.01.01340:10(3961-3969)Online publication date: 1-Aug-2013
https://dl.acm.org/doi/10.1016/j.eswa.2013.01.013
Lubarski PMorzy M(2012)Measuring the Importance of Users in a Social Network Based on Email Communication PatternsProceedings of the 2012 International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2012)10.1109/ASONAM.2012.24(86-90)Online publication date: 26-Aug-2012
https://dl.acm.org/doi/10.1109/ASONAM.2012.24
Cardoso BMagalhães J(2011)Google, bing and a new perspective on ranking similarityProceedings of the 20th ACM international conference on Information and knowledge management10.1145/2063576.2063858(1933-1936)Online publication date: 24-Oct-2011
https://dl.acm.org/doi/10.1145/2063576.2063858
Tekli JChbeir RFerri FGrifoni P(2010)Toward approximate GML retrieval based on structural and semantic characteristicsProceedings of the 10th international conference on Web engineering10.5555/1884110.1884113(16-34)Online publication date: 5-Jul-2010
https://dl.acm.org/doi/10.5555/1884110.1884113
Zaragoza HCambazoglu BBaeza-Yates RHuang JKoudas NJones GWu XCollins-Thompson KAn A(2010)Web search solved?Proceedings of the 19th ACM international conference on Information and knowledge management10.1145/1871437.1871507(529-538)Online publication date: 26-Oct-2010
https://dl.acm.org/doi/10.1145/1871437.1871507
Webber WMoffat AZobel J(2010)A similarity measure for indefinite rankingsACM Transactions on Information Systems10.1145/1852102.185210628:4(1-38)Online publication date: 23-Nov-2010
https://dl.acm.org/doi/10.1145/1852102.1852106
Mahoney WHospodka PSousan WNickell RZhu Q(2009)A coherent measurement of web-search relevanceIEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans10.1109/TSMCA.2009.202761039:6(1176-1187)Online publication date: 1-Nov-2009
https://dl.acm.org/doi/10.1109/TSMCA.2009.2027610
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents