research-article

The influence of commercial intent of search results on their perceived relevance

Author:

Dirk LewandowskiAuthors Info & Claims

iConference '11: Proceedings of the 2011 iConference

Pages 452 - 458

https://doi.org/10.1145/1940761.1940823

Published: 08 February 2011 Publication History

Abstract

We carried out a retrieval effectiveness test on the three major web search engines (i.e., Google, Microsoft and Yahoo). In addition to relevance judgments, we classified the results according to their commercial intent and whether or not they carried any advertising. We found that all search engines provide a large number of results with a commercial intent. Google provides significantly more commercial results than the other search engines do. However, the commercial intent of a result did not influence jurors in their relevance judgments.

References

[1]

Borlund, P. (2003). The concept of relevance in IR. Journal of the American Society for Information Science and Technology, 54(10), 913--925.

Digital Library

[2]

Broder, A. (2002). A taxonomy of web search. SIGIR Forum, 36(2), 3--10.

Digital Library

[3]

Buckley, C., & Voorhees, E. M. (2000). Evaluating evaluation measure stability. SIGIR Forum (ACM Special Interest Group on Information Retrieval), 33--40.

Digital Library

[4]

Chu, H., & Rosenthal, M. (1996). Search engines for the World Wide Web: A comparative study and evaluation methodology. Proceedings of the Ninth Annual Meeting of the American Society for Information Science, Baltimore, Maryland, 21--4 Oct 1996. Edited by Steve Hardin. Medford, New Jersey: Information Today, Inc., p. 127--35.

[5]

Couvering, E. V. (2007). Is relevance relevant? Market, science, and war: Discourses of search engine quality. Journal of Computer-Mediated Communication, 12(3), 866--887.

[6]

Cutrell, E., & Guan, Z. (2007). Eye tracking in MSN Search: Investigating snippet length, target position and task types. Microsoft Technical Report, MSR-TR-2007-01. ftp://ftp.research.microsoft.com/pub/tr/TR-2007-01.pdf

[7]

Ding, W., & Marchionini, G. (1996). A comparative study of web search service performance. Proceedings of the Ninth Annual Meeting of the American Society for Information Science, Baltimore, Maryland, 21--4 Oct 1996. Edited by Steve Hardin. Medford, New Jersey: Information Today, Inc., p. 136--42.

[8]

Dresel, R., Hörnig, D., Kaluza, H., Peter, A., Roßmann, N., & Sieber, W. (2001). Evaluation deutscher Web-Suchwerkzeuge. Nachrichten füür Dokumentation, 52(7), 381--392.

[9]

Gordon, M., & Pathak, P. (1999). Finding information on the World Wide Web: The retrieval effectiveness of search engines. Information Processing & Management, 35(2), 141--180.

Digital Library

[10]

Granka, L., Hembrooke, H., & Gay, G. (2005). Location, Location, Location: Viewing Patterns on WWW Pages. Proc. of Eye Tracking Research & Applications (ETRA): Symposium 2006, ACM

Digital Library

[11]

Griesbaum, J. (2004). Evaluation of three German search engines: Altavista.de, Google.de and Lycos.de. Information Research, 9(4). http://informationr.net/ir/9-4/paper189.html

[12]

Griesbaum, J., Rittberger, M., & Bekavac, B. (2002). In R. Hammwööhner, C. Wolff & C. Womser-Hacker (Eds.), Deutsche Suchmaschinen im Vergleich: AltaVista.de, Fireball.de, Google.de und Lycos.de (pp. 201--223). Paper presented at the Information und Mobilität. Optimierung und Vermeidung von Mobilität durch Information. 8. Internationales Symposium für Informationswissenschaft. UVK.

[13]

Hawking, D., & Craswell, N. (2005). The very large collection and Web tracks. In E. M. Voorhees & D. K. Harman (Eds.), TREC experiment and evaluation in information retrieval (pp. 199--231). Cambridge, Massachusetts: MIT Press.

[14]

Hawking, D., Craswell, N., Bailey, P., & Griffiths, K. (2001). Measuring search engine quality. Information Retrieval, 4(1), 33--59.

Digital Library

[15]

Hjorland, B. (2010). The foundation of the concept of relevance. Journal of the American Society for Information Science and Technology, 61(2), 217--237.

Digital Library

[16]

Hotchkiss, G. (2007). Search in the year 2010. Search Engine Land. http://searchengineland.com/search-in-the-year-2010-11917

[17]

Höchstötter, N., & Koch, M. (2008). Standard parameters for searching behaviour in search engines and their empirical evaluation. Journal of Information Science, 35(1), 45--64.

Digital Library

[18]

Höchstötter, N., & Lewandowski, D. (2009). What users see -- Structures in search engine results pages. Information Sciences, 179(12), 1796--1812.

Digital Library

[19]

Jansen, B. J. (2007). The comparative effectiveness of sponsored and nonsponsored links for web e-commerce queries. ACM Transactions on the Web, 1(1), 1--25.

Digital Library

[20]

Jansen, B. J., & Molina, P. R. (2006). The effectiveness of Web search engines for retrieving relevant ecommerce links. Information Processing & Management, 42(4), 1075--1098.

Digital Library

[21]

Jansen, B. J., & Resnick, M. (2006). An examination of searcher's perceptions of nonsponsored and sponsored links during ecommerce Web searching. Journal of the American Society for Information Science and Technology, 57(14), 1949--1961.

Digital Library

[22]

Joachims, T., Granka, L., Pan, B., Hembrooke, H., and Gay, G. (2005). Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Salvador, Brazil, August 15--19, 2005). SIGIR '05. ACM, New York, NY, 154--161.

Digital Library

[23]

Lazarinis, F., Vilares, J., Tait, J., & Efthimiadis, E. (2009). Current research issues and trends in non-English Web searching. Information Retrieval, 12(3), 230--250.

Digital Library

[24]

Leighton, H. V., & Srivastava, J. (1999). First 20 precision among World Wide Web search services (search engines). Journal of the American Society for Information Science, 50(10), 870--881.

Digital Library

[25]

Lewandowski, D. (2006). Query types and search topics of German Web search engine users. Information Services & Use, 26(4), 261--269.

[26]

Lewandowski, D. (2008). The retrieval effectiveness of Web search engines: Considering results descriptions. Journal of Documentation, 64(6), 915--937.

[27]

Lewandowski, D. (2009). The retrieval effectiveness of search engines on navigational queries. ASLIB Proceedings (in press).

[28]

Saracevic, T. (2007a). Relevance: A review of the literature and a framework for thinking on the notion in Information Science. Part II: Nature and manifestations of relevance. Journal of the American Society for Information Science and Technology, 58(13), 1915--1933.

[29]

Saracevic, T. (2007b). Relevance: A review of the literature and a framework for thinking on the notion in Information Science. Part III: Behavior and effects of relevance. Journal of the American Society for Information Science and Technology, 58(13), 2126--2144.

Digital Library

[30]

Sullivan, D. (2007). Major Search Engines and Directories. Retrieved 10.08.2010, from http://searchenginewatch.com/showPage.html?page=2156221

[31]

Tague-Sucliffe, J. (1992). The pragmatics of information retrieval experimentation, revisited. Information Processing & Management, 28(4), 467--490.

Digital Library

[32]

Vaughan, L., & Thelwall, M. (2004). Search Engine coverage bias: Evidence and possible causes. Information Processing & Management, 40(4), 693--707.

Digital Library

[33]

Vaughan, L., & Zhang, Y. (2007). Equal representation by search engines? A comparison of websites across countries and domains. Journal of Computer-Mediated Communication, 12(3), article 7.

[34]

Véronis, J. (2006). A comparative study of six search engines. Retrieved 10.08.2010, from http://www.up.univmrs.fr/veronis/pdf/2006-comparative-study.pdf

[35]

Webhits. (2010). Webhits Web-Barometer. Retrieved 10.08.2010, from http://www.webhits.de/deutsch/index.shtml?webstats.ht

Cited By

Martens BMartens B(2023)Databases, Search EnginesKeywords In and Out of Context10.1007/978-3-031-32530-4_9(127-139)Online publication date: 4-Jun-2023
https://doi.org/10.1007/978-3-031-32530-4_9
Kammerer YGottschling SBråten I(2020)The Role of Internet-Specific Justification Beliefs in Source Evaluation and Corroboration During Web Search on an Unsettled Socio-Scientific IssueJournal of Educational Computing Research10.1177/073563312095273159:2(342-378)Online publication date: 28-Aug-2020
https://doi.org/10.1177/0735633120952731
Kammerer YBrand-Gruwel S(2020)Trainings and Tools to Foster Source Credibility Evaluation During Web SearchUnderstanding and Improving Information Search10.1007/978-3-030-38825-6_11(213-243)Online publication date: 30-May-2020
https://doi.org/10.1007/978-3-030-38825-6_11
Show More Cited By

Index Terms

The influence of commercial intent of search results on their perceived relevance
1. Information systems
  1. Information retrieval
  2. World Wide Web
    1. Web applications
    2. Web services

Recommendations

The influence of search engine optimization on Google's results: A multi-dimensional approach for detecting SEO
WebSci '21: Proceedings of the 13th ACM Web Science Conference 2021

Search engine optimization (SEO) can significantly influence what is shown on the result pages of commercial search engines. However, it is unclear what proportion of (top) results have actually been optimized. We developed a tool that uses a semi-...
What users see - Structures in search engine results pages

This paper investigates the composition of search engine results pages. We define what elements the most popular web search engines use on their results pages (e.g., organic results, advertisements, shortcuts) and to which degree they are used for ...
The effect of user intent on the stability of search engine results

Previous work has established that search engine queries can be classified according to the intent of the searcher (i.e., why is the user searching, what specifically do they intend to do). In this article, we describe an experiment in which four sets ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

iConference '11: Proceedings of the 2011 iConference

February 2011

858 pages

ISBN:9781450301213

DOI:10.1145/1940761

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 February 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

iConference '11

iConference '11: iConference 2011

February 8 - 11, 2011

Washington, Seattle, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
366
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Martens BMartens B(2023)Databases, Search EnginesKeywords In and Out of Context10.1007/978-3-031-32530-4_9(127-139)Online publication date: 4-Jun-2023
https://doi.org/10.1007/978-3-031-32530-4_9
Kammerer YGottschling SBråten I(2020)The Role of Internet-Specific Justification Beliefs in Source Evaluation and Corroboration During Web Search on an Unsettled Socio-Scientific IssueJournal of Educational Computing Research10.1177/073563312095273159:2(342-378)Online publication date: 28-Aug-2020
https://doi.org/10.1177/0735633120952731
Kammerer YBrand-Gruwel S(2020)Trainings and Tools to Foster Source Credibility Evaluation During Web SearchUnderstanding and Improving Information Search10.1007/978-3-030-38825-6_11(213-243)Online publication date: 30-May-2020
https://doi.org/10.1007/978-3-030-38825-6_11
Kanniainen LKiili CTolvanen AAro MLeppänen P(2019)Literacy skills and online research and comprehension: struggling readers face difficulties onlineReading and Writing10.1007/s11145-019-09944-932:9(2201-2222)Online publication date: 21-Mar-2019
https://doi.org/10.1007/s11145-019-09944-9
Adekotujo ALee JEnikuomehin AMazzara MAribisala S(2019)Bi-lingual Intent Classification of Twitter Posts: A RoadmapProceedings of 6th International Conference in Software Engineering for Defence Applications10.1007/978-3-030-14687-0_1(1-9)Online publication date: 19-Mar-2019
https://doi.org/10.1007/978-3-030-14687-0_1
Taddeo MFloridi L(2017)The Moral Responsibilities of Online Service ProvidersThe Responsibilities of Online Service Providers10.1007/978-3-319-47852-4_2(13-42)Online publication date: 1-Feb-2017
https://doi.org/10.1007/978-3-319-47852-4_2
Sünkler SLewandowski D(2017)Does it matter which search engine is used? A user study using post‐task relevance judgmentsProceedings of the Association for Information Science and Technology10.1002/pra2.2017.1450540104454:1(405-414)Online publication date: 24-Oct-2017
https://doi.org/10.1002/pra2.2017.14505401044
Scharrer LSalmerón L(2016)Sourcing in the reading process: introduction to the special issueReading and Writing10.1007/s11145-016-9676-229:8(1539-1548)Online publication date: 10-Aug-2016
https://doi.org/10.1007/s11145-016-9676-2
Taddeo MFloridi L(2015)The Debate on the Moral Responsibilities of Online Service ProvidersScience and Engineering Ethics10.1007/s11948-015-9734-122:6(1575-1603)Online publication date: 27-Nov-2015
https://doi.org/10.1007/s11948-015-9734-1
Lewandowski D(2015)Evaluating the retrieval effectiveness of web search engines using a representative query sampleJournal of the Association for Information Science and Technology10.1002/asi.2330466:9(1763-1775)Online publication date: 1-Sep-2015
https://dl.acm.org/doi/10.1002/asi.23304
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents