research-article

Which vertical search engines are relevant?

Authors:

Joemon M. JoseAuthors Info & Claims

WWW '13: Proceedings of the 22nd international conference on World Wide Web

Pages 1557 - 1568

https://doi.org/10.1145/2488388.2488524

Published: 13 May 2013 Publication History

Abstract

Aggregating search results from a variety of heterogeneous sources, so-called verticals, such as news, image and video, into a single interface is a popular paradigm in web search. Current approaches that evaluate the effectiveness of aggregated search systems are based on rewarding systems that return highly relevant verticals for a given query, where this relevance is assessed under different assumptions. It is difficult to evaluate or compare those systems without fully understanding the relationship between those underlying assumptions. To address this, we present a formal analysis and a set of extensive user studies to investigate the effects of various assumptions made for assessing query vertical relevance. A total of more than 20,000 assessments on 44 search tasks across 11 verticals are collected through Amazon Mechanical Turk and subsequently analysed. Our results provide insights into various aspects of query vertical relevance and allow us to explain in more depth as well as questioning the evaluation results published in the literature.

References

[1]

J. Arguello, F. Diaz, and J. Callan. Learning to aggregate vertical results into web search results. In CIKM 2011, pages 201--210.

Digital Library

[2]

J. Arguello, F. Diaz, J. Callan, and B. Carterette. A methodology for evaluating aggregated search results. In ECIR 2011: pages 141--152.

Digital Library

[3]

J. Arguello, F. Diaz, J. Callan, and J. Crespo. Sources of evidence for vertical selection. In SIGIR 2009: pages 315--322.

Digital Library

[4]

J. Arguello, F. Diaz, and J. Paiement. Vertical selection in the presence of unlabeled verticals. In SIGIR 2010: pages 691--698.

Digital Library

[5]

F. Diaz. Integration of news content into web results. In WSDM 2009: pages 182--191.

Digital Library

[6]

J. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5): pages 378--382, 1971.

[7]

D. Hawking and P. Thomas. Server selection methods in hybrid portal search. In SIGIR 2005: pages 75--82.

Digital Library

[8]

X. Li, Y. Wang, and A. Acero. Learning query intent from regularized click graphs. In SIGIR 2008: pages 339--346.

Digital Library

[9]

A. Ponnuswami, K. Pattabiraman, Q. Wu, R. Gilad-Bachrach, and T. Kanungo. On composition of a federated web search result page: using online users to provide pairwise preference for heterogeneous verticals. In WSDM 2011: pages 715--724.

Digital Library

[10]

M. Schulze. A new monotonic, clone-independent, reversal symmetric, and condorcet-consistent single-winner election method. In Social Choice and Welfare, 2010.

[11]

S. Sushmita, H. Joho, M. Lalmas, and R. Villa. Factors affecting click-through behavior in aggregated search interfaces. In CIKM 2010: pages 519--528.

Digital Library

[12]

M. Shokouhi, and L. Si. Federated Search. Foundations and Trends in Information Retrieval (FTIR) 5(1): pages 1--102, 2011.

Digital Library

[13]

M. Sanderson, M. L. Paramita, P. Clough, and E. Kanoulas. Do user preferences and evaluation measures line up? In SIGIR 2010: pages 555--562.

Digital Library

[14]

E. M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. In Information Process Management. 36(5): pages 697--716, 2000.

Digital Library

[15]

K. Zhou, R. Cummins, M. Lalmas, and J.M. Jose. Evaluating large-scale distributed vertical search. In CIKM Workshop LSDS-IR 2011.

Digital Library

[16]

K. Zhou, R. Cummins, M. Halvey, M. Lalmas, and J. M. Jose. Assessing and predicting vertical intent for web queries. In ECIR 2012: pages 499--502.

Digital Library

[17]

K. Zhou, R. Cummins, M. Lalmas, and J. M. Jose. Evaluating aggregated search pages. In SIGIR 2012: pages 115--124.

Digital Library

[18]

K. Zhou, R. Cummins, M. Lalmas, and J. M. Jose. Evaluating reward and risk for vertical selection. In CIKM 2012: pages 2631--2634.

Digital Library

Cited By

Choi WKim S(2022)A Novel Probabilistic Graphical Model-Based Click Model for Vertical SearchJournal of the Korean Institute of Industrial Engineers10.7232/JKIIE.2022.48.2.13848:2(138-150)Online publication date: 15-Apr-2022
https://doi.org/10.7232/JKIIE.2022.48.2.138
Achsas SNfaoui E(2022)Academic Aggregated Search Approach Based on BERT Language Model2022 2nd International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET)10.1109/IRASET52964.2022.9737888(1-9)Online publication date: 3-Mar-2022
https://doi.org/10.1109/IRASET52964.2022.9737888
Almaatouq AKrafft PDunham YRand DPentland A(2019)Turkers of the World Unite: Multilevel In-Group Bias Among Crowdworkers on Amazon Mechanical TurkSocial Psychological and Personality Science10.1177/194855061983700211:2(151-159)Online publication date: 17-Apr-2019
https://doi.org/10.1177/1948550619837002
Show More Cited By

Index Terms

Which vertical search engines are relevant?
1. Information systems
  1. Information retrieval

Recommendations

Aligning Vertical Collection Relevance with User Intent
CIKM '14: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management

Selecting and aggregating different types of content from multiple vertical search engines is becoming popular in web search. The user vertical intent, the verticals the user expects to be relevant for a particular information need, might not correspond ...
Interest and Evaluation of Aggregated Search
WI-IAT '11: Proceedings of the 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology - Volume 01

Major search engines perform what is known as Aggregated Search (AS). They integrate results coming from different vertical search engines (images, videos, news, etc.) with typical Web search results. Aggregated search is relatively new and its ...
The Effects of Vertical Rank and Border on Aggregated Search Coherence and Search Behavior
CIKM '14: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management

Aggregated search is the task of blending results from different search services, or verticals, into a set of web search results. Aggregated search coherence is the extent to which results from different sources focus on similar senses of an ambiguous or ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

WWW '13: Proceedings of the 22nd international conference on World Wide Web

May 2013

1628 pages

ISBN:9781450320351

DOI:10.1145/2488388

General Chairs:
Daniel Schwabe
PUC-Rio - Brazil
,
Virgílio Almeida
UFMG - Brazil
,
Hartmut Glaser
CGI.br - Brazil
,
Program Chairs:
Ricardo Baeza-Yates
Yahoo! Labs - Spain & Chile
,
Sue Moon
KAIST - South Korea

Copyright © 2013 Copyright is held by the International World Wide Web Conference Committee (IW3C2).

Sponsors

NICBR: Nucleo de Informatcao e Coordenacao do Ponto BR
CGIBR: Comite Gestor da Internet no Brazil

In-Cooperation

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

WWW '13

Sponsor:

NICBR
CGIBR

WWW '13: 22nd International World Wide Web Conference

May 13 - 17, 2013

Rio de Janeiro, Brazil

Acceptance Rates

WWW '13 Paper Acceptance Rate 125 of 831 submissions, 15%;

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
313
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)1

Reflects downloads up to 24 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Choi WKim S(2022)A Novel Probabilistic Graphical Model-Based Click Model for Vertical SearchJournal of the Korean Institute of Industrial Engineers10.7232/JKIIE.2022.48.2.13848:2(138-150)Online publication date: 15-Apr-2022
https://doi.org/10.7232/JKIIE.2022.48.2.138
Achsas SNfaoui E(2022)Academic Aggregated Search Approach Based on BERT Language Model2022 2nd International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET)10.1109/IRASET52964.2022.9737888(1-9)Online publication date: 3-Mar-2022
https://doi.org/10.1109/IRASET52964.2022.9737888
Almaatouq AKrafft PDunham YRand DPentland A(2019)Turkers of the World Unite: Multilevel In-Group Bias Among Crowdworkers on Amazon Mechanical TurkSocial Psychological and Personality Science10.1177/194855061983700211:2(151-159)Online publication date: 17-Apr-2019
https://doi.org/10.1177/1948550619837002
Takanobu RZhuang THuang MFeng JTang HZheng B(2019)Aggregating E-commerce Search Results from Heterogeneous Sources via Hierarchical Reinforcement LearningThe World Wide Web Conference10.1145/3308558.3313455(1771-1781)Online publication date: 13-May-2019
https://dl.acm.org/doi/10.1145/3308558.3313455
Achsas SNfaoui E(2019)An Analysis Study of Vertical Selection Task in Aggregated SearchProcedia Computer Science10.1016/j.procs.2019.01.021148(171-180)Online publication date: 2019
https://doi.org/10.1016/j.procs.2019.01.021
Azzopardi LThomas PCraswell NCollins-Thompson KMei QDavison BLiu YYilmaz E(2018)Measuring the Utility of Search Engine Result PagesThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval10.1145/3209978.3210027(605-614)Online publication date: 27-Jun-2018
https://dl.acm.org/doi/10.1145/3209978.3210027
Levi OGuy IRaiber FKurland O(2018)Selective Cluster Presentation on the Search Results PageACM Transactions on Information Systems10.1145/315867236:3(1-42)Online publication date: 28-Feb-2018
https://dl.acm.org/doi/10.1145/3158672
Wu YLi YHao G(2018)A Web-Based Theme-Related Word Set Construction AlgorithmWeb and Big Data10.1007/978-3-030-01298-4_17(188-200)Online publication date: 21-Oct-2018
https://doi.org/10.1007/978-3-030-01298-4_17
Arguello J(2017)Aggregated SearchFoundations and Trends in Information Retrieval10.1561/150000005210:5(365-502)Online publication date: 6-Mar-2017
https://dl.acm.org/doi/10.1561/1500000052
Li XLiu YCai RMa Sde Rijke MShokouhi MTomkins AZhang M(2017)Investigation of User Search Behavior While Facing Heterogeneous Search ServicesProceedings of the Tenth ACM International Conference on Web Search and Data Mining10.1145/3018661.3018673(161-170)Online publication date: 2-Feb-2017
https://dl.acm.org/doi/10.1145/3018661.3018673
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten