Article

AllInOneNews: development and evaluation of a large-scale news metasearch engine

Authors:

Vijay Raghavan,

Hongkun ZhaoAuthors Info & Claims

SIGMOD '07: Proceedings of the 2007 ACM SIGMOD international conference on Management of data

Pages 1017 - 1028

https://doi.org/10.1145/1247480.1247601

Published: 11 June 2007 Publication History

Abstract

AllInOneNews is the largest news metasearch engine in the world, connecting to over 1,000 news sites over 150 countries. Implementing a large-scale metasearch engine like AllInOneNews needs to overcome unique challenges not faced by building small metasearch engines such as developing highly scalable search engine selection techniques. In this paper, we discuss these unique challenges and our solutions to these challenges. We also discuss some novel features of AllInOneNews such as highly automated solution and semantic query match. This paper also reports the results of a comparative evaluation of three commercial news search systems, one search engine - Google News and two metasearch engines - Mamma News and AllInOneNews. Several measures such as effectiveness, diversity and time-sensitivity are used to perform the comparison. Another contribution of this paper is that we introduce a novel scheme to compare multiple news search systems in a combined measure that takes both relevance and time-sensitivity of retrieved information into consideration.

References

[1]

C. Baumgarten. A probabilistic solutions to the selection and fusion problem in distributed information retrieval. ACM SIGIR Conference, 1999.

Digital Library

[2]

M. Bergman. The Deep Web: Surfacing Hidden Value. White Paper of CompletePlanet at http://brightplanet.com/pdf/deepwebwhitepaper.pdf, 2001.

[3]

L. Barbosa, J. Freire. Searching for hidden-web databases. 8th International Workshop on WebDB, 2005.

[4]

J. Callan, Z. Lu, and. W. Croft. Searching Distributed Collections with Inference Networks. ACM SIGIR, 1995, pp.21--28.

Digital Library

[5]

J. Cope, N. Craswell, D. Hawking. Automated Discovery of Search Interfaces on the Web. ADC 2003: 181--189.

Digital Library

[6]

D. Dreilinger, and A. Howe. Experiences with selecting search engines using metasearch. ACM Transactions on Information Systems, July, 1997, pp.195--222.

Digital Library

[7]

Y. Fan, and S. Gauch. Adaptive Agents for Information Gathering from Multiple, Distributed Information Sources. 1999 AAAI Symposium on Intelligent Agents in Cyberspace, Stanford University, March 1999.

[8]

S. Gauch, G. Wang, and M. Gomez. ProFusion: Intelligent fusion from multiple, distributed search engines. Journal of Universal Computer Science, 1996.

[9]

L. Gravano, and H. Garcia-Molina. Generalizing gloss to vector-space databases and broker hierarchies. VLDB, 1995, pp.78--89.

Digital Library

[10]

L. Gravano, and H. Garcia-Molina. Merging ranks from heterogeneous Internet sources. VLDB, 1997, pp.196--205.

Digital Library

[11]

D. Hawking, N. Craswell, and K. Griffiths. Which search engine is best at finding online services? WWW conference, poster, 2001.

[12]

D. Hawking, N. Craswell, P. Bailey, K. Griffiths. Measuring Search Engine Quality. Information Retrieval, 4(1), 2001.

Digital Library

[13]

K. L. Liu, C. Yu, W. Meng, W. Wu, and N. Rishe. A Statistical Method for Estimating the Usefulness of Text Databases. IEEE TKDE, 2002.

Digital Library

[14]

Y. Lu, W. Meng, L. Shu, C. Yu, and K. L. Liu. Evaluation of Result Merging Strategies for Metasearch Engines. WISE Conference, pp.53--66, November 2005.

Digital Library

[15]

Y. Lu, W. Meng, W. Zhang, K. L. Liu, and C. Yu. Automatic Extraction of Publication Time from News Search Results. Int'l Workshop on Challenges in Web Information Retrieval and Integration (WIRI2006), April 2006.

[16]

U. Manber, and P. Bigot. The Search Broker. USENIX Symposium and Internet Techniques and Systems, Monterey, California, December, 1997, pp.231--239.

Digital Library

[17]

W. Meng, K. L. Liu, C. Yu, X. Wang, Y. Chang and N. Rishe. Determining Text Databases to Search in the Internet. VLDB, 1998.

Digital Library

[18]

W. Meng, Z. Wu, C. Yu, and Z. Li. A Highly-Scalable and Effective Method for Metasearch. ACM Transactions on Information Systems 19(3), pp.310--335, July 2001.

Digital Library

[19]

W. Meng, C. Yu, and K. L. Liu. Building Efficient and Effective Metasearch Engines. ACM Computing Surveys, 34(1), March 2002, pp.48--84.

Digital Library

[20]

Y. Rasolofo, D. Hawking, and J. Savoy. Result merging strategies for a current news metasearcher. Information Processing & Management, 39, 2003, pp.581--609.

Digital Library

[21]

Z. Wu, W. Meng, C. Yu, and Z. Li. Towards a highly scalable and effective metasearch engine. WWW Conference, Hong Kong, 2001.

Digital Library

[22]

Z. Wu, V. Raghavan, H. Qian, V. Rama K, W. Meng, H. He, and C. Yu. Towards Automatic Incorporation of Search Engines into a large-scale Metasearch Engine. IEEE/WIC International Conference on Web Intelligence, 2003.

Digital Library

[23]

C. Yu, W. Meng, K.L. Liu, W. Wu and N. Rishe. Efficient and Effective Metasearch for a Large Number of Text Databases. ACM CIKM, November 1999.

Digital Library

[24]

C. Yu, K. Liu, W. Meng, Z. Wu, and N. Rishe. A Methodology to Retrieve Text Documents from Multiple Databases. IEEE TKDE, Vol.14, No.6, November/December 2002, pp.1347--1361.

Digital Library

[25]

C. Yu, and W. Meng. Web Search Technology. In The Internet Encyclopedia edited by Hossein Bidgoli, Wiley Publishers, pp.738--753, 2003.

[26]

B. Yuwono, and D. Lee. Server Ranking for Distributed Text Resource Systems on the Internet. DASFAA, 1997, pp.391--400.

Digital Library

[27]

H. Zhao, W. Meng, Z. Wu, V. Raghavan, and C. Yu. Fully Automatic Wrapper Generation for Search Engines. WWW Conference, pp.66--75, 2005.

Digital Library

Cited By

Ergashev UDragut EMeng W(2023)Learning To Rank Resources with GNNProceedings of the ACM Web Conference 202310.1145/3543507.3583360(3247-3256)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583360
Umamageswari BKalpana R(2018)Web HarvestingThe Dark Web10.4018/978-1-5225-3163-0.ch010(199-226)Online publication date: 2018
https://doi.org/10.4018/978-1-5225-3163-0.ch010
Meng W(2018)Metasearch EnginesEncyclopedia of Database Systems10.1007/978-1-4614-8265-9_217(2248-2253)Online publication date: 7-Dec-2018
https://doi.org/10.1007/978-1-4614-8265-9_217
Show More Cited By

Index Terms

AllInOneNews: development and evaluation of a large-scale news metasearch engine
1. Information systems
  1. Information retrieval
  2. World Wide Web
    1. Web applications
    2. Web services

Recommendations

Discovering the representative of a search engine
CIKM '01: Proceedings of the tenth international conference on Information and knowledge management

Given a large number of search engines on the Internet, it is difficult for a person to determine which search engines could serve his/her information needs. A common solution is to construct a metasearch engine on top of the search engines. Upon ...
Discovering the representative of a search engine
CIKM '02: Proceedings of the eleventh international conference on Information and knowledge management

Given a large number of search engines on the Internet, it is difficult for a person to determine which search engines could serve his/her information needs. A common solution is to construct a metasearch engine on top of the search engines. Upon ...
Re-ranking search results using query logs
CIKM '06: Proceedings of the 15th ACM international conference on Information and knowledge management

This work addresses two common problems in search, frequently occurring with underspecified user queries: the top-ranked results for such queries may not contain documents relevant to the user's search intent, and fresh and relevant pages may not get ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '07: Proceedings of the 2007 ACM SIGMOD international conference on Management of data

June 2007

1210 pages

ISBN:9781595936868

DOI:10.1145/1247480

General Chairs:
Lizhu Zhou
Tsinghua University, China
,
Tok Wang Ling
National University of Singapore, Singapore
,
Program Chair:
Beng Chin Ooi
National University of Singapore, Singapore

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 June 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGMOD/PODS07

Sponsor:

SIGMOD/PODS07: International Conference on Management of Data

June 11 - 14, 2007

Beijing, China

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
35
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)1

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ergashev UDragut EMeng W(2023)Learning To Rank Resources with GNNProceedings of the ACM Web Conference 202310.1145/3543507.3583360(3247-3256)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583360
Umamageswari BKalpana R(2018)Web HarvestingThe Dark Web10.4018/978-1-5225-3163-0.ch010(199-226)Online publication date: 2018
https://doi.org/10.4018/978-1-5225-3163-0.ch010
Meng W(2018)Metasearch EnginesEncyclopedia of Database Systems10.1007/978-1-4614-8265-9_217(2248-2253)Online publication date: 7-Dec-2018
https://doi.org/10.1007/978-1-4614-8265-9_217
Umamageswari BKalpana R(2017)Web HarvestingWeb Usage Mining Techniques and Applications Across Industries10.4018/978-1-5225-0613-3.ch014(351-378)Online publication date: 2017
https://doi.org/10.4018/978-1-5225-0613-3.ch014
Meng W(2016)Metasearch EnginesEncyclopedia of Database Systems10.1007/978-1-4899-7993-3_217-2(1-6)Online publication date: 22-Dec-2016
https://doi.org/10.1007/978-1-4899-7993-3_217-2
Sleiman HCorchuelo R(2013)A Survey on Region Extractors from Web DocumentsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2012.13525:9(1960-1981)Online publication date: 1-Sep-2013
https://dl.acm.org/doi/10.1109/TKDE.2012.135
Thomas P(2012)To what problem is distributed information retrieval the solution?Journal of the American Society for Information Science and Technology10.1002/asi.2268463:7(1471-1476)Online publication date: 1-Jul-2012
https://dl.acm.org/doi/10.1002/asi.22684
Wang HOu WPeng ZZhai WChen R(2011)User Demand Description and Optimization in Web Data ManagementProceedings of the 2011 Eighth Web Information Systems and Applications Conference10.1109/WISA.2011.25(90-95)Online publication date: 21-Oct-2011
https://dl.acm.org/doi/10.1109/WISA.2011.25
Li RZhai WPeng Z(2011)Supporting query over dynamic combination of data sources for social mediaProceedings of the 2011 international conference on Web-Age Information Management10.1007/978-3-642-28635-3_18(185-196)Online publication date: 14-Sep-2011
https://dl.acm.org/doi/10.1007/978-3-642-28635-3_18
Meng WYu C(2010)Advanced Metasearch Engine TechnologySynthesis Lectures on Data Management10.2200/S00307ED1V01Y201011DTM0112:1(1-129)Online publication date: Jan-2010
https://doi.org/10.2200/S00307ED1V01Y201011DTM011
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents