Article

First-order focused crawling

Authors:

Qingyang Xu,

Wanli ZuoAuthors Info & Claims

WWW '07: Proceedings of the 16th international conference on World Wide Web

Pages 1159 - 1160

https://doi.org/10.1145/1242572.1242744

Published: 08 May 2007 Publication History

Get Access

Abstract

This paper reports a new general framework of focused web crawling based on "relational subgroup discovery". Predicates are used explicitly to represent the relevance clues of those unvisited pages in the crawl frontier, and then first-order classification rules are induced using subgroup discovery technique. The learned relational rules with sufficient support and confidence will guide the crawling process afterwards. We present the many interesting features of our proposed first-order focused crawler, together with preliminary promising experimental results.

References

[1]

Soumen Chakrabarti, Kunal Punera, and Mallela Subramanyam. Accelerated focused crawling through online relevance feedback. In WWW, pages 148--159, 2002.

Digital Library

Google Scholar

[2]

Soumen Chakrabarti, Martin van den Berg, and Byron Dom. Focused crawling: A new approach to topic-specific web resource discovery. Computer Networks, 31(11-16):1623--1640, 1999.

Digital Library

Google Scholar

[3]

Michelangelo Diligenti, Frans Coetzee, Steve Lawrence, C. Lee Giles, and Marco Gori. Focused crawling using context graphs. In VLDB, pages 527--534, 2000.

Digital Library

Google Scholar

[4]

Nada Lavrac. Subgroup discovery techniques and applications. In PAKDD, pages 2--14, 2005.

Digital Library

Google Scholar

Cited By

View all

Hassan TCruz CBertaux A(2017)Ontology-based approach for unsupervised and adaptive focused crawlingProceedings of The International Workshop on Semantic Big Data10.1145/3066911.3066912(1-6)Online publication date: 19-May-2017
https://dl.acm.org/doi/10.1145/3066911.3066912
Hassan TCruz CBertaux A(2017)Predictive and evolutive cross-referencing for web textual sources2017 Computing Conference10.1109/SAI.2017.8252230(1114-1122)Online publication date: Jul-2017
https://doi.org/10.1109/SAI.2017.8252230
Kumar MBhatia RRattan D(2017)A survey of Web crawlers for information retrievalWIREs Data Mining and Knowledge Discovery10.1002/widm.12187:6Online publication date: 7-Aug-2017
https://doi.org/10.1002/widm.1218
Show More Cited By

Index Terms

First-order focused crawling

Recommendations

Current challenges in web crawling
ICWE'13: Proceedings of the 13th international conference on Web Engineering

Web crawling, a process of collecting web pages in an automated manner, is the primary and ubiquitous operation used by a large number of web systems and agents starting from a simple program for website backup to a major web search engine. Due to an ...
Using HMM to learn user browsing patterns for focused web crawling
Special issue: WIDM 2004

A focused crawler is designed to traverse the Web to gather documents on a specific topic. It can be used to build domain-specific Web search portals and online personalized search tools. To estimate the relevance of a newly seen URL, it must use ...
Exploiting Multiple Features with MEMMs for Focused Web Crawling
NLDB '08: Proceedings of the 13th international conference on Natural Language and Information Systems: Applications of Natural Language to Information Systems

Focused web crawling traverses the Web to collect documents on a specific topic. This is not an easy task, since focused crawlers need to identify the next most promising link to follow based on the topic and the content and links of previously crawled ...

Comments

Information & Contributors

Information

Published In

WWW '07: Proceedings of the 16th international conference on World Wide Web

May 2007

1382 pages

ISBN:9781595936547

DOI:10.1145/1242572

General Chairs:
Carey Williamson
University of Calgary, Canada
,
Mary Ellen Zurko
IBM, USA
,
Program Chairs:
Peter Patel-Schneider
Bell Labs Research, USA
,
Prashant Shenoy
University of Massachusetts at Amherst, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 May 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

WWW'07

Sponsor:

WWW'07: 16th International World Wide Web Conference

May 8 - 12, 2007

Alberta, Banff, Canada

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
380
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Hassan TCruz CBertaux A(2017)Ontology-based approach for unsupervised and adaptive focused crawlingProceedings of The International Workshop on Semantic Big Data10.1145/3066911.3066912(1-6)Online publication date: 19-May-2017
https://dl.acm.org/doi/10.1145/3066911.3066912
Hassan TCruz CBertaux A(2017)Predictive and evolutive cross-referencing for web textual sources2017 Computing Conference10.1109/SAI.2017.8252230(1114-1122)Online publication date: Jul-2017
https://doi.org/10.1109/SAI.2017.8252230
Kumar MBhatia RRattan D(2017)A survey of Web crawlers for information retrievalWIREs Data Mining and Knowledge Discovery10.1002/widm.12187:6Online publication date: 7-Aug-2017
https://doi.org/10.1002/widm.1218
Saini CArora V(2016)Information retrieval in web crawling: A survey2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI)10.1109/ICACCI.2016.7732456(2635-2643)Online publication date: Sep-2016
https://doi.org/10.1109/ICACCI.2016.7732456
Kumar MVig R(2013)Focused Crawling Based Upon TF-IDF Semantics and Hub Score LearningJournal of Emerging Technologies in Web Intelligence10.4304/jetwi.5.1.70-775:1Online publication date: 1-Feb-2013
https://doi.org/10.4304/jetwi.5.1.70-77
Samarawickrama SJayaratne L(2011)Automatic text classification and focused crawling2011 Sixth International Conference on Digital Information Management10.1109/ICDIM.2011.6093329(143-148)Online publication date: Sep-2011
https://doi.org/10.1109/ICDIM.2011.6093329
Batsakis SPetrakis EMilios E(2009)Improving the performance of focused web crawlersData & Knowledge Engineering10.1016/j.datak.2009.04.00268:10(1001-1013)Online publication date: 1-Oct-2009
https://dl.acm.org/doi/10.1016/j.datak.2009.04.002
Jing TZuo WHe F(2008)A Hybrid Method for Semantic Annotation of Chinese Web PagesProceedings of the 2008 International Seminar on Future Information Technology and Management Engineering10.1109/FITME.2008.105(252-256)Online publication date: 20-Nov-2008
https://dl.acm.org/doi/10.1109/FITME.2008.105
Tsekouras GGavalas DFilios SNiros ABafaloukas G(2008)A Clustering Framework to Build Focused Web Crawlers for Automatic Extraction of Cultural InformationProceedings of the 5th Hellenic conference on Artificial Intelligence: Theories, Models and Applications10.1007/978-3-540-87881-0_43(419-424)Online publication date: 2-Oct-2008
https://dl.acm.org/doi/10.1007/978-3-540-87881-0_43

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Current challenges in web crawling

Using HMM to learn user browsing patterns for focused web crawling

Exploiting Multiple Features with MEMMs for Focused Web Crawling