Article

Random web crawls

Authors:

Toufik Bennouas,

Fabien de MontgolfierAuthors Info & Claims

WWW '07: Proceedings of the 16th international conference on World Wide Web

Pages 451 - 460

https://doi.org/10.1145/1242572.1242634

Published: 08 May 2007 Publication History

Abstract

This paper proposes a random Web crawl model. A Web crawl is a (biased and partial) image of the Web. This paper deals with the hyperlink structure, i.e. a Web crawl is a graph, whose vertices are the pages and whose edges are the hypertextual links. Of course a Web crawl has a very special structure; we recall some known results about it. We then propose a model generating similar structures. Our model simply simulates a crawling, i.e. builds and crawls the graph at the same time. The graphs generated have lot of known properties of Web crawls. Our model is simpler than most random Web graph models, but captures the sames properties. Notice that it models the crawling process instead of the page writing process of Web graph models.

References

[1]

R. Agrawal and R. Srikant. Fast algorithms for mining association rules in large databases. In VLDB '94, pages 487--499. Morgan Kaufmann Publishers Inc., 1994.

Digital Library

[2]

W. Aiello, F. Chung, and L. Lu. A random graph model for massive graphs. In Proceedings of the thirty-second annual ACM symposium on Theory of computing, pages 171--180, 2000.

Digital Library

[3]

R. Albert, H. Jeong, and A.-L. Barabasi. Diameter of the world-wide web. Nature, 401:130--131, September 1999.

[4]

A.-L. Barabasi and R. Albert. Emergence of scaling in random networks. Science, 286:509--512, 1999.

[5]

P. Boldi, M. Santini, and S. Vigna. Do your worst to make the best: Paradoxical effects in pagerank incremental computations. In Workshop on Algorithms and Models for the Web-Graph, 2004.

[6]

B. Bollobás, O. Riordan, J. Spencer, and G. Tusn'ady. The degree sequence of a scale-free random graph process. Random Structures and Algorithms, 18(3):279--290, May 2001.

Digital Library

[7]

A. Bonato. A survey of models of the web graph. In The Proceedings of Combinatorial and Algorithmic Aspects of Networking, August 2004.

Digital Library

[8]

A. Broder, R. Kumar, F. Maghoul, P. Raghavan, S. Rajagopalan, R. Stata, A. Tomkins, J. Wiener, A. Borodin, G. O. Roberts, J. S. Rosenthal, and P. Tsaparas. Graph structure of the web. In Proceedings of the ninth international conference on World Wide Web, pages 309--320. Foretec Seminars, Inc., 2000.

Digital Library

[9]

F. Chung and L. Lu. Coupling on-line and off-line analyses for random power graphs. Internet Mathematics, 1(4):409--461, 2003.

[10]

A. F. Colin Cooper and J. Vera. Random deletion in a scale free random graph process. Internet Mathematics, 1(4):463--483, 2003.

[11]

K. Efe, V. Raghavan, C. H. Chu, A. L. Broadwater, L. Bolelli, and S. Ertekin. The shape of the Web and its implications for searching the Web, 31--6 2000.

[12]

N. Eiron, K. S. McCurley, and J. A. Tomlin. Ranking the web frontier. In WWW conference, 2004.

Digital Library

[13]

P. Erdös and A. Rényi. On random graphs. In Publicationes of the Mathematicae, volume 6, 1959.

[14]

H. Ino, M. Kudo, and A. Nakamura. Partitioning of web graphs by community topology. In WWW '05: Proceedings of the 14th international conference on World Wide Web, pages 661--669, 2005.

Digital Library

[15]

J. M. Kleinberg. Authoritative sources in a hyperlinked environment. J. ACM, 46(5):604--632, 1999.

Digital Library

[16]

J. M. Kleinberg. Hubs, authorities, and communities. ACM Computing Surveys (CSUR), 31(4es):5, 1999.

Digital Library

[17]

R. Kumar, P. Raghavan, S. Rajagopalan, D. Sivakumar, A. Tomkins, and E. Upfal. Stochastic models for the web graph. In Proceedings of the 41st Annual Symposium on Foundations of Computer Science, page 57. IEEE Computer Society, 2000.

Digital Library

[18]

R. Kumar, P. Raghavan, S. Rajagopalan, and A. Tomkins. Extracting large-scale knowledge bases from the web. In VLDB, pages 639--650, 1999.

Digital Library

[19]

R. Kumar, P. Raghavan, S. Rajagopalan, and A. Tomkins. Trawling the web for emerging cyber-communities. In WWW conference, 1999.

Digital Library

[20]

M. Najork and J. L. Wiener. Breadth-first crawling yields high-quality pages. In Proceedings International WWW Conference(10), Hong-Kong, 2001.

Digital Library

[21]

L. Page, S. Brin, R. Motwani, and T. Winograd. The PageRank Citation Ranking: Bringing Order to the Web. Technical report, Computer Science Department, Stanford University, 1998.

[22]

G. Pandurangan, P. Raghavan, and E. Upfal. Using pagerank to characterize web structure. In 8th Annual International Conference on Computing and Combinatorics, pages 330--339, 2002.

Digital Library

[23]

Nature. 405:113, 11 May 2000.

[24]

D. J. Watts and S. H. Strogatz. Collective dynamics of small-world networks. Nature, 393(1-7):440--442, 1998.

Cited By

Yang HCui HTang H(2017)Web page sorting algorithm based on query keyword distance relation10.1063/1.4992844(020027)Online publication date: 2017
https://doi.org/10.1063/1.4992844
Xiang GPendleton BHong JRose C(2010)A hierarchical adaptive probabilistic approach for zero hour phish detectionProceedings of the 15th European conference on Research in computer security10.5555/1888881.1888903(268-285)Online publication date: 20-Sep-2010
https://dl.acm.org/doi/10.5555/1888881.1888903
Xiang GPendleton BHong JRose C(2010)A Hierarchical Adaptive Probabilistic Approach for Zero Hour Phish DetectionComputer Security – ESORICS 201010.1007/978-3-642-15497-3_17(268-285)Online publication date: 2010
https://doi.org/10.1007/978-3-642-15497-3_17
Show More Cited By

Index Terms

Random web crawls
1. Computing methodologies
  1. Modeling and simulation

Recommendations

Current challenges in web crawling
ICWE'13: Proceedings of the 13th international conference on Web Engineering

Web crawling, a process of collecting web pages in an automated manner, is the primary and ubiquitous operation used by a large number of web systems and agents starting from a simple program for website backup to a major web search engine. Due to an ...
Intelligent crawling of web applications for web archiving
WWW '12 Companion: Proceedings of the 21st International Conference on World Wide Web

The steady growth of the World Wide Web raises challenges regarding the preservation of meaningful Web data. Tools used currently by Web archivists blindly crawl and store Web pages found while crawling, disregarding the kind of Web site currently ...
Effective web-scale crawling through website analysis
WWW '06: Proceedings of the 15th international conference on World Wide Web

The web crawler space is often delimited into two general areas: full-web crawling and focused crawling. We present netSifter, a crawler system which integrates features from these two areas to provide an effective mechanism for web-scale crawling. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '07: Proceedings of the 16th international conference on World Wide Web

May 2007

1382 pages

ISBN:9781595936547

DOI:10.1145/1242572

General Chairs:
Carey Williamson
University of Calgary, Canada
,
Mary Ellen Zurko
IBM, USA
,
Program Chairs:
Peter Patel-Schneider
Bell Labs Research, USA
,
Prashant Shenoy
University of Massachusetts at Amherst, USA

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 May 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

WWW'07

Sponsor:

ACM

WWW'07: 16th International World Wide Web Conference

May 8 - 12, 2007

Alberta, Banff, Canada

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
502
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)3

Reflects downloads up to 02 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yang HCui HTang H(2017)Web page sorting algorithm based on query keyword distance relation10.1063/1.4992844(020027)Online publication date: 2017
https://doi.org/10.1063/1.4992844
Xiang GPendleton BHong JRose C(2010)A hierarchical adaptive probabilistic approach for zero hour phish detectionProceedings of the 15th European conference on Research in computer security10.5555/1888881.1888903(268-285)Online publication date: 20-Sep-2010
https://dl.acm.org/doi/10.5555/1888881.1888903
Xiang GPendleton BHong JRose C(2010)A Hierarchical Adaptive Probabilistic Approach for Zero Hour Phish DetectionComputer Security – ESORICS 201010.1007/978-3-642-15497-3_17(268-285)Online publication date: 2010
https://doi.org/10.1007/978-3-642-15497-3_17
Alaba OSpyratos NKapetanios ETraina AChbeir RBadr Y(2009)Networks, fractals and complexityProceedings of the International Conference on Management of Emergent Digital EcoSystems10.1145/1643823.1643827(16-20)Online publication date: 27-Oct-2009
https://dl.acm.org/doi/10.1145/1643823.1643827
Baeza-Yates RPereira ÁZiviani NHuai JChen RHon HLiu YMa WTomkins AZhang X(2008)Genealogical trees on the webProceedings of the 17th international conference on World Wide Web10.1145/1367497.1367548(367-376)Online publication date: 21-Apr-2008
https://dl.acm.org/doi/10.1145/1367497.1367548
Buluc AGilbert J(2008)On the representation and multiplication of hypersparse matrices2008 IEEE International Symposium on Parallel and Distributed Processing10.1109/IPDPS.2008.4536313(1-11)Online publication date: Apr-2008
https://doi.org/10.1109/IPDPS.2008.4536313
Dany C(undefined)Structural Power in Internet Governance. Enabling and Constraining Effects of the Institutionalization of NGO Participation at the WSISSSRN Electronic Journal10.2139/ssrn.2798300
https://doi.org/10.2139/ssrn.2798300

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents