research-article

Databases on the web: national web domain survey

Author:

Denis ShestakovAuthors Info & Claims

IDEAS '11: Proceedings of the 15th Symposium on International Database Engineering & Applications

Pages 179 - 184

https://doi.org/10.1145/2076623.2076646

Published: 21 September 2011 Publication History

Abstract

The deep Web, the part of the Web consisting of web pages filled with information from myriads of online databases, is to date relatively unexplored. Even its basic characteristics such as, for instance, the number of searchable databases on the Web are disputable. In this paper, we address the problem of accurate estimation of the deep Web by sampling one national web domain. We report some of our results obtained when surveying the Russian Web. The survey findings, namely the size estimates of the deep Web, could be useful for further studies to handle data in the deep Web.

References

[1]

Internet Archive's snapshot of Yandex statistics page as of September 18, 2006. http://web.archive.org/web/20060918081218/http://company.yandex.ru/.

[2]

Runet in March 2006: domains, hosting, geographical location. http://www.rukv.ru/analytics-200603.html. In Russian.

[3]

Runet in March 2007: domains, hosting, geographical location. http://www.rukv.ru/runet-2007.html. In Russian.

[4]

April 2004 Web Server Survey. http://news.netcraft.com/archives/2004/04/01/april_2004_web_server_survey.html, April 2004.

[5]

DNS load balancing report. http://www.securityspace.com/s_survey/data/man.200404/dnsmult.html, April 2004.

[6]

R. Baeza-Yates and C. Castillo. Crawling the infinite Web: five levels are enough. In Proceedings of the third Workshop on Web Graphs (WAW), pages 156--167, 2004.

[7]

R. Baeza-Yates, C. Castillo, and E. N. Efthimiadis. Characterization of national Web domains. ACM Trans. Internet Technol., 7(2), 2007.

Digital Library

[8]

R. Baeza-Yates, C. Castillo, and V. López. Characteristics of the Web of Spain. Cybermetrics, 9(1), 2005.

[9]

M. Bergman. The deep Web: surfacing hidden value. Journal of Electronic Publishing, 7(1), 2001.

[10]

K. Bharat and A. Broder. A technique for measuring the relative size and overlap of public web search engines. Comput. Netw. ISDN Syst., 30(1--7):379--388, 1998.

Digital Library

[11]

K. Bharat, A. Broder, J. Dean, and M. Henzinger. A comparison of techniques to find mirrored hosts on the WWW. J. Am. Soc. Inf. Sci., 51(12):1114--1122, 2000.

Digital Library

[12]

M. J. Cafarella, A. Halevy, and J. Madhavan. Structured data on the Web. Commun. ACM, 54:72--79, 2011.

Digital Library

[13]

K. C.-C. Chang, B. He, C. Li, M. Patel, and Z. Zhang. Structured databases on the Web: observations and implications. SIGMOD Rec., 33(3):61--70, 2004.

Digital Library

[14]

D. Gomes and M. J. Silva. Characterizing a national community web. ACM Trans. Internet Technol., 5(3):508--531, 2005.

Digital Library

[15]

C. A. Lynch. The Z39.50 information retrieval protocol: an overview and status report. SIGCOMM Comput. Commun. Rev., 21(1):58--70, 1991.

Digital Library

[16]

E. T. O'Neill, P. D. McClain, and B. F. Lavoie. A methodology for sampling the World Wide Web. Annual Review of OCLC Research 1997, 1997.

[17]

D. Shestakov. Deep Web: databases on the Web. In: Handbook of Research on Innovations in Database Technologies and Applications, pp. 581--588, IGI Global (2009).

[18]

D. Shestakov. Sampling the national deep Web. In Proceedings of DEXA 2011, pages 331--340, 2011.

Digital Library

[19]

D. Shestakov and T. Salakoski. On estimating the scale of national deep Web. In Proceedings of DEXA'07, pages 780--789, 2007.

Digital Library

[20]

D. Shestakov and T. Salakoski. Characterization of national deep Web. Technical Report 892, Turku Centre for Computer Science, May 2008.

[21]

G. Tolosa, F. Bordignon, R. Baeza-Yates, and C. Castillo. Characterization of the Argentinian Web. Cybermetrics, 11(1), 2007.

Cited By

Deshmukh M(2019)2 Way CrawlingInternational Journal of Applied Evolutionary Computation10.4018/IJAEC.201907010510:3(34-39)Online publication date: 1-Jul-2019
https://doi.org/10.4018/IJAEC.2019070105
Díaz MÁlvarez VSeijo F(2018)The Evolution of the (Hidden) Web and Its Hidden DataThe Dark Web10.4018/978-1-5225-3163-0.ch006(84-113)Online publication date: 2018
https://doi.org/10.4018/978-1-5225-3163-0.ch006
Mahale VDhande MPandit A(2018)Advanced Web Crawler For Deep Web Interface Using Binary Vector & Page Rank2018 2nd International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC)I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), 2018 2nd International Conference on10.1109/I-SMAC.2018.8653765(500-503)Online publication date: Aug-2018
https://doi.org/10.1109/I-SMAC.2018.8653765
Show More Cited By

Index Terms

Databases on the web: national web domain survey
1. Applied computing
  1. Computers in other domains
    1. Digital libraries and archives
2. Information systems

Recommendations

Characterization of national Web domains

During the last few years, several studies on the characterization of the public Web space of various national domains have been published. The pages of a country are an interesting set for studying the characteristics of the Web because at the same ...
Sampling the national deep web
DEXA'11: Proceedings of the 22nd international conference on Database and expert systems applications - Volume Part I

A huge portion of today's Web consists of web pages filled with information from myriads of online databases. This part of theWeb, known as the deep Web, is to date relatively unexplored and even major characteristics such as number of searchable ...
Host-IP clustering technique for deep web characterization
SAC '10: Proceedings of the 2010 ACM Symposium on Applied Computing

The part of the Web, known as the deep Web, is to date relatively unexplored and even major characteristics such as number of searchable databases on the Web is somewhat disputable. In this paper, we are aimed at more accurate estimation of main ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

IDEAS '11: Proceedings of the 15th Symposium on International Database Engineering & Applications

September 2011

274 pages

ISBN:9781450306270

DOI:10.1145/2076623

General Chair:
Bipin C. Desai
Concordia University
,
Program Chairs:
Isabel Cruz
University of Illinois at Chicago
,
Jorge Bernardino
Instituto Politecnico de Coimbra

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 September 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IDEAS '11

IDEAS '11: 15th International Database Engineering & Applications Symposium

September 21 - 23, 2011

Lisboa, Portugal

Acceptance Rates

Overall Acceptance Rate 74 of 210 submissions, 35%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
193
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Deshmukh M(2019)2 Way CrawlingInternational Journal of Applied Evolutionary Computation10.4018/IJAEC.201907010510:3(34-39)Online publication date: 1-Jul-2019
https://doi.org/10.4018/IJAEC.2019070105
Díaz MÁlvarez VSeijo F(2018)The Evolution of the (Hidden) Web and Its Hidden DataThe Dark Web10.4018/978-1-5225-3163-0.ch006(84-113)Online publication date: 2018
https://doi.org/10.4018/978-1-5225-3163-0.ch006
Mahale VDhande MPandit A(2018)Advanced Web Crawler For Deep Web Interface Using Binary Vector & Page Rank2018 2nd International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC)I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), 2018 2nd International Conference on10.1109/I-SMAC.2018.8653765(500-503)Online publication date: Aug-2018
https://doi.org/10.1109/I-SMAC.2018.8653765
Bhakare UChatur P(2017)Content extraction from deep web interfaces2017 International conference of Electronics, Communication and Aerospace Technology (ICECA)10.1109/ICECA.2017.8203702(349-353)Online publication date: Apr-2017
https://doi.org/10.1109/ICECA.2017.8203702
Zhao FZhou JNie CHuang HJin H(2016)SmartCrawler: A Two-Stage Crawler for Efficiently Harvesting Deep-Web InterfacesIEEE Transactions on Services Computing10.1109/TSC.2015.24149319:4(608-620)Online publication date: 1-Jul-2016
https://doi.org/10.1109/TSC.2015.2414931
Sundarde SRathod P(2016)Smart crawler for hidden web interfaces2016 Online International Conference on Green Engineering and Technologies (IC-GET)10.1109/GET.2016.7916710(1-4)Online publication date: Nov-2016
https://doi.org/10.1109/GET.2016.7916710
Díaz MÁlvarez VSeijo F(2015)The Evolution of the (Hidden) Web and its Hidden DataDesign Strategies and Innovations in Multimedia Presentations10.4018/978-1-4666-8696-0.ch001(1-30)Online publication date: 2015
https://doi.org/10.4018/978-1-4666-8696-0.ch001

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents