Building Data-Intensive Grid Applications with Globus Toolkit --- An Evaluation Based on Web Crawling
Pages 570 - 581
Abstract
Nowadays, there is a trend to create resource-consuming applications without building heavy computer centers, but to use resources on computer systems distributed over the internet. Grid middleware is a framework to access these resources. The concern of this paper is the evaluation of a specific grid middleware, namely Globus Toolkit, for data-intensive applications. As a test case, we have designed and implemented a service-based distributed web crawler on top of this middleware: A web crawler is a complex application consisting of many nodes. It imposes significantly higher demands on grid middleware regarding administrative flexibility compared to grid applications that allocate computing power of grid nodes. We have observed that some components of Globus Toolkit are flexible enough to provide the control functionality necessary for a web crawler, while others are not. For these other components, we propose possible extensions. Since we expect the combination of those characteristics to occur with many other grid applications as well, our study is of broader interest, beyond web crawling.
References
[1]
Austin, J.: DAME - Distributed Aircraft Maintenance Environment: (last visited 2006-07- 24) (2004) http://www.cs.york.ac.uk/dame/
[2]
Bharat, K., et al.: Who links to whom: Mining linkage between web sites. In: ICDM '01. Proceedings of the IEEE, International Conference on Data Mining, San Jose, USA, IEEE Computer Society Press, Los Alamitos (2001).
[3]
BOINC, http://boinc.berkeley.edu
[4]
Brin, S., Page, L.: The anatomy of a large-scale hyper textual Web search engine. In: Computer Networks and ISDN Systems, vol. 30 (1998).
[5]
Chinnici, R., et al.: Web Services Description Language (WSDL) Version 2.0, W3C Whitepaper last visited (2006-07-24) (March 2006), http://www.w3.org/TR/2006/CRwsdl20- 20060327/
[6]
Condor - High Throughput Computing, http://www.cs.wisc.edu/condor
[7]
Foster, I., Kesselman, C.: The Anatomy of the Grid. In: Sakellariou, R., Keane, J.A., Gurd, J.R., Freeman, L. (eds.) Euro-Par 2001. LNCS, vol. 2150, Springer, Heidelberg (2001).
[8]
Foster, I., Kesselman, C., Nick, J., Tuecke, S.: The Physiology of the Grid, Global Grid Forum (June 2002).
[9]
Foster, I., Kesselman, C.: The Grid. Blueprint for a New Computing Infrastructure, 2nd edn. Morgan Kaufmann Publishers, San Francisco (2003).
[10]
Foster, I.: Globus Toolkit Version 4: Software for Service-Oriented Systems. In: Jin, H., Reed, D., Jiang, W. (eds.) NPC 2005. LNCS, vol. 3779, Springer, Heidelberg (2005).
[11]
Globus Toolkit, http://www.globus.org
[12]
Gray, J., Szalay, A.: The World Wide Telescope. Science Bd. 293 (2002).
[13]
Gudgin, et al.: Web Services Addressing 1.0 - SOAP Binding, W3C Whitepaper, (March 2006).
[14]
Planet Lab, http://www.planet-lab.org
[15]
Shkapenyuk, V., Suel, T.: Design and implementation of a high-performance distributed Web crawler. In: Proceedings of the 18th International Conference on Data Engineering, San Jose, pp. 357-368 (2002).
[16]
Sun N1 Grid Engine, http://www.sun.com/software/gridware/
[17]
Tomcat 5.5, tomcat.apache.org
[18]
The OGSA-DAI Project, http://www.ogsadai.org.uk
[19]
UNICORE, http://www.unicore.com
[20]
Walter, A., Schosser, S., Böhm, K: Überlegungen zur Entwicklung komplexer Grid-Anwendungen mit Globus Toolkit. In: Proceedings of the GI Fachtagung für Datenbanksysteme, Technologie und Web (BTW), Aachen, Germany (2007).
- Building Data-Intensive Grid Applications with Globus Toolkit --- An Evaluation Based on Web Crawling
Recommendations
MGC middleware for grid computing: the Globus Toolkit
ACAI '11: Proceedings of the International Conference on Advances in Computing and Artificial IntelligenceGrid computing has made substantial advances during the last decade. A major concern in Grid environments is dealing with the high degree of heterogeneity of resources that can range from laptops and PCs to supercomputers. The unified virtual view of ...
The Grid Resource Broker workflow engine
2nd International Workshop on Workflow Management and Applications in Grid Environments (WaGe2007)Increasingly, complex scientific applications are structured in terms of workflows. These applications are usually computationally and-or data intensive and thus are well suited for execution in grid environments. Distributed, geographically spread ...
Brokering multi-grid workflows in the P-GRADE portal
Euro-Par'06: Proceedings of the CoreGRID 2006, UNICORE Summit 2006, Petascale Computational Biology and Bioinformatics conference on Parallel processingGrid computing has gone through some generations and as a result only a few widely used middleware architectures remain. The Globus Toolkit is the most widespread middleware in most of the current production grid systems, but the LCG-2 middleware ...
Comments
Information & Contributors
Information
Published In
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 17 September 2007
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 25 Dec 2024