Towards dynamic data grid framework for eResearch
Abstract
The scale at which scientific data is produced will undergo a massive change in the near future. Many sophisticated scientific discovery laboratories or the installation of sensor networks would produce a large amount of data. Research in protein crystallography for instance can produce hundreds of Terabytes of data from a single crystallography beamline. These data have to be saved for future use and made available for collaborative use by researchers. There is a need to develop a framework which can deal with storing such data volumes. This framework should also handle disparate data sources tightly integrated with the users' applications and large data streams arising from instruments and sensors. This paper presents an initial study into the framework for servicing large dynamic data sets over a national grid for eResearch.
References
[1]
{1} Australia's Academic and Research Network. Available online: http://www.aarnet.edu.au/engineering/aarnet3/]]
[2]
{2} Allcock, W., Bester, J., Bresnahan, J., Chervenak, A., Liming, L., Meder, S., Tuecke, S., "GridFTP protocol specification", Technical report, Global Grid Forum GridFTP Working Group, September 2002.]]
[3]
{3} Atkinson, M., "Grid & Web services: Web Service Resource Framework (WSRF), Presentation at Core e-Science Programme Town Meeting to Discuss Grid & Web Services, Friday, 23rd April 2004, Jolly St Ermins Hotel, London.]]
[4]
{4} Almond, J. and Snelling, D., "UNICORE: uniform access to supercomputing as an element of electronic commerce", Future Generation Computer Systems, 15, pp. 539-548, NH-Elsevier, 1999.]]
[5]
{5} Australian Synchrotron. Available online: http://www.synchrotron.vic.gov.au/]]
[6]
{6} Bramley, R., Chiu, K., Huffman, J.C., Huffman, K. and McMullen, D.F., "Instruments and Sensors as Network Services: Making Instruments First Class Members of the Grid, Indiana University CS Department Technical Report 588, 2003.]]
[7]
{7} The Biomedical Informatics Research Network. Available online: http://www.nbirn.net]]
[8]
{8} Buyya, R., "The Virtual Laboratory Project: Molecular Modeling for Drug Design on Grid", IEEE Distributed Systems Online, Vol. 2, No. 5, 2001.]]
[9]
{9} Condor. Available online: http://www.cs.wisc.edu/condor/]]
[10]
{10} European DataGrid, Available online: http://eu-datagrid.web.cern.ch/eu-datagrid/]]
[11]
{11} Foster, I., Kesselman, C., The Grid: Blueprint for a New Computing Infrastructure, First Edition, Morgan Kaufmann Publisher, 1999.]]
[12]
{12} Foster, I., Kesselman, C., The Grid - Blueprint for a New Computing Infrastructure, Second Edition, Morgan Kaufman Publisher, 2004.]]
[13]
{13} Freier, A.O. Karlton, P. Kocher, P.C. "The SSL Protocol Version 3.0 Transport Layer Security Working Group INTERNET-DRAFT", 18 November 1996. Available online: http://wp.netscape.com/eng/ssl3/]]
[14]
{14} Foster, I., Kesselman, C., Tsudik, G., and Tuecke, S., "A Security Architecture for Computational Grid", 5th ACM conference on Computer and Communications Security, New York, 1998, pp. 83-91.]]
[15]
{15} Globus Project (The Globus Alliance). Available online: http://www.globus.org]]
[16]
{16} GRID computing and BUSiness technologies (GRIDBUS). Available online: http://www.gridbus.org/]]
[17]
{17} Grimshaw, A., Wulf, W. et al., "The Legion Vision of a Worldwide Virtual Computer". Communications of the ACM, vol. 40(1), January 1997.]]
[18]
{18} Grid and Next Generation Network. Available online: http://www.grangenet.net/]]
[19]
{19} Hey, T., Trefethen, A.E., "The UK e-Science Core Programme and the Grid," Future Generation Computer Systems, vol. 18, pp. 1017-1031, 2002.]]
[20]
{20} Housley, R. Ford, W. Polk, W. Solo, D., "RFC 2459 Internet X.509 Public Key Infrastructure Certificate and CRL Profile", January 1999.]]
[21]
{21} Iperf. Available online: http://dast.nlanr.net/Project/Iperf]]
[22]
{22} The Joint Information Systems Committee, 2004. "The Data Deluge: Preparing for the explosion in data", Available online: http://www.jisc.ac.uk/index.cfm?name=pub_datadeluge]]
[23]
{23} Johanstona, W.E., "Computational and Data Grids in Large-Scale Science & Engineering", Future Generation Computer Systems, Vol. 18, No. 8, October, 2002.]]
[24]
{24} Kerberos: The Network Authentication Protocol. Available online: http://web.mit.edu/kerberos/www/]]
[25]
{25} Load Sharing Facility, Platform Computing Inc. Available online: http://www.platform.com/products/LSF/]]
[26]
{26} The Long Term Ecological Research (LTER) Network. Avaiable online: http://www.lternet.edu/]]
[27]
{27} Networked File Systems (NFS), RFC 1094 (Version 2), RFC 1814 (Version 3 ), RFC 3010 (Version 4)]]
[28]
{28} Novotny, J. Tuecke, S. and Welch, V. "An Online Credential Repository for the Grid: MyProxy". Proceedings of the Tenth International Symposium on High Performance Distributed Computing (HPDC-10), IEEE Press, August 2001.]]
[29]
{29} OpenAFS UserGuide. Avaiable online: http://www.openafs.org/pages/doc/UserGuide/auusg010.htm]]
[30]
{30} Portable Batch System, Altair Grid Technologies Online: http://www.openpbs.org/]]
[31]
{31} Particle Physics data Grid. Available online: http://www.ppdg.net]]
[32]
{32} Rajasekar, A.K. and Moore, R.W. "Data and Metadata Collections for Scientific Applications", European High Performance Computing conference, Amsterdam, Holland, June 26, 2001.]]
[33]
{33} Rajasekar, A., Wan, M. Moore, R. "MySRB & SRB - Components of a Data Grid", The 11th International Symposium on High Performance Distributed Computing (HPDC-11) Edinburgh, Scotland, July 2002.]]
[34]
{34} Security Assertion Markup Language Technical Overview. Available online: http://xml.coverpages.org/SAML- TechOverviewV20-Draft7874.pdf]]
[35]
{35} Shibboleth. Available online: http://shibboleth.internet2.edu/shib-tech-intro.html]]
[36]
{36} Sun N1 Grid Engine 6, Sun Microsystems Inc. Online: http://www.sun.com/software/gridware/index.xml]]
[37]
{37} Sun, H, et al, 1994, "Study on an algorithm of multisensor data fusion", Proc. IEEE Proceedings of the National Aerospace and Electronics Conference, Vol.1, pp.239-245.]]
[38]
{38} Tuecke, S. Welch, V. Engert, D. Pearlman, L. Thompson, M. "RFC3820 Internet X.509 Public Key Infrastructure (PKI) Proxy Certificate Profile", Jun 2004.]]
[39]
{39} Wald, L., "Some terms of reference in data fusion". IEEE Transactions on Geosciences and Remote Sensing, 37, 3, 1190- 1193, 1999.]]
[40]
{40} Welch, V. Foster, I. Kesselman, C. Mulmo, O. Pearlman, L. Tuecke, S. Gawor, J. Meder, S. Siebenlist, F. "X.509 Proxy Certificates for Dynamic Delegation". 3rd Annual PKI R&D Workshop, 2004.]]
Index Terms
- Towards dynamic data grid framework for eResearch
Recommendations
Data management and transfer in high-performance computational grid environments
Parallel data-intensive algorithms and applicationsAn emerging class of data-intensive applications involve the geographically dispersed extraction of complex scientific information from very large collections of measured or computed data. Such applications arise, for example, in experimental physics, ...
Performance analysis of applying replica selection technology for data grid environments
PaCT'05: Proceedings of the 8th international conference on Parallel Computing TechnologiesThe Data Grid enables the sharing, selection, and connection of a wide variety of geographically distributed computational and storage resources for solving large-scale data intensive scientific applications. Such technology efficiently manage and ...
Workflow Level Interoperation of Grid Data Resources
CCGRID '08: Proceedings of the 2008 Eighth IEEE International Symposium on Cluster Computing and the GridThe lack of widely accepted standards and the use of different middleware solutions divide today’s Grid resources into non-interoperable production Grid islands. On the other hand, more and more experiments require such a large number of resources that ...
Comments
Information & Contributors
Information
Published In
January 2006
230 pages
ISBN:1920682368
Publisher
Australian Computer Society, Inc.
Australia
Publication History
Published: 01 January 2006
Author Tags
Qualifiers
- Article
Conference
ACSW Frontiers '06
Acceptance Rates
Overall Acceptance Rate 204 of 424 submissions, 48%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 441Total Downloads
- Downloads (Last 12 months)9
- Downloads (Last 6 weeks)4
Reflects downloads up to 22 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in