research-article

freeCycles: efficient data distribution for volunteer computing

Authors:

Paulo FerreiraAuthors Info & Claims

CloudDP '14: Proceedings of the Fourth International Workshop on Cloud Data and Platforms

Article No.: 4, Pages 1 - 6

https://doi.org/10.1145/2592784.2592788

Published: 13 April 2014 Publication History

Abstract

Volunteer Computing (VC) has been proving to be a way to access large amounts of computational power, network bandwidth and storage. With the recent developments of new programming paradigms and their adaptation to run on the large scale Internet, we believe that data distribution techniques need to be re-thought in order to cope with the high volumes of information handled by, for example, MapReduce. Thus, we present a VC solution called freeCycles, that supports MapReduce jobs. freeCycles presents two new contributions: i) improves data distribution (among mappers and reducers) by using the BitTorrent protocol to distribute (input, intermediate and output) data, ii) improves intermediate data availability by replicating it through volunteers in order to avoid losing intermediate data and consequently preventing big delays on the MapReduce execution time.

References

[1]

A. Alexandrov, M. Ibel, K. Schauser, and C. Scheiman. Superweb: towards a global web-based parallel computing infrastructure. In Parallel Processing Symposium, 1997. Proceedings., 11th International, pages 100--106, 1997.

Digital Library

[2]

D. Anderson. Boinc: a system for public-resource computing and storage. In Grid Computing, 2004. Proceedings. Fifth IEEE/ACM International Workshop on, pages 4--10, 2004.

Digital Library

[3]

A. Baratloo, M. Karaul, Z. Kedem, and P. Wijckoff. Charlotte: Metacomputing on the web. Future Generation Computer Systems, 1999. ISSN 0167-739X.

Digital Library

[4]

A. Chakravarti, G. Baumgartner, and M. Lauria. The organic grid: self-organizing computation on a peer-to-peer network. Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on, 2005. ISSN 1083-4427.

Digital Library

[5]

L. Cherkasova and J. Lee. Fastreplica: Efficient large file distribution within content delivery networks. In USENIX Symposium on Internet Technologies and Systems, 2003.

Digital Library

[6]

F. Costa, L. Veiga, and P. Ferreira. Internet-scale support for map-reduce processing. Journal of Internet Services and Applications, 2013. ISSN 1867-4828.

[7]

J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. Commun. ACM, Jan. 2008. ISSN 0001-0782.

Digital Library

[8]

D. T. Fabrizio Marozzo and P. Trunfio. Adapting mapreduce for dynamic environments using a peer-to-peer model, 2008.

[9]

G. Fedak, C. Germain, V. Neri, and F. Cappello. Xtremweb: a generic global computing system. In Cluster Computing and the Grid, 2001. Proceedings. First IEEE/ACM International Symposium on, pages 582--587, 2001.

Digital Library

[10]

G. Fedak, H. He, and F. Cappello. Bitdew: A data management and distribution service with multi-protocol file transfer and metadata abstraction. Journal of Network and Computer Applications, 2009. ISSN 1084-8045.

Digital Library

[11]

S. Y. Ko, I. Hoque, B. Cho, and I. Gupta. Making cloud intermediate data fault-tolerant. In Proceedings of the 1st ACM symposium on Cloud computing, pages 181--192. ACM, 2010.

Digital Library

[12]

H. Lin, X. Ma, J. Archuleta, W.-c. Feng, M. Gardner, and Z. Zhang. Moon: Mapreduce on opportunistic environments. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, HPDC '10, pages 95--106, New York, NY, USA, 2010. ACM. ISBN 978-1-60558-942-8.

Digital Library

[13]

V. Lo, D. Zappala, D. Zhou, Y. Liu, and S. Zhao. Cluster computing on the fly: P2p scheduling of idle cycles in the internet. In Peer-to-Peer Systems III, pages 227--236. Springer, 2005.

Digital Library

[14]

J. Pouwelse, P. Garbacki, D. Epema, and H. Sips. The bittorrent p2p file-sharing system: Measurements and analysis. In Peer-to-Peer Systems IV, pages 205--216. Springer, 2005.

Digital Library

[15]

L. F. Sarmenta and S. Hirano. Bayanihan: building and studying web-based volunteer computing systems using java. Future Generation Computer Systems. ISSN 0167-739X.

Digital Library

[16]

M. Silberstein, A. Sharov, D. Geiger, and A. Schuster. Gridbot: execution of bags of tasks in multiple grids. In Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, SC '09, pages 11:1--11:12, New York, NY, USA, 2009. ACM. ISBN 978-1-60558-744-8.

Digital Library

[17]

B. Tang, M. Moca, S. Chevalier, H. He, and G. Fedak. Towards mapreduce for desktop grid computing. In P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC), 2010 International Conference on, pages 193--200, 2010.

Digital Library

[18]

D. Thain, T. Tannenbaum, and M. Livny. Distributed computing in practice: the condor experience. Concurrency and Computation: Practice and Experience, 2005. ISSN 1532-0634.

Digital Library

[19]

B. Wei, G. Fedak, and F. Cappello. Scheduling independent tasks sharing large data distributed with bittorrent. In Proceedings of the 6th IEEE/ACM International Workshop on Grid Computing, pages 219--226. IEEE Computer Society, 2005.

Digital Library

[20]

B. Wei, G. Fedak, and F. Cappello. Towards efficient data distribution on computational desktop grids with bittorrent. Future Generation Computer Systems, 2007.

Digital Library

[21]

T. White. Hadoop: the definitive guide. O'Reilly, 2012.

Digital Library

Cited By

Kratzke N(2020)Volunteer Down: How COVID-19 Created the Largest Idling Supercomputer on EarthFuture Internet10.3390/fi1206009812:6(98)Online publication date: 6-Jun-2020
https://doi.org/10.3390/fi12060098
Mengistu TChe D(2019)Survey and Taxonomy of Volunteer ComputingACM Computing Surveys10.1145/332007352:3(1-35)Online publication date: 3-Jul-2019
https://dl.acm.org/doi/10.1145/3320073
Filep L(2019)Model for Improved Load Balancing in Volunteer Computing PlatformsInformation Systems10.1007/978-3-030-11395-7_13(131-143)Online publication date: 12-Jan-2019
https://doi.org/10.1007/978-3-030-11395-7_13
Show More Cited By

Index Terms

freeCycles: efficient data distribution for volunteer computing

Recommendations

freeCycles - Efficient Multi-Cloud Computing Platform

The growing adoption of the MapReduce programming model increases the appeal of using Internet-wide computing platforms to run MapReduce applications on the Internet. However, current data distribution techniques, used in such platforms to distribute ...
SCADAMAR: scalable and data-efficient internet MapReduce
CCB '14: Proceedings of the 2nd International Workshop on CrossCloud Systems

Recent developments of popular programming models, namely MapReduce, have raised the interest of running MapReduce applications over the large scale Internet. However, current data distribution techniques used in Internet wide computing platforms to ...
MOON: MapReduce On Opportunistic eNvironments
HPDC '10: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

MapReduce offers an ease-of-use programming paradigm for processing large data sets, making it an attractive model for distributed volunteer computing systems. However, unlike on dedicated resources, where MapReduce has mostly been deployed, such ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CloudDP '14: Proceedings of the Fourth International Workshop on Cloud Data and Platforms

April 2014

41 pages

ISBN:9781450327145

DOI:10.1145/2592784

Program Chairs:
Zbigniew Jerzak
SAP AG, Germany
,
Etienne Rivière
University of Neuchâtel, Switzerland
,
Luis Veiga
Técnico Lisboa - ULisboa / INESC-ID Lisboa, Portugal

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 April 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Fundação para a Ciência e a Tecnologia

Conference

EuroSys 2014

Sponsor:

SIGOPS

EuroSys 2014: Ninth Eurosys Conference 2014

April 13, 2014

Amsterdam, The Netherlands

Acceptance Rates

CloudDP '14 Paper Acceptance Rate 6 of 16 submissions, 38%;

Overall Acceptance Rate 6 of 16 submissions, 38%

Upcoming Conference

EuroSys '25

Sponsor:
sigops

Twentieth European Conference on Computer Systems

March 30 - April 3, 2025

Rotterdam , Netherlands

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
138
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kratzke N(2020)Volunteer Down: How COVID-19 Created the Largest Idling Supercomputer on EarthFuture Internet10.3390/fi1206009812:6(98)Online publication date: 6-Jun-2020
https://doi.org/10.3390/fi12060098
Mengistu TChe D(2019)Survey and Taxonomy of Volunteer ComputingACM Computing Surveys10.1145/332007352:3(1-35)Online publication date: 3-Jul-2019
https://dl.acm.org/doi/10.1145/3320073
Filep L(2019)Model for Improved Load Balancing in Volunteer Computing PlatformsInformation Systems10.1007/978-3-030-11395-7_13(131-143)Online publication date: 12-Jan-2019
https://doi.org/10.1007/978-3-030-11395-7_13
Alonso-Monsalve SGarcía-Carballeira FCalderón A(2018)A heterogeneous mobile cloud computing model for hybrid cloudsFuture Generation Computer Systems10.1016/j.future.2018.04.00587:C(651-666)Online publication date: 1-Oct-2018
https://dl.acm.org/doi/10.1016/j.future.2018.04.005
Li WGuo W(2017)The Scalability of Volunteer Computing for MapReduce Big Data ApplicationsData Science10.1007/978-981-10-6385-5_14(153-165)Online publication date: 16-Sep-2017
https://doi.org/10.1007/978-981-10-6385-5_14
Alonso‐Monsalve SGarcía‐Carballeira FCalderón A(2017)A new volunteer computing model for data‐intensive applicationsConcurrency and Computation: Practice and Experience10.1002/cpe.419829:24Online publication date: 20-Jun-2017
https://doi.org/10.1002/cpe.4198
Tapparello CFunai CHijazi SAquino AKaraoglu BBa HShi JHeinzelman W(2016)Volunteer Computing on Mobile DevicesMobile Computing and Wireless Networks10.4018/978-1-4666-8751-6.ch095(2171-2198)Online publication date: 2016
https://doi.org/10.4018/978-1-4666-8751-6.ch095
Alonso-Monsalve SGarcía-Carballeira FCalderón A(2016)Improving the Performance of Volunteer Computing with Data Volunteers: A Case Study with the ATLAS@home ProjectAlgorithms and Architectures for Parallel Processing10.1007/978-3-319-49583-5_13(178-191)Online publication date: 25-Nov-2016
https://doi.org/10.1007/978-3-319-49583-5_13
Tapparello CFunai CHijazi SAquino AKaraoglu BBa HShi JHeinzelman W(2015)Volunteer Computing on Mobile DevicesEnabling Real-Time Mobile Cloud Computing through Emerging Technologies10.4018/978-1-4666-8662-5.ch005(153-181)Online publication date: 2015
https://doi.org/10.4018/978-1-4666-8662-5.ch005

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten