research-article

Towards improved load balancing for data intensive distributed computing

Authors:

Masaru KitsuregawaAuthors Info & Claims

SAC '11: Proceedings of the 2011 ACM Symposium on Applied Computing

Pages 139 - 146

https://doi.org/10.1145/1982185.1982220

Published: 21 March 2011 Publication History

Abstract

Specialized frameworks for highly scalable data processing continue to gain prominence over traditional databases in many environments including the cloud. Perhaps the most well-known such framework is Google MapReduce, which has gained wide-spread popularity. However, the MapReduce model offers some significant challenges for workload balancing which have not been adequately explored so far. In this paper, we introduce techniques for improving load balancing -- particularly multi-stage jobs and dynamic partition assignment -- by using a modified programming model that offers greater flexibility but maintains the simplicity, scalability and fault tolerance of MapReduce. We then explore the effectiveness of our approach using a parallel frequent itemset mining algorithm.

References

[1]

R. Agrawal and R. Srikant. Quest synthetic data generator. IBM Almaden Research Center.

[2]

Amazon. Elastic MapReduce. http://aws.amazon.com/elasticmapreduce/.

[3]

Apache. Hadoop Core. http://hadoop.apache.org/core.

[4]

Apache. Mahout. http://mahout.apache.org/.

[5]

S. Babu. Towards automatic optimization of MapReduce programs. In Proc. of SoCC, pages 137--142, New York, NY, USA, 2010. ACM.

Digital Library

[6]

J. Dean and S. Ghemawat. MapReduce: simplified data processing on large clusters. In Proc. of OSDI, pages 137--150, Berkeley, CA, USA, 2004. USENIX.

Digital Library

[7]

J. Dean and S. Ghemawat. MapReduce: a flexible data processing tool. Commun. ACM, 53(1): 72--77, 2010.

Digital Library

[8]

S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google file system. In Proc. of SOSP, pages 29--43, New York, NY, USA, 2003. ACM.

Digital Library

[9]

K. Goda, T. Tamura, M. Oguchi, and M. Kitsuregawa. Run-time load balancing system on SAN-connected PC cluster for dynamic injection of CPU and disk resource -- a case study of data mining application --. In Proc. of DEXA, pages 182--192. Springer, 2002.

Digital Library

[10]

S. Groot, K. Goda, and M. Kitsuregawa. A study on workload imbalance issues in data intensive distributed computing. In Proc. of DNIS, pages 27--32. Springer, 2010.

Digital Library

[11]

J. Han, J. Pei, and Y. Yin. Mining frequent patterns without candidate generation. SIGMOD Rec., 29(2): 1--12, 2000.

Digital Library

[12]

M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. SIGOPS Oper. Syst. Rev., 41(3): 59--72, 2007.

Digital Library

[13]

K. Kambatla, A. Pathak, and H. Pucha. Towards optimizing hadoop provisioning in the cloud. In Proc. of HotCloud, Berkeley, CA, USA, 2009. USENIX.

Digital Library

[14]

H. Li, Y. Wang, D. Zhang, M. Zhang, and E. Y. Chang. PFP: Parallel FP-growth for query recommendation. In RecSys, pages 107--114, New York, NY, USA, 2008. ACM.

Digital Library

[15]

A. Pavlo, E. Paulson, A. Rasin, D. J. Abadi, D. J. DeWitt, S. Madden, and M. Stonebraker. A comparison of approaches to large-scale data analysis. In Proc. of SIGMOD, pages 165--178, New York, NY, USA, 2009. ACM.

Digital Library

[16]

M. Randles, D. Lamb, and A. Taleb-Bendiab. A comparative study into distributed load balancing algorithms for cloud computing. In Proc. of WAINA, pages 551--556, Los Alamitos, CA, USA, 2010. IEEE.

Digital Library

[17]

M. Tamura and M. Kitsuregawa. Dynamic load balancing for parallel association rule mining on heterogenous pc cluster systems. In Proc. of VLDB, pages 162--173, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc.

Digital Library

[18]

C. Yang, C. Yen, C. Tan, and S. Madden. Osprey: Implementing MapReduce-style fault tolerance in a shared-nothing distributed database. In Proc. of ICDE, pages 657--668, Los Alamitos, CA, USA, 2010. IEEE.

[19]

M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica. Improving MapReduce performance in heterogeneous environments. In Proc. of OSDI, pages 29--42, Berkeley, CA, USA, 2008. USENIX.

Digital Library

Cited By

Xun YZhang JQin XZhao X(2017)FiDoop-DPIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2016.256017628:1(101-114)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1109/TPDS.2016.2560176
Wang JZeng Y(2014)The Optimization of Parallel Frequent Pattern Growth Algorithm Based on Mahout in Cloud Manufacturing EnvironmentProceedings of the 2014 Seventh International Symposium on Computational Intelligence and Design - Volume 0210.1109/ISCID.2014.258(420-423)Online publication date: 13-Dec-2014
https://dl.acm.org/doi/10.1109/ISCID.2014.258

Recommendations

Modeling I/O interference for data intensive distributed applications
SAC '13: Proceedings of the 28th Annual ACM Symposium on Applied Computing

Data intensive applications such as MapReduce can have large performance degradation from the effects of I/O interference when multiple processes access the same I/O resources simultaneously, particularly in the case of disks. It is necessary to ...
The Automatic Transformation of Linked List Data Structures
PACT '07: Proceedings of the 16th International Conference on Parallel Architecture and Compilation Techniques

Irregular access patterns are a major problem for today's optimizing compilers. It prevents the application of many optimizations as data dependence analysis cannot be performed on code containing such patterns. We present a novel approach that ...
Modeling I/O Interference in Data Intensive Map-Reduce Applications
SAINT '12: Proceedings of the 2012 IEEE/IPSJ 12th International Symposium on Applications and the Internet

Map-Reduce is a popular framework for very large-scale data mining and processing. Recently, some works have attempted to model the behavior of Map-Reduce, but these existing models ignore the non-linearity of disk I/O performance under contention, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '11: Proceedings of the 2011 ACM Symposium on Applied Computing

March 2011

1868 pages

ISBN:9781450301138

DOI:10.1145/1982185

Conference Chairs:
William Chu
Tunghai University, TaiChung, Taiwan
,
W. Eric Wong
University of Texas at Dallas, Richardson, Texas
,
Program Chairs:
Mathew J. Palakal
Indiana University Purdue University, Indianapolis
,
Chih-Cheng Hung
Southern Polytechnic State University, Marietta

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 March 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

SAC'11

Sponsor:

SIGAPP

SAC'11: The 2011 ACM Symposium on Applied Computing

March 21 - 24, 2011

TaiChung, Taiwan

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
452
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xun YZhang JQin XZhao X(2017)FiDoop-DPIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2016.256017628:1(101-114)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1109/TPDS.2016.2560176
Wang JZeng Y(2014)The Optimization of Parallel Frequent Pattern Growth Algorithm Based on Mahout in Cloud Manufacturing EnvironmentProceedings of the 2014 Seventh International Symposium on Computational Intelligence and Design - Volume 0210.1109/ISCID.2014.258(420-423)Online publication date: 13-Dec-2014
https://dl.acm.org/doi/10.1109/ISCID.2014.258

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents