short-paper

The research of mapreduce load balancing based on multiple partition algorithm

Authors:

Suzhen Wang,

Haowei ZhouAuthors Info & Claims

UCC '16: Proceedings of the 9th International Conference on Utility and Cloud Computing

Pages 339 - 342

https://doi.org/10.1145/2996890.3007886

Published: 06 December 2016 Publication History

Get Access

Abstract

In this paper, we propose a strategy to solve the load imbalance problem at MapReduce stage that caused from using the default partition algorithm of Hadoop platform. Through using multiple partitioning technique, this proposed strategy can refine the tasks and balance the inputs of reduce stage in the map phase. Furthermore, this proposed strategy can fully employ idle nodes to balance the high load nodes, in order to achieve the optimized job scheduling during the execution process of reduce stage.

References

[1]

Yufei Gao, Yangjie Cao, Yongcai Tao, Lei Shi. The inclined data processing method based on virtual partition under MapReduce model{J}. Micro computer system, 2015, 08:1706--1710.(In Chinese)

Google Scholar

[2]

Dean J, Ghewat S. MapReduce: simplified data processing on large clusters {J}. Communications of the ACM, 2008, 51(1):107--113.

Digital Library

Google Scholar

[3]

Xiangzhen Kong, Chuang Lin, Yixin Jiang, et al. Efficient dynamic task scheduling in virtualized data centers with fuzzy prediction{J}. Journal of network and Computer Applications, 2011, 34(4):068--1077.

Digital Library

Google Scholar

[4]

Jie Fu, Zhihui Du. A load balancing strategy of periodic MapReduce operation{J}. Computer science, 2013, 03:38--40. (In Chinese)

Google Scholar

[5]

Cong Wan, Cuirong Wang, Cong Wang, Shuo Jia. Study on equilibrium partitioning algorithm for load reduce stage of the MapReduce model{J}. Micro computer system, 2015, 02:240--243. (In Chinese)

Google Scholar

[6]

Zhuo Wang, Chen Qun, Zhanhuai Li, Wei Pan, Li You. MapReduce data balancing method based on incremental partitioning strategy {J}. Journal of Computer Science, 2016, 01:19--35.(In Chinese)

Google Scholar

[7]

Morton K, Balazinska M, Grossman D. ParaTimer: a progress indicator for MapReduceDAGs {C}. Proceedings of the 2010 ACM SIGMOD International Conference on Management of data ACM, 2010: 507--518.

Digital Library

Google Scholar

[8]

Hangchen Li, Xiaolin Qin, Yao Shen. MapReduce load balancing strategy based on pressure feedback{ J}. Computer science, 2015, 04:146.(In Chinese)

Google Scholar

[9]

Khayyat Z, Awara K, Alonazi A, et al. Mizan: A system for dynamic load balancing in large-scale graph {C}. ACM European Conference on Computer Systems. 2013:169--182

Digital Library

Google Scholar

[10]

Cheng Xu. Integrated optimization of Hadoop cloud platform {D}. Sichuan Normal University, 2014.(In Chinese)

Google Scholar

[11]

Langjun Tian, Weidong Chen, Tao Li. Research on dynamic load balancing algorithm in cloud storage system {J}. Computer Engineering,2013,10:19--23.(In Chinese)

Google Scholar

[12]

Yi Chen. The research of load balancing in MapReduce based on data locality {D}. Dalian Maritime University, 2014. (In Chinese)

Google Scholar

[13]

Hanmei Liu, Hongying Han. Research on MapReduce load balancing partitioning algorithm based on feedback scheduling {J}. Information communication, 2015, 10:42. (In Chinese)

Google Scholar

Cited By

View all

Tiwari SBhatt C(2023)A Comprehensive Study on Cloud Computing: Architecture, Load Balancing, Task Scheduling and Meta-Heuristic OptimizationIntelligent Cyber Physical Systems and Internet of Things10.1007/978-3-031-18497-0_11(137-162)Online publication date: 4-Feb-2023
https://doi.org/10.1007/978-3-031-18497-0_11
Wu H(2020)Data Association Rules Mining Method Based on Improved Apriori AlgorithmProceedings of the 4th International Conference on Big Data Research10.1145/3445945.3445948(12-17)Online publication date: 27-Nov-2020
https://dl.acm.org/doi/10.1145/3445945.3445948
Sharma ASingh G(2019)A Review of Scheduling Algorithms in HadoopProceedings of ICRIC 201910.1007/978-3-030-29407-6_11(125-135)Online publication date: 22-Nov-2019
https://doi.org/10.1007/978-3-030-29407-6_11
Show More Cited By

Index Terms

The research of mapreduce load balancing based on multiple partition algorithm
1. Computing methodologies
  1. Distributed computing methodologies
    1. Distributed algorithms
      1. MapReduce algorithms
2. Theory of computation
  1. Design and analysis of algorithms
    1. Distributed algorithms
      1. MapReduce algorithms

Recommendations

Balancing reducer skew in MapReduce workloads using progressive sampling
SoCC '12: Proceedings of the Third ACM Symposium on Cloud Computing

The elapsed time of a parallel job depends on the completion time of its longest running constituent. We present a static load balancing algorithm that distributes work evenly across the reducers in a MapReduce job resulting in significant elapsed time ...
Polymorphic Load Balancing Algorithm Based on Packet Classification
ICTCE '18: Proceedings of the 2nd International Conference on Telecommunications and Communication Engineering

Through the analysis of the existing load balancing algorithm, aiming at the local hot spots appearing in network traffic may cause "starvation" or overload in the processing core, this paper proposes a multi-state load balancing algorithm based on ...
On Datacenter-Network-Aware Load Balancing in MapReduce
CLOUD '15: Proceedings of the 2015 IEEE 8th International Conference on Cloud Computing

MapReduce has emerged as a powerful tool for distributed and scalable processing of voluminous data. For skewed data input, load balancing is necessary among the MapReduce worker nodes to minimize the overall finishing time, which however can incur ...

Comments

Information & Contributors

Information

Published In

UCC '16: Proceedings of the 9th International Conference on Utility and Cloud Computing

December 2016

549 pages

ISBN:9781450346160

DOI:10.1145/2996890

General Chairs:
Changjun Jiang
Tongji University, China
,
Omer Rana
Cardiff University, UK
,
Nick Antonopoulos
University of Derby, UK

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 December 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

UCC '16

UCC '16: 9th International Conference on Utility and Cloud Computing

December 6 - 9, 2016

Shanghai, China

Acceptance Rates

Overall Acceptance Rate 38 of 125 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
87
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Tiwari SBhatt C(2023)A Comprehensive Study on Cloud Computing: Architecture, Load Balancing, Task Scheduling and Meta-Heuristic OptimizationIntelligent Cyber Physical Systems and Internet of Things10.1007/978-3-031-18497-0_11(137-162)Online publication date: 4-Feb-2023
https://doi.org/10.1007/978-3-031-18497-0_11
Wu H(2020)Data Association Rules Mining Method Based on Improved Apriori AlgorithmProceedings of the 4th International Conference on Big Data Research10.1145/3445945.3445948(12-17)Online publication date: 27-Nov-2020
https://dl.acm.org/doi/10.1145/3445945.3445948
Sharma ASingh G(2019)A Review of Scheduling Algorithms in HadoopProceedings of ICRIC 201910.1007/978-3-030-29407-6_11(125-135)Online publication date: 22-Nov-2019
https://doi.org/10.1007/978-3-030-29407-6_11
Nabavinejad SGoudarzi MHaddad HWainwright RChbeir R(2018)Data locality and VM interference aware mitigation of data skew in hadoop leveraging modern portfolio theoryProceedings of the 33rd Annual ACM Symposium on Applied Computing10.1145/3167132.3167150(175-182)Online publication date: 9-Apr-2018
https://dl.acm.org/doi/10.1145/3167132.3167150
Xu JYang X(2017)Research on key technologies of technological service and management based on cluster load balancingCluster Computing10.1007/s10586-017-1103-120:4(3409-3415)Online publication date: 1-Dec-2017
https://dl.acm.org/doi/10.1007/s10586-017-1103-1

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Balancing reducer skew in MapReduce workloads using progressive sampling

Polymorphic Load Balancing Algorithm Based on Packet Classification

On Datacenter-Network-Aware Load Balancing in MapReduce

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations