An Efficient Data Distribution Strategy for Distributed Graph Processing System
Pages 360 - 373
Abstract
Big data applications like social networks, biological networks, etc. are often realized on graphs. Graph processing, if done on a single node, increases time complexity. Partitioning of graphs has been proved to be useful towards handle this well-known issue. There are several partitioning algorithms that are used to partition a graph. Each partition is assigned to a node within a cluster. However, the storage capacity of a node is limited. Therefore, an effective data distribution mechanism is required. This work aims to propose a novel strategy that would define an efficient distribution of graphs into nodes using genetic algorithms. The proposed data distribution strategy, when applied on two benchmark data set, shows improved data availability without increasing the number of replicas. It has also observed that the execution time will almost became half after applying the proposed method.
References
[1]
Akbari M, Rashidi H, and Alizadeh SH An enhanced genetic algorithm with new operators for task scheduling in heterogeneous computing systems Eng. Appl. Artif. Intell. 2017 61 35-46
[2]
Alekseev VE, Boliac R, Korobitsyn DV, and Lozin VV NP-hard graph problems and boundary classes of graphs Theoret. Comput. Sci. 2007 389 1–2 219-236
[3]
Bentley JL Multidimensional divide-and-conquer Commun. ACM 1980 23 4 214-229
[4]
Cameron K, Eschen EM, Hoàng CT, and Sritharan R The complexity of the list partition problem for graphs SIAM J. Discret. Math. 2008 21 4 900-929
[5]
Day WH and Edelsbrunner H Efficient algorithms for agglomerative hierarchical clustering methods J. Classif. 1984 1 1 7-24
[6]
Golab, L., Hadjieleftheriou, M., Karloff, H., Saha, B.: Distributed data placement to minimize communication costs via graph partitioning. In: Proceedings of the 26th International Conference on Scientific and Statistical Database Management, pp. 1–12 (2014)
[7]
Gowda KC and Krishna G Agglomerative clustering using the concept of mutual nearest neighbourhood Pattern Recogn. 1978 10 2 105-112
[8]
Kalavri V, Vlassov V, and Haridi S High-level programming abstractions for distributed graph processing IEEE Trans. Knowl. Data Eng. 2017 30 2 305-324
[9]
Leskovec, J., Mcauley, J.: Learning to discover social circles in ego networks. Adv. Neural Inf. Process. Syst. 25 (2012)
[10]
Lu W, Shen Y, Wang T, Zhang M, Jagadish HV, and Du X Fast failure recovery in vertex-centric distributed graph processing systems IEEE Trans. Knowl. Data Eng. 2018 31 4 733-746
[11]
Margo D and Seltzer M A scalable distributed graph partitioner Proc. VLDB Endow. 2015 8 12 1478-1489
[12]
Murtagh, F., Contreras, P.: Algorithms for hierarchical clustering: an overview, II. Wiley Interdisc. Rev.: Data Min. Knowl. Discov. 7(6), e1219 (2017)
[13]
Paranjape, A., Benson, A.R., Leskovec, J.: Motifs in temporal networks. In: Proceedings of the tenth ACM International Conference on Web Search and Data Mining, pp. 601–610 (2017)
[14]
Phan T, Ranganathan K, and Sion R Feitelson D, Frachtenberg E, Rudolph L, and Schwiegelshohn U Evolving toward the perfect schedule: co-scheduling job assignments and data replication in wide-area systems using a genetic algorithm Job Scheduling Strategies for Parallel Processing 2005 Heidelberg Springer 173-193
[15]
Prakash S and Vidyarthi DP Maximizing availability for task scheduling in computational grid using genetic algorithm Concurr. Comput.: Pract. Exp. 2015 27 1 193-210
[16]
Rahimian F, Payberah AH, Girdzijauskas S, Jelasity M, and Haridi S A distributed algorithm for large-scale graph partitioning ACM Trans. Auton. Adapt. Syst. (TAAS) 2015 10 2 1-24
[17]
Sajjad, H.P., Rahimian, F., Vlassov, V.: Smart partitioning of geo-distributed resources to improve cloud network performance. In: 2015 IEEE 4th International Conference on Cloud Networking (CloudNet), pp. 112–118. IEEE (2015)
[18]
Shahapure, K.R., Nicholas, C.: Cluster quality analysis using silhouette score. In: 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pp. 747–748. IEEE (2020)
[19]
Sun, J., Dong, X., Zhang, X., Wang, Y.: An availability approached task scheduling algorithm in heterogeneous fault-tolerant system. In: 2014 9th IEEE International Conference on Networking, Architecture, and Storage, pp. 275–280. IEEE (2014)
[20]
Whitley D A genetic algorithm tutorial Stat. Comput. 1994 4 2 65-85
Index Terms
- An Efficient Data Distribution Strategy for Distributed Graph Processing System
Index terms have been assigned to the content through auto-classification.
Recommendations
GraBi: Communication-Efficient and Workload-Balanced Partitioning for Bipartite Graphs
ICPP '20: Proceedings of the 49th International Conference on Parallel ProcessingMachine Learning and Data Mining (MLDM) applications, such as recommendation and topic modeling, generally represent their input data in bipartite graphs with two disjoint vertex-subsets connected only by edges between them. Despite the prevalence of ...
Flexible application-aware approximation for modern distributed graph processing frameworks
GRADES-NDA '22: Proceedings of the 5th ACM SIGMOD Joint International Workshop on Graph Data Management Experiences & Systems (GRADES) and Network Data Analytics (NDA)The interest in the ability of processing data that has an underlying graph structure has grown in the recent past. This has led to the development of many distributed graph processing systems. However, due to rapidly growing amount of data, e.g., web ...
Solving the Graph Planarization Problem Using an Improved Genetic Algorithm
An improved genetic algorithm for solving the graph planarization problem is presented. The improved genetic algorithm which is designed to embed a graph on a plane, performs crossover and mutation conditionally instead of probability. The improved ...
Comments
Information & Contributors
Information
Published In
Jul 2022
415 pages
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2022.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 15 July 2022
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 11 Jan 2025