research-article

Window-based Streaming Graph Partitioning Algorithm

Authors:

Md Anwarul Kaium Patwary,

Byeong KangAuthors Info & Claims

ACSW '19: Proceedings of the Australasian Computer Science Week Multiconference

Article No.: 51, Pages 1 - 10

https://doi.org/10.1145/3290688.3290711

Published: 29 January 2019 Publication History

Abstract

In the recent years, the scale of graph datasets has increased to such a degree that a single machine is not capable of efficiently processing large graphs. Thereby, efficient graph partitioning is necessary for those large graph applications. Traditional graph partitioning generally loads the whole graph data into the memory before performing partitioning; this is not only a time consuming task but it also creates memory bottlenecks. These issues of memory limitation and enormous time complexity can be resolved using stream-based graph partitioning. A streaming graph partitioning algorithm reads vertices once and assigns that vertex to a partition accordingly. This is also called an one-pass algorithm. This paper proposes an efficient window-based streaming graph partitioning algorithm called WStream. The WStream algorithm is an edge-cut partitioning algorithm, which distributes a vertex among the partitions. Our results suggest that the WStream algorithm is able to partition large graph data efficiently while keeping the load balanced across different partitions, and communication to a minimum. Evaluation results with real workloads also prove the effectiveness of our proposed algorithm, and it achieves a significant reduction in load imbalance and edge-cut with different ranges of dataset.

References

[1]

2017. Nectar Cloud - Nectar. https://nectar.org.au/research-cloud/. (Accessed on 15/12/2017).

[2]

A. Abdolrashidi and L. Ramaswamy. 2016. Continual and Cost-Effective Partitioning of Dynamic Graphs for Optimizing Big Graph Processing Systems. In 2016 IEEE International Congress on Big Data (BigData Congress). 18--25.

[3]

Amr Ahmed, Nino Shervashidze, Shravan Narayanamurthy, Vanja Josifovski, and Alexander J. Smola. 2013. Distributed Large-scale Natural Graph Factorization. In Proceedings of the 22Nd International Conference on World Wide Web (WWW '13). ACM, New York, NY, USA, 37--48.

Digital Library

[4]

D. A. Bader and K. Madduri. 2008. SNAP, Small-world Network Analysis and Partitioning: An open-source parallel graph framework for the exploration of large-scale networks. In 2008 IEEE International Symposium on Parallel and Distributed Processing. 1--12.

[5]

Hugo Firth and Paolo Missier. {n. d.}. Workload-aware Streaming Graph Partitioning.

[6]

Joseph E. Gonzalez, Yucheng Low, Haijie Gu, Danny Bickson, and Carlos Guestrin. 2012. PowerGraph: Distributed Graph-parallel Computation on Natural Graphs. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation (OSDI'12). USENIX Association, Berkeley, CA, USA, 17--30. http://dl.acm.org/citation.cfm?id=2387880.2387883

Digital Library

[7]

B.W. Kernighan and S. Lin. 1970. An Efficient Heuristic Procedure for Partitioning Graphs. The Bell Systems Technical Journal 49, 2 (1970).

[8]

Jure Leskovec and Andrej Krevl. 2014. SNAP Datasets: Stanford Large Network Dataset Collection. http://snap.stanford.edu/data.

[9]

Yucheng Low, Danny Bickson, Joseph Gonzalez, Carlos Guestrin, Aapo Kyrola, and Joseph M. Hellerstein. 2012. Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud. Proc. VLDB Endow. 5, 8 (2012), 716--727.

Digital Library

[10]

Grzegorz Malewicz, Matthew H. Austern, Aart J.C Bik, James C. Dehnert, Ilan Horn, Naty Leiser, and Grzegorz Czajkowski. 2010. Pregel: A System for Large-scale Graph Processing. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data (SIGMOD '10). ACM, New York, NY, USA, 135--146.

Digital Library

[11]

M. E. J. Newman and Juyong Park. 2003. Why social networks are different from other types of networks. Physical review. E, Statistical, nonlinear, and soft matter physics 68 3 Pt 2 (2003), 036122.

[12]

Joel Nishimura and Johan Ugander. 2013. Restreaming Graph Partitioning: Simple Versatile Algorithms for Advanced Balancing. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '13). ACM, New York, NY, USA, 1106--1114.

Digital Library

[13]

Fabio Petroni, Leonardo Querzoni, Khuzaima Daudjee, Shahin Kamali, and Giorgio Iacoboni. 2015. HDRF: Stream-Based Partitioning for Power-Law Graphs. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management (CIKM '15). ACM, New York, NY, USA, 243--252.

Digital Library

[14]

F. Rahimian, A. H. Payberah, S. Girdzijauskas, M. Jelasity, and S. Haridi. 2013. JA-BE-JA: A Distributed Algorithm for Balanced Graph Partitioning. In 2013 IEEE 7th International Conference on Self-Adaptive and Self-Organizing Systems. 51--60.

Digital Library

[15]

Jason Riedy and David A. Bader. 2013. Massive Streaming Data Analytics: A Graph-based Approach. XRDS 19, 3 (2013), 37--43.

Digital Library

[16]

H. P. Sajjad, A. H. Payberah, F. Rahimian, V. Vlassov, and S. Haridi. 2016. Boosting Vertex-Cut Partitioning for Streaming Graphs. In 2016 IEEE International Congress on Big Data (BigData Congress). 1--8.

[17]

Isabelle Stanton. 2014. Streaming Balanced Graph Partitioning Algorithms for Random Graphs. In Proceedings of the Twenty-fifth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA '14). Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 1287--1301. http://dl.acm.org/citation.cfm?id=2634074.2634169

Digital Library

[18]

Isabelle Stanton and Gabriel Kliot. 2012. Streaming Graph Partitioning for Large Distributed Graphs. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '12). ACM, New York, NY, USA, 1222--1230.

Digital Library

[19]

Charalampos Tsourakakis. 2015. Streaming Graph Partitioning in the Planted Partition Model. In Proceedings of the 2015 ACM on Conference on Online Social Networks (COSN '15). ACM, New York, NY, USA, 27--35.

Digital Library

[20]

Charalampos Tsourakakis, Christos Gkantsidis, Bozidar Radunovic, and Milan Vojnovic. 2014. FENNEL: Streaming Graph Partitioning for Massive Scale Graphs. In Proceedings of the 7th ACM International Conference on Web Search and Data Mining (WSDM '14). ACM, New York, NY, USA, 333--342.

Digital Library

[21]

Christopher Walshaw. 2016. The Graph Partitioning Archive. http://http://chriswalshaw.co.uk/partition/.

[22]

R. Wang and K. Chiu. 2013. A stream partitioning approach to processing large scale distributed graph datasets. In 2013 IEEE International Conference on Big Data. 537--542.

[23]

Roy D. Williams. 1991. Performance of Dynamic Load Balancing Algorithms for Unstructured Mesh Calculations. Concurrency: Pract. Exper. 3, 5 (Oct. 1991), 457--481.

Digital Library

[24]

Cong Xie, Wu-Jun Li, and Zhihua Zhang. 2015. S-PowerGraph: Streaming Graph Partitioning for Natural Graphs by Vertex-Cut. CoRR abs/1511.02586 (2015). http://arxiv.org/abs/1511.02586

[25]

Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: Cluster Computing with Working Sets. In Proceedings of the 2Nd USENIX Conference on Hot Topics in Cloud Computing (HotCloud '10). USENIX Association, Berkeley, CA, USA, 10--10. http://dl.acm.org/citation.cfm?id=1863103.1863113

Digital Library

[26]

B. Zheng, H. Su, W. Hua, K. Zheng, X. Zhou, and G. Li. 2017. Efficient Clue-Based Route Search on Road Networks. IEEE Transactions on Knowledge and Data Engineering 29, 9 (Sept 2017), 1846--1859.

Digital Library

Cited By

Sun ZZeng GDing CCheng T(2024)A Streaming Graph Partitioning Method to Achieve High Cohesion and Equilibrium via Multiplayer Repeated GameIEEE Transactions on Computational Social Systems10.1109/TCSS.2022.322623011:1(803-814)Online publication date: Mar-2024
https://doi.org/10.1109/TCSS.2022.3226230
Merkel NMayer RFakir TJacobsen H(2023)Partitioner Selection with EASE to Optimize Distributed Graph Processing2023 IEEE 39th International Conference on Data Engineering (ICDE)10.1109/ICDE55515.2023.00185(2400-2414)Online publication date: May-2023
https://doi.org/10.1109/ICDE55515.2023.00185
Sakouhi CKhaldi ABen Ghezala H(2023)VSCT algorithm for graph partitioning based on volume, size, cuts and timeInternational Journal of Parallel, Emergent and Distributed Systems10.1080/17445760.2023.217454038:3(181-197)Online publication date: 13-Feb-2023
https://doi.org/10.1080/17445760.2023.2174540
Show More Cited By

Recommendations

Experimental Analysis of Streaming Algorithms for Graph Partitioning
SIGMOD '19: Proceedings of the 2019 International Conference on Management of Data

We report a systematic performance study of streaming graph partitioning algorithms. Graph partitioning plays a crucial role in overall system performance as it has a significant impact on both load balancing and inter-machine communication. The ...
Streaming graph partitioning for large distributed graphs
KDD '12: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining

Extracting knowledge by performing computations on graphs is becoming increasingly challenging as graphs grow in size. A standard approach distributes the graph over a cluster of nodes, but performing computations on a distributed graph is expensive if ...
Partitioning of a graph into induced subgraphs not containing prescribed cliques
Abstract
Let K p be a complete graph of order p ≥ 2. A K p-free k-coloring of a graph H is a partition of V ( H ) into V 1 , V 2 … , V k such that H [ V i ] does not contain K p for each i ≤ k. In 1977 Borodin and Kostochka conjectured that any graph H ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ACSW '19: Proceedings of the Australasian Computer Science Week Multiconference

January 2019

486 pages

ISBN:9781450366038

DOI:10.1145/3290688

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

CORE - Computing Research and Education
Macquarie University-Sydney

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 January 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ACSW 2019

ACSW 2019: Australasian Computer Science Week 2019

January 29 - 31, 2019

NSW, Sydney, Australia

Acceptance Rates

ACSW '19 Paper Acceptance Rate 61 of 141 submissions, 43%;

Overall Acceptance Rate 61 of 141 submissions, 43%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
289
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sun ZZeng GDing CCheng T(2024)A Streaming Graph Partitioning Method to Achieve High Cohesion and Equilibrium via Multiplayer Repeated GameIEEE Transactions on Computational Social Systems10.1109/TCSS.2022.322623011:1(803-814)Online publication date: Mar-2024
https://doi.org/10.1109/TCSS.2022.3226230
Merkel NMayer RFakir TJacobsen H(2023)Partitioner Selection with EASE to Optimize Distributed Graph Processing2023 IEEE 39th International Conference on Data Engineering (ICDE)10.1109/ICDE55515.2023.00185(2400-2414)Online publication date: May-2023
https://doi.org/10.1109/ICDE55515.2023.00185
Sakouhi CKhaldi ABen Ghezala H(2023)VSCT algorithm for graph partitioning based on volume, size, cuts and timeInternational Journal of Parallel, Emergent and Distributed Systems10.1080/17445760.2023.217454038:3(181-197)Online publication date: 13-Feb-2023
https://doi.org/10.1080/17445760.2023.2174540
Wan LYin GWang JBen-Dor GOgulenko AHuang Z(2023)PATRIC: A high performance parallel urban transport simulation framework based on traffic clusteringSimulation Modelling Practice and Theory10.1016/j.simpat.2023.102775126(102775)Online publication date: Jul-2023
https://doi.org/10.1016/j.simpat.2023.102775
Faraj MSchulz C(2022)Buffered Streaming Graph PartitioningACM Journal of Experimental Algorithmics10.1145/354691127(1-26)Online publication date: 21-Oct-2022
https://dl.acm.org/doi/10.1145/3546911
Kong DXie XZhang Z(2022)Clustering-based Partitioning for Large Web Graphs2022 IEEE 38th International Conference on Data Engineering (ICDE)10.1109/ICDE53745.2022.00049(593-606)Online publication date: May-2022
https://doi.org/10.1109/ICDE53745.2022.00049
Zhang HDing LZhang GPan YSong B(2022)An Efficient Vertex-Driven Temporal Graph Model and Subgraph Clustering MethodIEEE Access10.1109/ACCESS.2022.320836010(100627-100645)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3208360
Patwary MGarg SBattula SKang B(2021)SDP: Scalable Real-time Dynamic Graph PartitionerIEEE Transactions on Services Computing10.1109/TSC.2021.3137932(1-1)Online publication date: 2021
https://doi.org/10.1109/TSC.2021.3137932
Choi DHan JLim JHan JBok KYoo J(2021)Dynamic Graph Partitioning Scheme for Supporting Load Balancing in Distributed Graph EnvironmentsIEEE Access10.1109/ACCESS.2021.30754579(65254-65265)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3075457
Bhattacharya IGupta S(2021)A Novel Partitioning Algorithm to Process Large-Scale DataProceedings of Research and Applications in Artificial Intelligence10.1007/978-981-16-1543-6_15(163-171)Online publication date: 11-Jun-2021
https://doi.org/10.1007/978-981-16-1543-6_15
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents