research-article

Local Graph Edge Partitioning

Authors:

Xindong WuAuthors Info & Claims

ACM Transactions on Intelligent Systems and Technology (TIST), Volume 12, Issue 5

Article No.: 61, Pages 1 - 25

https://doi.org/10.1145/3466685

Published: 23 September 2021 Publication History

Abstract

Graph edge partitioning, which is essential for the efficiency of distributed graph computation systems, divides a graph into several balanced partitions within a given size to minimize the number of vertices to be cut. Existing graph partitioning models can be classified into two categories: offline and streaming graph partitioning models. The former requires global graph information during the partitioning, which is expensive in terms of time and memory for large-scale graphs. The latter creates partitions based solely on the received graph information. However, the streaming model may result in a lower partitioning quality compared with the offline model. Therefore, this study introduces a Local Graph Edge Partitioning model, which considers only the local information (i.e., a portion of a graph instead of the entire graph) during the partitioning. Considering only the local graph information is meaningful because acquiring complete information for large-scale graphs is expensive. Based on the Local Graph Edge Partitioning model, two local graph edge partitioning algorithms—Two-stage Local Partitioning and Adaptive Local Partitioning—are given. Experimental results obtained on 14 real-world graphs demonstrate that the proposed algorithms outperform rival algorithms in most tested cases. Furthermore, the proposed algorithms are proven to significantly improve the efficiency of the real graph computation system GraphX.

References

[1]

Ghazaleh Beigi, Jiliang Tang, and Huan Liu. 2020. Social science–guided feature engineering: A novel approach to signed link analysis. ACM Transactions on Intelligent Systems and Technology 11, 1(Jan. 2020), Article 11, 27 pages.

Digital Library

[2]

Florian Bourse, Marc Lelarge, and Milan Vojnovic. 2014. Balanced graph edge partition. In Proceedings of the International Conference on Knowledge Discovery and Data Mining (ACM SIGKDD’14). ACM, New York, NY, 1456–1465.

Digital Library

[3]

Sergey N. Dorogovtsev and Jose F. F. Mendes. 2002. Evolution of networks. Advances in Physics 51, 4 (Oct. 2002), 1079–1187.

[4]

Joseph E. Gonzalez, Yucheng Low, Haijie Gu, Danny Bickson, and Carlos Guestrin. 2012. Powergraph: Distributed graph-parallel computation on natural graphs. In Proceedings of the USENIX Conference on Operating Systems Design and Implementation (OSDI’12). 17–30.

Digital Library

[5]

Joseph E. Gonzalez, Reynold S. Xin, Ankur Dave, Daniel Crankshaw, Michael J. Franklin, and Ion Stoica. 2014. GraphX: Graph processing in a distributed dataflow framework. In Proceedings of the USENIX Conference on Operating Systems Design and Implementation (OSDI’14). 599–613.

Digital Library

[6]

Yong Guo, Sungpack Hong, Hassan Chafi, Alexandru Iosup, and Dick Epema. 2017. Modeling, analysis, and experimental comparison of streaming graph-partitioning policies. Journal of Parallel and Distributed Computing 108, 1 (Oct. 2017), 106–121.

[7]

Shengwei Ji, Chenyang Bu, Lei Li, and Xindong Wu. 2019. Local graph edge partitioning with a two-stage heuristic method. In Proceedings of the IEEE International Conference on Distributed Computing Systems (ICDCS’19). IEEE, Los Alamitos, CA, 228–237.

[8]

George Karypis and Vipin Kumar. 1998. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal on Scientific Computing 20, 1 (Aug. 1998), 359–392.

Digital Library

[9]

George Karypis and Vipin Kumar. 1998. Multilevel k-way partitioning scheme for irregular graphs. Journal of Parallel and Distributed Computing 48, 1 (Jan. 1998), 96–129.

Digital Library

[10]

George Karypis and Vipin Kumar. 1998. A parallel algorithm for multilevel graph partitioning and sparse matrix ordering. Journal of Parallel and Distributed Computing 48, 1 (Jan. 1998), 71–95.

Digital Library

[11]

Brian W. Kernighan and Shen Lin. 1970. An efficient heuristic procedure for partitioning graphs. Bell System Technical Journal 49, 2 (Feb. 1970), 291–307.

[12]

Jure Leskovec and Rok Sosič. 2016. Snap: A general-purpose network analysis and graph-mining library. ACM Transactions on Intelligent Systems and Technology 8, 1 (July 2016), 1–20.

Digital Library

[13]

Lei Li, Fang Zhang, Zan Zhang, Peipei Li, and Chenyang Bu. 2019. Multi-fuzzy-objective graph pattern matching in big graph environments with reliability, trust and social relationship. World Wide Web 23, 1 (Aug. 2019), 649–669.

[14]

Yucheng Low, Joseph E. Gonzalez, Aapo Kyrola, Danny Bickson, Carlos E. Guestrin, and Joseph Hellerstein. 2014. GraphLab: A new framework for parallel machine learning. In Proceedings of the Annual Conference on Uncertainty in Artificial Intelligence (UAI’14). 340–349.

Digital Library

[15]

Ruqian Lu, Xiaolong Jin, Songmao Zhang, Meikang Qiu, and Xindong Wu. 2019. A study on big knowledge and its engineering issues. IEEE Transactions on Knowledge and Data Engineering 31, 9(Sept. 2019), 1630–1644.

Digital Library

[16]

Feng Luo, Yunfeng Yang, Chin Fu Chen, Roger Chang, Jizhong Zhou, and Richard H. Scheuermann. 2006. Modular organization of protein interaction networks. Bioinformatics 23, 2 (Nov. 2006), 207–214.

Digital Library

[17]

Wenjian Luo, Zhenglong Yan, Chenyang Bu, and Daofu Zhang. 2020. Community detection by fuzzy relations. IEEE Transactions on Emerging Topics in Computing 8, 2 (Sept. 2020), 478–492.

[18]

Wenjian Luo, Daofu Zhang, Hao Jiang, Li Ni, and Yamin Hu. 2018. Local community detection with the dynamic membership function. IEEE Transactions on Fuzzy Systems 26, 5 (March 2018), 3136–3150.

Digital Library

[19]

Wenjian Luo, Daofu Zhang, Li Ni, and Nannan Lu. 2019. Multiscale local community detection in social networks (early access). IEEE Transactions on Knowledge and Data Engineering 58, 1(Aug. 2019), 1. https://doi.org/10.1109/TKDE.2019.2938173

[20]

Grzegorz Malewicz, Matthew H. Austern, Aart J. C. Bik, James C. Dehnert, Ilan Horn, Naty Leiser, and Grzegorz Czajkowski. 2010. Pregel: A system for large-scale graph processing. In Proceedings of the ACM International Conference on Management of Data (SIGMOD’10). ACM, New York, NY, 135–146.

Digital Library

[21]

Daniel Margo and Margo Seltzer. 2015. A scalable distributed graph partitioner. In Proceedings of the International Conference on Very Large Data Bases (VLDB’15). ACM, New York, NY, 1478–1489.

Digital Library

[22]

Christian Mayer, Ruben Mayer, Muhammad Adnan Tariq, Heiko Geppert, Larissa Laich, Lukas Rieger, and Kurt Rothermel. 2018. ADWISE: Adaptive window-based streaming edge partitioning for high-speed graph processing. In Proceedings of the IEEE International Conference on Distributed Computing Systems (ICDCS’18). IEEE, Los Alamitos, CA, 685–695.

[23]

Azade Nazi, Will Hang, Anna Goldie, Sujith Ravi, and Azalia Mirhoseini. 2019. GAP: Generalizable approximate graph partitioning framework. arXiv:1903.00614.

[24]

Li Ni, Wenjian Luo, Wenjie Zhu, and Bei Hua. 2019. Local overlapping community detection. ACM Transactions on Knowledge Discovery from Data 14, 1 (Dec. 2019), 1–25.

Digital Library

[25]

Sergio Oramas, Vito Claudio Ostuni, Tommaso Di Noia, Xavier Serra, and Eugenio Di Sciascio. 2016. Sound and music recommendation with knowledge graphs. ACM Transactions on Intelligent Systems and Technology 8, 2 (Oct. 2016), Article 21, 21 pages.

Digital Library

[26]

Fabio Petroni, Leonardo Querzoni, Khuzaima Daudjee, Shahin Kamali, and Giorgio Iacoboni. 2015. HDRF: Stream-based partitioning for power-law graphs. In Proceedings of the ACM International Conference on Information and Knowledge Management (CIKM’15). ACM, New York, NY, 243–252.

Digital Library

[27]

Ryan A. Rossi and Nesreen K. Ahmed. 2015. The network data repository with interactive graph analytics and visualization. In Proceedings of the 29th AAAI Conference on Artificial Intelligence. http://networkrepository.com.

Digital Library

[28]

Amitabha Roy, Laurent Bindschaedler, Jasmina Malicevic, and Willy Zwaenepoel. 2015. Chaos: Scale-out graph processing from secondary storage. In Proceedings of the Symposium on Operating Systems Principles (SOSP’15). ACM, New York, NY, 410–424.

Digital Library

[29]

Yingxia Shao, Bin Cui, and Lin Ma. 2015. PAGE: A partition aware engine for parallel graph computation. IEEE Transactions on Knowledge and Data Engineering 27, 2(Feb. 2015), 518–530.

[30]

Isabelle Stanton and Gabriel Kliot. 2012. Streaming graph partitioning for large distributed graphs. In Proceedings of the International Conference on Knowledge Discovery and Data Mining (SIGKDD’12). ACM, New York, NY, 1222–1230.

Digital Library

[31]

Charalampos Tsourakakis, Christos Gkantsidis, Bozidar Radunovic, and Milan Vojnovic. 2014. Fennel: Streaming graph partitioning for massive scale graphs. In Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM’14). ACM, New York, NY, 333–342.

Digital Library

[32]

Minghui Wu and Xindong Wu. 2018. On big wisdom. Knowledge and Information Systems 58, 1 (Oct. 2018), 1–8.

Digital Library

[33]

Xindong Wu, Huanhuan Chen, Jun Liu, Gongqing Wu, Ruqian Lu, and Nanning Zheng. 2014. Knowledge engineering with big data (BigKE): A 54-month, 45-million RMB, 15-institution national grand project. IEEE Access 5, 1 (July 2014), 12696–12701.

[34]

Zhonggang Wu, Zhao Lu, and Shan Yuan Ho. 2016. Community detection with topological structure and attributes in information networks. ACM Transactions on Intelligent Systems and Technology 8, 2 (Nov. 2016), Article 33, 17 pages.

Digital Library

[35]

Cong Xie, Ling Yan, Wu Jun Li, and Zhihua Zhang. 2014. Distributed power-law graph computing: Theoretical and empirical analysis. In Advances in Neural Information Processing Systems (NIPS’14). 1673–1681.

Digital Library

[36]

Xindong Wu, Huanhuan Chen, Gongqing Wu, Jun Liu, Qinghua Zheng, Xiaofeng He, Aoying Zhou, et al. 2015. Knowledge engineering with big data. IEEE Intelligent Systems 30, 5 (July 2015), 46–55.

Digital Library

[37]

Hongteng Xu, Dixin Luo, and Lawrence Carin. 2019. Scalable Gromov-Wasserstein learning for graph partitioning and matching. In Advances in Neural Information Processing Systems 32. 3046–3056.

Digital Library

[38]

Ke Yang, MingXing Zhang, Kang Chen, Xiaosong Ma, Yang Bai, and Yong Jiang. 2019. KnightKing: A fast distributed graph random walk engine. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP’19). ACM, New York, NY, 524–537.

Digital Library

[39]

Chenzi Zhang, Fan Wei, Qin Liu, Zhihao Gavin Tang, and Zhenguo Li. 2017. Graph edge partitioning via neighborhood heuristic. In Proceedings of the International Conference on Knowledge Discovery and Data Mining (SIGKDD’17). ACM, New York, NY, 605–614.

Digital Library

Cited By

Shi MPeng PZhou XLiu JXiao GLi K(2024)Connectivity-Oriented Property Graph Partitioning for Distributed Graph Pattern Query ProcessingProceedings of the ACM on Management of Data10.1145/36988042:6(1-26)Online publication date: 20-Dec-2024
https://dl.acm.org/doi/10.1145/3698804
Ni LYe RLuo WZhang Y(2024)Local Community Detection in Multiple Private NetworksACM Transactions on Knowledge Discovery from Data10.1145/364407818:5(1-21)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3644078
Ji SBu CLi LWu X(2024)LocalTGEP: A Lightweight Edge Partitioner for Time-Varying GraphIEEE Transactions on Emerging Topics in Computing10.1109/TETC.2023.323833312:2(455-466)Online publication date: Apr-2024
https://doi.org/10.1109/TETC.2023.3238333
Show More Cited By

Index Terms

Local Graph Edge Partitioning
1. Computing methodologies
  1. Distributed computing methodologies

Recommendations

Graph Edge Partitioning via Neighborhood Heuristic
KDD '17: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

We consider the edge partitioning problem that partitions the edges of an input graph into multiple balanced components, while minimizing the total number of vertices replicated (one vertex might appear in more than one partition). This problem is ...
LocalDGP: local degree-balanced graph partitioning for lightweight GNNs: LocalDGP: local degree-balanced graph partitioning...
Abstract
Graph neural networks (GNNs) have been widely employed in various fields including knowledge graphs and social networks. When dealing with large-scale graphs, traditional full-batch training methods suffer from excessive GPU memory consumption. To ...
DETER: Streaming Graph Partitioning via Combined Degree and Cluster Information
Algorithms and Architectures for Parallel Processing
Abstract
Efficient graph partitioning plays an important role in distributed graph processing systems with the rapid growth of the scale of graph data. The quality of partitioning affects the performance of systems greatly. However, most existing vertex-...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Intelligent Systems and Technology

ACM Transactions on Intelligent Systems and Technology Volume 12, Issue 5

October 2021

383 pages

ISSN:2157-6904

EISSN:2157-6912

DOI:10.1145/3484925

Editor:
Huan Liu
Arizona State University, USA

Issue’s Table of Contents

Copyright © 2021 Association for Computing Machinery.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 September 2021

Accepted: 01 May 2021

Revised: 01 February 2021

Received: 01 July 2020

Published in TIST Volume 12, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

National Key Research and Development Program of China
National Natural Science Foundation of China
Fundamental Research Funds for the Central Universities

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
366
Total Downloads

Downloads (Last 12 months)72
Downloads (Last 6 weeks)6

Reflects downloads up to 24 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Shi MPeng PZhou XLiu JXiao GLi K(2024)Connectivity-Oriented Property Graph Partitioning for Distributed Graph Pattern Query ProcessingProceedings of the ACM on Management of Data10.1145/36988042:6(1-26)Online publication date: 20-Dec-2024
https://dl.acm.org/doi/10.1145/3698804
Ni LYe RLuo WZhang Y(2024)Local Community Detection in Multiple Private NetworksACM Transactions on Knowledge Discovery from Data10.1145/364407818:5(1-21)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3644078
Ji SBu CLi LWu X(2024)LocalTGEP: A Lightweight Edge Partitioner for Time-Varying GraphIEEE Transactions on Emerging Topics in Computing10.1109/TETC.2023.323833312:2(455-466)Online publication date: Apr-2024
https://doi.org/10.1109/TETC.2023.3238333
Sun ZZeng GDing CCheng T(2024)A Streaming Graph Partitioning Method to Achieve High Cohesion and Equilibrium via Multiplayer Repeated GameIEEE Transactions on Computational Social Systems10.1109/TCSS.2022.322623011:1(803-814)Online publication date: Feb-2024
https://doi.org/10.1109/TCSS.2022.3226230
Sheng SZhang ZZhou PWu X(2024)An effective algorithm for genealogical graph partitioningApplied Intelligence10.1007/s10489-023-05265-154:2(1798-1817)Online publication date: 22-Jan-2024
https://dl.acm.org/doi/10.1007/s10489-023-05265-1
Nguyen TNguyen TNguyen TYin HNguyen TJo JNguyen Q(2023)Isomorphic Graph Embedding for Progressive Maximal Frequent Subgraph MiningACM Transactions on Intelligent Systems and Technology10.1145/363063515:1(1-26)Online publication date: 19-Dec-2023
https://dl.acm.org/doi/10.1145/3630635
Wang ZYang ZWang NDu YNie JWei ZGu YYu G(2023)Lightweight Streaming Graph Partitioning by Fully Utilizing Knowledge from Local View2023 IEEE 43rd International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS57875.2023.00079(614-625)Online publication date: Jul-2023
https://doi.org/10.1109/ICDCS57875.2023.00079
Sakouhi CKhaldi ABen Ghezala H(2023)VSCT algorithm for graph partitioning based on volume, size, cuts and timeInternational Journal of Parallel, Emergent and Distributed Systems10.1080/17445760.2023.217454038:3(181-197)Online publication date: 13-Feb-2023
https://doi.org/10.1080/17445760.2023.2174540
Hong BLiu JShen LXie QYuan JEmrouznejad AHan H(2023)Graph partitioning algorithms with biological connectivity decisions for neuron reconstruction in electron microscope volumesExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.119776222:COnline publication date: 15-Jul-2023
https://dl.acm.org/doi/10.1016/j.eswa.2023.119776
Greenman KGreen WGómez-Bombarelli R(2022)Multi-fidelity prediction of molecular optical peaks with deep learningChemical Science10.1039/D1SC05677H13:4(1152-1162)Online publication date: 2022
https://doi.org/10.1039/D1SC05677H
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Issue’s Table of Contents