OMRGx: Programmable and Transparent Out-of-Core Graph Partitioning and Processing
Authors: 
Gurneet Kaur, Rajiv GuptaAuthors Info & Claims
Gurneet Kaur, Rajiv GuptaAuthors Info & ClaimsPages 137 - 149
Abstract
Partitioning and processing of large graphs on a single machine with limited memory is a challenge. While many custom solutions for out-of-core processing have been developed, limited work has been done on out-of-core partitioning that can be far more memory intensive than processing. In this paper we present the OMRGx system whose programming interface allows the programmer to rapidly prototype existing as well as new partitioning and processing strategies with minimal programming effort and oblivious of the graph size. The OMRGx engine transparently implements these strategies in an out-of-core manner while hiding the complexities of managing limited memory, parallel computation, and parallel IO from the programmer. The execution model allows multiple partitions to be simultaneously constructed and simultaneously processed by dividing the machine memory among the partitions. In contrast, existing systems process partitions one at a time. Using OMRGx we developed the first out-of-core implementation of the popular MtMetis partitioner. OMRGx implementations of existing GridGraph and GraphChi out-of-core processing frameworks deliver performance better than their standalone optimized implementations. The runtimes of implementations produced by OMRGx decrease with the number of partitions requested and increase linearly with the graph size. Finally OMRGx default implementation performs the best of all.
References
[1]
T. Bui and C. Jones. Finding good approximate vertex and edge partitions is NP-hard. Information Processing Letters, pages 153-159, 1992.
[2]
R. Chen, J. Shi, Y. Chen, B. Zang, H. Guan, and H. Chen. Powerlyra: Differentiated graph computation and partitioning on skewed graphs. ACM Transactions on Parallel Computing, 5(3), 13, 2019.
[3]
J. R. Gilbert and E. Zmijewski. A parallel graph partitioning algorithm for a message-passing multiprocessor. International Journal of Parallel Programming, (16):498-513, 1987.
[4]
J.E. Gonzalez, Y. Low, H. Gu, D. Bickson, and C. Guestrin. Powergraph: Distributed graph-parallel computation on natural graphs. In USENIX Symposium on Operating Systems Design and Implementation, 2012.
[5]
X. Zhu, W. Chen, W. Zheng, and X. Ma. Gemini: A Computation-Centric Distributed Graph Processing System. In USENIX Symposium on Operating Systems Design and Implementation, 2016.
[6]
B. Hendrickson and R. Leland. A Multilevel Algorithm for Partitioning Graphs. In Technical Report SAND93-1301, Sandia National Labs, 1993.
[7]
B. Hendrickson and R. Leland. A multilevel algorithm for partitioning graphs. In ACM/IEEE Conference on Supercomputing, 1995.
[8]
G. Karypis, and V. Kumar. Multilevel k-way partitioning scheme for irregular graphs. In Journal of Parallel Distributed Computing, 48(1):96-129, 1998.
[9]
G. Karypis, and V. Kumar. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM J. Sci. Comput., vol. 20, no. 1, pp. 359-392, Dec. 1998.
[10]
G. Karypis and V. Kumar. Parallel multilevel k-way partitioning scheme for irregular graphs. In ACM/IEEE Conference on Supercomputing, 1996.
[11]
B.W. Kernighan and S. Lin. An efficient heuristic procedure for partitioning graphs. Bell System Technical Journal, 49: 291–307, 1970.
[12]
S-C. Koduru, R. Gupta, and I. Neamtiu. Size oblivious programming with InfiniMem. In Workshop on Languages and Compilers for Parallel Computing, pages 3–19, 2016.
[13]
G. Kaur, K. Vora, S-C. Koduru and R. Gupta. Out-of-Core MapReduce For Large Data Sets. In Proceedings of 2018 ACM SIGPLAN International Symposium on Memory Management, ACM, NY, USA, 2018.
[14]
G. Kaur and R. Gupta. GO: Out-of-Core Partitioning of Large Irregular Graphs. In 15th IEEE International Conference on Networking, Architecture and Storage, IEEE NAS 2021.
[15]
A. Kyrola, G. Blelloch, and C. Guestrin. Graphchi: Large-scale graph computation on just a pc. In USENIX Symposium on Operating Systems Design and Implementation, pages 31–46, 2012.
[16]
J. Leskovec. “Stanford large network dataset collection,” http://snap.stanford.edu/data/index.html, 2011.
[17]
D. LaSalle, and G. Karypis. Multithreaded Graph Partitioning. In IEEE International Parallel and Distributed Processing Symposium, 2013.
[18]
D. LaSalle, Md M. A. Patwary, N. Satish, N. Sundaram, G. Karypis and P. Dubey. Improving Graph Partitioning for Modern Graphs and Architectures. In Workshop on Irregular Applications: Architectures and Algorithms, 2015.
[19]
Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, and J. M. Hellerstein. Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud. In Proceedings of the VLDB Endowment 5, 8 (2012), 716-727.
[20]
G. Malewicz, M.H. Austern, A.J.C Bik, J.C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: a system for large-scale graph processing. In SIGMOD International Conf. on Management of Data, pages 135-146, 2010.
[21]
Roy, Amitabha and Mihailovic, Ivo and Zwaenepoel, Willy. X-Stream: Edge-centric graph processing using streaming partitions. In SOSP 2013 - Proceedings of the 24th ACM Symposium on Operating Systems Principles, 2013.
[22]
D. Nguyen, A. Lenharth, and K. Pingali. A Lightweight Infrastructure for Graph Analytics. In ACM Symposium on Operating Systems Principles, pages 456-471, 2013.
[23]
F. Pellegrini and J. Roman. Scotch: A software package for static mapping by dual recursive bipartitioning of process and architecture graphs. In International Conference and Exhibition on High-Performance Computing and Networking, pages 493-498, 1996.
[24]
P. Sanders and C. Schulz. Distributed evolutionary graph partitioning. CoRR, vol. abs/1110.0477, 2011.
[25]
J. Shun and G. Blelloch. Ligra: a lightweight graph processing framework for shared memory. In ACM Symposium on Principles and Practice of Parallel Programming, pages 135-146, 2013.
[26]
K. Vora. LUMOS: Dependency-Driven Disk-based Graph Processing. In USENIX Annual Technical Conference, pages 429-442, 2019.
[27]
K. Vora, G. Xu, and R. Gupta. Load the edges you need: A generic i/o optimization for disk-based graph processing. In USENIX Annual Technical Conference, pages 507–522, 2016.
[28]
C. Walshaw and M. Cross. Parallel Optimization Algorithms for Multilevel Mesh Partitioning, In Parallel Computing, 26(12):1635-60, 2000.
[29]
X. Zhu, W. Han, and W. Chen. GridGraph: Large-Scale Graph Processing on a Single Machine Using 2-Level Hierarchical Partitioning. In USENIX Annual Technical Conference, pages 375–386, 2015.
[30]
Konect: http://konect.cc/networks/
[31]
Protocol Buffers. Google’s data interchange format, https://developers.google.com/protocol-buffers
Index Terms
- OMRGx: Programmable and Transparent Out-of-Core Graph Partitioning and Processing
Recommendations
Some properties of vertex-oblique graphs
The type t G ( v ) of a vertex v V ( G ) is the ordered degree-sequence ( d 1 , , d d G ( v ) ) of the vertices adjacent with v , where d 1 d d G ( v ) . A graph G is called vertex-oblique if it contains no two vertices of the same type. In this paper ...
Partitioning of a graph into induced subgraphs not containing prescribed cliques
AbstractLet K p be a complete graph of order p ≥ 2. A K p-free k-coloring of a graph H is a partition of V ( H ) into V 1 , V 2 … , V k such that H [ V i ] does not contain K p for each i ≤ k. In 1977 Borodin and Kostochka conjectured that any graph H ...
Local Graph Edge Partitioning
Graph edge partitioning, which is essential for the efficiency of distributed graph computation systems, divides a graph into several balanced partitions within a given size to minimize the number of vertices to be cut. Existing graph partitioning models ...
Comments
Information & Contributors
Information
Published In
June 2023
175 pages
ISBN:9798400701795
DOI:10.1145/3591195
- General Chair:
- Stephen M. Blackburn,
- Program Chair:
- Erez Petrank
Copyright © 2023 Owner/Author.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 06 June 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
ISMM '23
Sponsor:
ISMM '23: 2023 ACM SIGPLAN International Symposium on Memory Management
June 18, 2023
FL, Orlando, USA
Acceptance Rates
Overall Acceptance Rate 72 of 156 submissions, 46%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 215Total Downloads
- Downloads (Last 12 months)107
- Downloads (Last 6 weeks)25
Reflects downloads up to 27 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in