research-article

CGraph: A Distributed Storage and Processing System for Concurrent Iterative Graph Analysis Jobs

Authors:

Bingsheng He, and

Ligang HeAuthors Info & Claims

ACM Transactions on Storage (TOS), Volume 15, Issue 2

Article No.: 10, Pages 1 - 26

https://doi.org/10.1145/3319406

Published: 20 April 2019 Publication History

Abstract

Distributed graph processing platforms usually need to handle massive Concurrent iterative Graph Processing (CGP) jobs for different purposes. However, existing distributed systems face high ratio of data access cost to computation for the CGP jobs, which incurs low throughput. We observed that there are strong spatial and temporal correlations among the data accesses issued by different CGP jobs, because these concurrently running jobs usually need to repeatedly traverse the shared graph structure for the iterative processing of each vertex. Based on this observation, this article proposes a distributed storage and processing system CGraph for the CGP jobs to efficiently handle the underlying static/evolving graph for high throughput. It uses a data-centric load-trigger-pushing model, together with several optimizations, to enable the CGP jobs to efficiently share the graph structure data in the cache/memory and their accesses by fully exploiting such correlations, where the graph structure data is decoupled from the vertex state associated with each job. It can deliver much higher throughput for the CGP jobs by effectively reducing their average ratio of data access cost to computation. Experimental results show that CGraph improves the throughput of the CGP jobs by up to 3.47× in comparison with existing solutions on distributed platforms.

References

[1]

Facebook. 2018. Retrieved from http://www.facebook.com/.

[2]

LAW. 2018. Retrieved from http://law.di.unimi.it/datasets.php.

[3]

SNAP. 2018. Retrieved from http://snap.stanford.edu/data/index.html.

[4]

WDC. 2018. Retrieved from http://webdatacommons.org/hyperlinkgraph/.

[5]

Khaled Ammar and Tamer Ozsu. 2018. Experimental analysis of distributed graph systems. Proc. VLDB Endow. 11, 10 (2018), 1151--1164.

Digital Library

[6]

Shumeet Baluja, Rohan Seth, Dharshi Sivakumar, Yushi Jing, Jay Yagnik, Shankar Kumar, Deepak Ravichandran, and Mohamed Aly. 2008. Video suggestion and discovery for YouTube: Taking random walks through the view graph. In Proceedings of the 17th International Conference on World Wide Web. 895--904.

Digital Library

[7]

Mihaela A. Bornea, Julian Dolby, Anastasios Kementsietsidis, Kavitha Srinivas, Patrick Dantressangle, Octavian Udrea, and Bishwaranjan Bhattacharjee. 2013. Building an efficient RDF store over a relational database. In Proceedings of the ACM SIGMOD International Conference on Management of Data. 121--132.

Digital Library

[8]

Nathan Bronson, Zach Amsden, George Cabrera, Prasad Chakka, Peter Dimov, Hui Ding, Jack Ferris, Anthony Giardullo, Sachin Kulkarni, and Harry Li. 2013. TAO: Facebook’s distributed data store for the social graph. In Proceedings of the USENIX Annual Technical Conference. 49--60.

Digital Library

[9]

Yingyi Bu, Vinayak Borkar, Jianfeng Jia, Michael J. Carey, and Tyson Condie. 2014. Pregelix: Big(ger) graph analytics on a dataflow engine. Proc. VLDB Endow. 8, 2 (2014), 161--172.

Digital Library

[10]

Yingyi Bu, Bill Howe, Magdalena Balazinska, and Michael D. Ernst. 2010. HaLoop: Efficient iterative data processing on large clusters. Proc. VLDB Endow. 3, 1--2 (2010), 285--296.

Digital Library

[11]

Aydin Buluc and Kamesh Madduri. 2011. Parallel breadth-first search on distributed memory systems. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1--12.

Digital Library

[12]

Hanhua Chen, Hai Jin, and Xiaolong Cui. 2017. Hybrid followee recommendation in microblogging systems. Sci. China Inform. Sci. 60, 012102 (2017), 1--14.

[13]

Rong Chen, Jiaxin Shi, Yanzhe Chen, and Haibo Chen. 2015. PowerLyra: Differentiated graph computation and partitioning on skewed graphs. In Proceedings of the 10th European Conference on Computer Systems. 1--15.

Digital Library

[14]

Rishan Chen, Mao Yang, Xuetian Weng, Byron Choi, Bingsheng He, and Xiaoming Li. 2012. Improving large graph processing on partitioned graphs in the cloud. In Proceedings of the 3rd ACM Symposium on Cloud Computing. 3:1--3:13.

Digital Library

[15]

Jiefeng Cheng, Qin Liu, Zhenguo Li, Wei Fan, John C. S. Lui, and Cheng He. 2015. VENUS: Vertex-centric streamlined graph computation on a single PC. In Proceedings of the 31st IEEE International Conference on Data Engineering. 1131--1142.

[16]

Avery Ching, Sergey Edunov, Maja Kabiljo, Dionysios Logothetis, and Sambavi Muthukrishnan. 2015. One trillion edges: Graph processing at Facebook-scale. Proc. VLDB Endow. 8, 12 (2015), 1804--1815.

Digital Library

[17]

Dong Dai, Wei Zhang, and Yong Chen. 2017. IOGP: An incremental online graph partitioning algorithm for distributed graph databases. In Proceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing. 219--230.

Digital Library

[18]

Jaliya Ekanayake, Hui Li, Bingjing Zhang, Thilina Gunarathne, Seung Hee Bae, Judy Qiu, and Geoffrey Fox. 2010. Twister: A runtime for iterative mapreduce. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing. 810--818.

Digital Library

[19]

Joseph E. Gonzalez, Yucheng Low, Haijie Gu, Danny Bickson, and Carlos Guestrin. 2012. PowerGraph: Distributed graph-parallel computation on natural graphs. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation. 17--30.

Digital Library

[20]

Joseph E. Gonzalez, Reynold S. Xin, Ankur Dave, Daniel Crankshaw, Michael J. Franklin, and Ion Stoica. 2014. GraphX: Graph processing in a distributed dataflow framework. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation. 599--613.

Digital Library

[21]

Zhenyu Guo, Dong Zhou, Haoxiang Lin, Mao Yang, Fan Long, Chaoqiang Deng, Changshu Liu, and Lidong Zhou. 2011. G2: A graph processing system for diagnosing distributed systems. In Proceedings of the USENIX Annual Technical Conference. 1--14.

Digital Library

[22]

Minyang Han and Khuzaima Daudjee. 2015. Giraph unchained: Barrierless asynchronous parallel execution in pregel-like graph processing systems. Proc. VLDB Endow. 8, 9 (2015), 950--961.

Digital Library

[23]

Bingsheng He, Mao Yang, Zhenyu Guo, Rishan Chen, Bing Su, Wei Lin, and Lidong Zhou. 2010. Comet: Batched stream processing for data intensive distributed computing. In Proceedings of the 1st ACM Symposium on Cloud Computing. 63--74.

Digital Library

[24]

Sungpack Hong, Nicole C. Rodia, and Kunle Olukotun. 2013. On fast parallel detection of strongly connected components (SCC) in small-world graphs. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis. 1--11.

Digital Library

[25]

Chuan Hu and Huiping Cao. 2016. Aspect-level influence discovery from graphs. IEEE Trans. Knowl. Data Eng. 28, 7 (2016), 1635--1649.

[26]

Xiaoen Ju, Williams Dan, Hani Jamjoom, and G. Shin Kang. 2016. Version traveler: Fast and memory-efficient version switching in graph processing systems. In Proceedings of the 2016 USENIX Annual Technical Conference. 523--536.

Digital Library

[27]

Sang-Woo Jun, Andy Wright, Sizhuo Zhang, Shuotao Xu, and Arvind. 2018. GraFBoost: Using accelerated flash storage for external graph analytics. In Proceedings of the 45th ACM/IEEE International Symposium on Computer Architecture. 411--424.

Digital Library

[28]

Pavlos Kefalas, Panagiotis Symeonidis, and Yannis Manolopoulos. 2016. A graph-based taxonomy of recommendation algorithms and systems in LBSNs. IEEE Trans. Knowl. Data Eng. 28, 3 (2016), 604--622.

Digital Library

[29]

Seongyun Ko and Wook-Shin Han. 2018. TurboGraph++: A scalable and fast graph analytics system. In Proceedings of the International Conference on Management of Data. 395--410.

Digital Library

[30]

Nicolas Kourtellis, Gianmarco De Francisci Morales, and Francesco Bonchi. 2015. Scalable online betweenness centrality in evolving graphs. IEEE Trans. Knowl. Data Eng 27, 9 (2015), 2494--2506.

Digital Library

[31]

Pradeep Kumar and H. Howie Huang. 2019. GraphOne: A data store for real-time analytics on evolving graphs. In Proceedings of the 17th USENIX Conference on File and Storage Technologies. 249--263.

Digital Library

[32]

Shalmoli Gupta, Ravi Kumar, Kefu Lu, Benjamin Moseley, and Sergei Vassilvitskii. 2017. Local search methods for k-means with outliers. Proc. VLDB Endow. 10, 7 (2017), 757--768.

Digital Library

[33]

Yucheng Low, Danny Bickson, Joseph Gonzalez, Carlos Guestrin, Aapo Kyrola, and Joseph M. Hellerstein. 2012. Distributed GraphLab: A framework for machine learning and data mining in the cloud. Proc. VLDB Endow. 5, 8 (2012), 716--727.

Digital Library

[34]

Lingxiao Ma, Zhi Yang, Han Chen, Jilong Xue, and Yafei Dai. 2017. Garaph: Efficient GPU-accelerated graph processing on a single machine with balanced replication. In Proceedings of the USENIX Annual Technical Conference. 195--207.

Digital Library

[35]

Grzegorz Malewicz, Matthew H. Austern, Aart J. C. Bik, James C. Dehnert, Ilan Horn, Naty Leiser, and Grzegorz Czajkowski. 2010. Pregel: A system for large-scale graph processing. In Proceedings of the ACM SIGMOD International Conference on Management of data. 135--146.

Digital Library

[36]

Jasmina Malicevic, Baptiste Joseph Eustache Lepers, and Willy Zwaenepoel. 2017. Everything you always wanted to know about multicore graph processing but were afraid to ask. In Proceedings of the USENIX Annual Technical Conference. 631--643.

Digital Library

[37]

Claudio Martella, Dionysios Logothetis, Andreas Loukas, and Georgos Siganos. 2017. Spinner: Scalable graph partitioning in the cloud. In Proceedings of the 33rd International Conference on Data Engineering. 1083--1094.

[38]

Ulrich Meyer. 2001. Single-source shortest-paths on arbitrary directed graphs in linear average-case time. In Proceedings of the 12th ACM-SIAM Symposium on Discrete Algorithms. 797--806.

Digital Library

[39]

Anurag Mukkara, Nathan Beckmann, Maleen Abeydeera, Xiaosong Ma, and Daniel Sanchez. 2018. Exploiting locality in graph analytics through hardware-accelerated traversal scheduling. In Proceedings of the 51st IEEE/ACM International Symposium on Microarchitecture. 1--14.

Digital Library

[40]

Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1998. The PageRank Citation Ranking: Bringing Order to the Web. Technical Report. Stanford Digital Library Technologies Project.

[41]

Amitabha Roy, Laurent Bindschaedler, Jasmina Malicevic, and Willy Zwaenepoel. 2015. Chaos: Scale-out graph processing from secondary storage. In Proceedings of the 25th Symposium on Operating Systems Principles. 410--424.

Digital Library

[42]

Feng Sheng, Qiang Cao, Haoran Cai, Jie Yao, and Changsheng Xie. 2018. GraPU: Accelerate streaming graph analysis through preprocessing buffered updates. In Proceedings of the ACM Symposium on Cloud Computing. 301--312.

Digital Library

[43]

Jiaxin Shi, Youyang Yao, Rong Chen, Haibo Chen, and Feifei Li. 2016. Fast and concurrent RDF queries with RDMA-based distributed graph exploration. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation. 317--332.

Digital Library

[44]

Junshuai Song, Xiongcai Luo, Jun Gao, Chang Zhou, Hu Wei, and Jeffrey Xu Yu. 2018. UniWalk: Unidirectional random walk based scalable SimRank computation over large graph. IEEE Trans. Knowl. Data Eng 30, 5 (2018), 992--1006.

[45]

Luis M. Vaquero, Felix Cuadrado, Dionysios Logothetis, and Claudio Martella. 2014. Adaptive partitioning for large-scale dynamic graphs. In Proceedings of the 34th International Conference on Distributed Computing Systems. 144--153.

Digital Library

[46]

Shiv Verma, Luke M. Leslie, Yosub Shin, and Indranil Gupta. 2017. An experimental comparison of partitioning strategies in distributed graph processing. Proc. VLDB Endow. 10, 5 (2017), 493--504.

Digital Library

[47]

Keval Vora, Chen Tian, Rajiv Gupta, and Ziang Hu. 2017. CoRAL: Confined recovery in distributed asynchronous graph processing. In Proceedings of the 22nd International Conference on Architectural Support for Programming Languages and Operating Systems. 223--236.

Digital Library

[48]

Kai Wang, Aftab Hussain, Zhiqiang Zuo, Guoqing Xu, and Ardalan Amiri Sani. 2017. Graspan: A single-machine disk-based graph system for interprocedural static analyses of large-scale systems code. In Proceedings of the 22nd International Conference on Architectural Support for Programming Languages and Operating Systems. 389--404.

Digital Library

[49]

Siyuan Wang, Chang Lou, Rong Chen, and Haibo Chen. 2018. Fast and concurrent RDF queries using RDMA-assisted GPU graph exploration. In Proceedings of the 2018 USENIX Annual Technical Conference. 651--664.

Digital Library

[50]

Ming Wu, Fan Yang, Jilong Xue, Wencong Xiao, Youshan Miao, Lan Wei, Haoxiang Lin, Yafei Dai, and Lidong Zhou. 2015. GraM: Scaling graph computation to the trillions. In Proceedings of the 6th ACM Symposium on Cloud Computing. 408--421.

Digital Library

[51]

Chenning Xie, Rong Chen, Haibing Guan, Binyu Zang, and Haibo Chen. 2015. SYNC or ASYNC: Time to fuse for distributed graph-parallel computation. In Proceedings of the 20th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 194--204.

Digital Library

[52]

Jilong Xue, Zhi Yang, Shian Hou, and Yafei Dai. 2017. Processing concurrent graph analytics with decoupled computation model. IEEE Trans. Comput. 66, 5 (2017), 876--890.

Digital Library

[53]

Jilong Xue, Zhi Yang, Zhi Qu, Shian Hou, and Yafei Dai. 2014. Seraph: An efficient, low-cost system for concurrent graph processing. In Proceedings of the 23rd International Symposium on High-performance Parallel and Distributed Computing. 227--238.

Digital Library

[54]

Da Yan, James Cheng, Yi Lu, and Wilfred Ng. 2014. Blogel: A block-centric framework for distributed computation on real-world graphs. Proc. VLDB Endow. 7, 14 (2014), 1981--1992.

Digital Library

[55]

Pingpeng Yuan, Wenya Zhang, Changfeng Xie, Hai Jin, Ling Liu, and Kisung Lee. 2014. Fast iterative graph computation: A path centric approach. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis. 401--412.

Digital Library

[56]

Yu Zhang, Xiaofei Liao, Hai Jin, Lin Gu, Ligang He, Bingsheng He, and Haikun Liu. 2018. CGraph: A correlations-aware approach for efficient concurrent iterative graph processing. In Proceedings of the USENIX Annual Technical Conference. 441--452.

Digital Library

[57]

Yu Zhang, Xiaofei Liao, Hai Jin, Bingsheng He, Haikun Liu, and Lin Gu. 2019. DiGraph: An efficient path-based iterative directed graph processing system on multiple GPUs. In Proceedings of the Architectural Support for Programming Languages and Operating Systems. 1--14.

Digital Library

[58]

Xiaowei Zhu, Wenguang Chen, Weimin Zheng, and Xiaosong Ma. 2016. Gemini: A computation-centric distributed graph processing system. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation. 301--316.

Digital Library

Cited By

Zhao JZhang YHe LLi QZhang XJiang XYu HLiao XJin HGu LLiu HHe BZhang JSong XWang LZhou J(2023)GraphTune: An Efficient Dependency-Aware Substrate to Alleviate Irregularity in Concurrent Graph ProcessingACM Transactions on Architecture and Code Optimization10.1145/360009120:3(1-24)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3600091
Jin HQi HZhao JJiang XHuang YGui CWang QShen XZhang YHu AChen DLiu CLiu HHe HYe XWang RYuan JYao PZhang YZheng LLiao X(2022)Software Systems Implementation and Domain-Specific Architectures towards Graph AnalyticsIntelligent Computing10.34133/2022/98067582022Online publication date: 29-Oct-2022
https://doi.org/10.34133/2022/9806758
Bok KCho JLee HChoi DLim JYoo J(2022)Cost Model Based Incremental Processing in Dynamic GraphsElectronics10.3390/electronics1104066011:4(660)Online publication date: 21-Feb-2022
https://doi.org/10.3390/electronics11040660
Show More Cited By

Index Terms

CGraph: A Distributed Storage and Processing System for Concurrent Iterative Graph Analysis Jobs
1. Computer systems organization
  1. Architectures
    1. Parallel architectures
2. Computing methodologies
  1. Parallel computing methodologies

Recommendations

DiGraph: An Efficient Path-based Iterative Directed Graph Processing System on Multiple GPUs
ASPLOS '19: Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems

Many systems are recently proposed for large-scale iterative graph analytics on a single machine with GPU accelerators. Despite of many research efforts, for iterative directed graph processing over GPUs, existing solutions suffer from slow convergence ...
Read More
Cgraph: a correlations-aware approach for efficient concurrent iterative graph processing
USENIX ATC '18: Proceedings of the 2018 USENIX Conference on Usenix Annual Technical Conference

With the fast growing of iterative graph analysis applications, the graph processing platform has to efficiently handle massive Concurrent iterative Graph Processing (CGP) jobs. Although it has been extensively studied to optimize the execution of a ...
Read More
Brief Announcement: Scheduling Parallelizable Jobs Online to Maximize Throughput
SPAA '17: Proceedings of the 29th ACM Symposium on Parallelism in Algorithms and Architectures

We consider scheduling parallelizable jobs online to maximize the throughput or profit of the schedule. A set of n jobs arrive online and each job J_i has an associated function p_i(t), the profit obtained for finishing job J_i at time t. Each job has its ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Storage

ACM Transactions on Storage Volume 15, Issue 2

Systor 2018 Special Section on ATC 2018, Special Section on OSDI 2018 and Regular Papers

May 2019

187 pages

ISSN:1553-3077

EISSN:1553-3093

DOI:10.1145/3326597

Editor:
Sam H. Noh
Ulsan National Institute of Science and Technology, Ulsan, Republic of Korea

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 April 2019

Accepted: 01 March 2019

Received: 01 September 2018

Published in TOS Volume 15, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Natural Science Foundation of China
National Key Research and Development Program of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
831
Total Downloads

Downloads (Last 12 months)64
Downloads (Last 6 weeks)6

Other Metrics

View Author Metrics

Citations

Cited By

Zhao JZhang YHe LLi QZhang XJiang XYu HLiao XJin HGu LLiu HHe BZhang JSong XWang LZhou J(2023)GraphTune: An Efficient Dependency-Aware Substrate to Alleviate Irregularity in Concurrent Graph ProcessingACM Transactions on Architecture and Code Optimization10.1145/360009120:3(1-24)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3600091
Jin HQi HZhao JJiang XHuang YGui CWang QShen XZhang YHu AChen DLiu CLiu HHe HYe XWang RYuan JYao PZhang YZheng LLiao X(2022)Software Systems Implementation and Domain-Specific Architectures towards Graph AnalyticsIntelligent Computing10.34133/2022/98067582022Online publication date: 29-Oct-2022
https://doi.org/10.34133/2022/9806758
Bok KCho JLee HChoi DLim JYoo J(2022)Cost Model Based Incremental Processing in Dynamic GraphsElectronics10.3390/electronics1104066011:4(660)Online publication date: 21-Feb-2022
https://doi.org/10.3390/electronics11040660
Liao XZhao JZhang YHe BHe LJin HGu L(2022)A Structure-Aware Storage Optimization for Out-of-Core Concurrent Graph ProcessingIEEE Transactions on Computers10.1109/TC.2021.309897671:7(1612-1625)Online publication date: 1-Jul-2022
https://doi.org/10.1109/TC.2021.3098976
Si BLiang YZhao JZhang YLiao XJin HLiu HGu L(2022)GGraph: An Efficient Structure-Aware Approach for Iterative Graph ProcessingIEEE Transactions on Big Data10.1109/TBDATA.2020.30196418:5(1182-1194)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TBDATA.2020.3019641
Zhang YPeng DLiao XJin HLiu HGu LHe B(2021)LargeGraphACM Transactions on Architecture and Code Optimization10.1145/347760318:4(1-24)Online publication date: 29-Sep-2021
https://dl.acm.org/doi/10.1145/3477603
Zhao JZhang YLiao XHe LHe BJin HLiu Hde Supinski BHall MGamblin T(2021)LCCGProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3458817.3480854(1-14)Online publication date: 14-Nov-2021
https://dl.acm.org/doi/10.1145/3458817.3480854
Zhang YLiao XJin HHe LHe BLiu HGu L(2021)DepGraph: A Dependency-Driven Accelerator for Efficient Iterative Graph Processing2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA)10.1109/HPCA51647.2021.00039(371-384)Online publication date: Feb-2021
https://doi.org/10.1109/HPCA51647.2021.00039
Liu NLi DZhang YLi X(2020)Large-scale graph processing systems: a surveyFrontiers of Information Technology & Electronic Engineering10.1631/FITEE.190012721:3(384-404)Online publication date: 1-Apr-2020
https://doi.org/10.1631/FITEE.1900127
Xia QLv JMa S(2020)The distributed user trace collection and storage system based on interface window tree modelJournal of Ambient Intelligence and Humanized Computing10.1007/s12652-020-01897-0Online publication date: 31-Mar-2020
https://doi.org/10.1007/s12652-020-01897-0
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents