Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content

Advertisement

Springer Nature Link
Log in

Large-scale graph processing systems: a survey

  • Review
  • Published:
Frontiers of Information Technology & Electronic Engineering Aims and scope Submit manuscript

Abstract

Graph is a significant data structure that describes the relationship between entries. Many application domains in the real world are heavily dependent on graph data. However, graph applications are vastly different from traditional applications. It is inefficient to use general-purpose platforms for graph applications, thus contributing to the research of specific graph processing platforms. In this survey, we systematically categorize the graph workloads and applications, and provide a detailed review of existing graph processing platforms by dividing them into general-purpose and specialized systems. We thoroughly analyze the implementation technologies including programming models, partitioning strategies, communication models, execution models, and fault tolerance strategies. Finally, we analyze recent advances and present four open problems for future research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abou-Rjeili A, Karypis G, 2006. Multilevel algorithms for partitioning power-law graphs. Proc 20th IEEE Int Parallel and Distributed Processing Symp, Article 10. https://doi.org/10.1109/IPDPS.2006.1639360

  • Ajwani D, Dementiev R, Meyer U, 2006. A computational study of external-memory BFS algorithms. Proc 17th Annual ACM-SIAM Symp on Discrete Algorithm, p.601–610.

  • Ajwani D, Meyer U, Osipov V, 2007. Improved external memory BFS implementations. Proc Meeting on Algorithm Engineering and Expermiments, p.3–12.

  • Arge L, Brodal GS, Toma L, 2000. On external-memory MST, SSSP, and multi-way planar graph separation. Proc 7th Scandinavian Workshop on Algorithm Theory, p.433–447. https://doi.org/10.1007/3-540-44985-X_37

  • Atwood J, Towsley D, 2016. Diffusion-convolutional neural networks. https://arxiv.org/abs/1511.02136

  • Avery C, 2011. Giraph: large-scale graph processing infrastructure on Hadoop. Proc Hadoop Summit, p.5–9.

  • Awerbuch B, Gallager RG, 1985. Distributed BFS algorithms. 26th Annual Symp on Foundations of Computer Science, p.250–256. https://doi.org/10.1109/SFCS.1985.20

  • Bader DA, Cong G, 2006. Fast shared-memory algorithms for computing the minimum spanning forest of sparse graphs. J Parall Distr Comput, 66(11): 1366–1378. https://doi.org/10.1016/j.jpdc.2006.06.001

    Article  MATH  Google Scholar 

  • Bader DA, Madduri K, 2006. Parallel algorithms for evaluating centrality indices in real-world networks. Int Conf on Parallel Processing, p.539–550. https://doi.org/10.1109/ICPP.2006.57

  • Bao NT, Suzumura T, 2013. Towards highly scalable pregel-based graph processing platform with x10. Proc 22nd Int Conf on World Wide Web, p.501–508. https://doi.org/10.1145/2487788.2487984

  • Batarfi O, El Shawi R, Fayoumi AG, et al., 2015. Large scale graph processing systems: survey and an experimental evaluation. Clust Comput, 18(3): 1189–1213. https://doi.org/10.1007/s10586-015-0472-6

    Article  Google Scholar 

  • Baumes J, Goldberg M, Magdon-Ismail M, 2005. Efficient identification of overlapping communities. IEEE Int Conf on Intelligence and Security Informatics, p.27–36. https://doi.org/10.1007/11427995_3

  • Becchetti L, Boldi P, Castillo C, et al., 2008. Efficient semi-streaming algorithms for local triangle counting in massive graphs. Proc 14th ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining, p.16–24. https://doi.org/10.1145/1401890.1401898

  • Belkin M, Niyogi P, 2001. Laplacian eigenmaps and spectral techniques for embedding and clustering. Proc 14th Int Conf on Neural Information Processing Systems, p.585–591.

  • Binnig C, Crotty A, Galakatos A, et al., 2016. The end of slow networks: it’s time for a redesign. Proc VLDB Endowm, 9(7): 528–539. https://doi.org/10.14778/2904483.2904485

    Article  Google Scholar 

  • Borgelt C, Berthold MR, 2002. Mining molecular fragments: finding relevant substructures of molecules. IEEE Int Conf on Data Mining, p.51–58. https://doi.org/10.1109/ICDM.2002.1183885

  • Brandes U, 2001. A faster algorithm for betweenness centrality. J Math Sociol, 25(2): 163–177. https://doi.org/10.1080/0022250X.2001.9990249

    Article  MATH  Google Scholar 

  • Bruna J, Zaremba W, Szlam A, et al., 2014. Spectral networks and locally connected networks on graphs. https://arxiv.org/abs/1312.6203

  • Bu YY, Howe B, Balazinska M, et al., 2010. HaLoop: efficient iterative data processing on large clusters. Proc VLDB Endowm, 3(1-2): 285–296. https://doi.org/10.14778/1920841.1920881

    Article  Google Scholar 

  • Bu YY, Borkar V, Jia J, et al., 2014. Pregelix: big(ger) graph analytics on a dataflow engine. Proc VLDB Endowm, 8(2): 161–172. https://doi.org/10.14778/2735471.2735477

    Article  Google Scholar 

  • Bulug A, Madduri K, 2011. Parallel breadth-first search on distributed memory systems. Proc Int Conf for High Performance Computing, Networking, Storage and Analysis, Article 65. https://doi.org/10.1145/2063384.2063471

  • Bulug A, Meyerhenke H, Safro I, et al., 2016. Recent advances in graph partitioning. In: Kliemann L, Sanders P (Eds.), Algorithm Engineering. Springer, Cham, p.117–158. https://doi.org/10.1007/978-3-319-49487-6_4

    Chapter  Google Scholar 

  • Chan TM, 2010. More algorithms for all-pairs shortest paths in weighted graphs. SIAM J Comput, 39(5): 2075–2089. https://doi.org/10.1137/08071990X

    Article  MathSciNet  MATH  Google Scholar 

  • Chang LJ, Lin XM, Zhang WJ, et al., 2015. Optimal enumeration: efficient top-k tree matching. Proc VLDB Endowm, 8(5): 533–544. https://doi.org/10.14778/2735479.2735486

    Article  Google Scholar 

  • Chen R, Weng X, He B, et al., 2010. Large graph processing in the cloud. Proc ACM SIGMOD Int Conf on Management of Data, p.1123–1126. https://doi.org/10.1145/1807167.1807297

  • Chen R, Ding X, Wang P, et al., 2014. Computation and communication efficient graph processing with distributed immutable view. Proc 23rd Int Symp on High-Performance Parallel and Distributed Computing, p.215–226. https://doi.org/10.1145/2600212.2600233

  • Chen R, Shi J, Chen Y, et al., 2015. PowerLyra: differentiated graph computation and partitioning on skewed graphs. 10th European Conf on Computer Systems, Article 1.

  • Chen YZ, Wei XD, Shi JX, et al., 2016. Fast and general distributed transactions using RDMA and HTM. Proc 11th European Conf on Computer Systems, Article 26. https://doi.org/10.1145/2901318.2901349

  • Cheung TY, 1983. Graph traversal techniques and the maximum flow problem in distributed computation. IEEE Trans Softw Eng, 9(4): 504–512. https://doi.org/10.1109/TSE.1983.234958

    Article  MATH  Google Scholar 

  • Chi Y, Dai G, Wang Y, et al., 2016. NXgraph: an efficient graph processing system on a single machine. IEEE 32nd Int Conf on Data Engineering, p.409–420. https://doi.org/10.1109/ICDE.2016.7498258

  • Da Z, Mhembere D, Burns R, et al., 2015. FlashGraph: processing billion-node graphs on an array of commodity SSDS. Proc 13th USENIX Conf on File and Storage Technologies, p.45–58.

  • Dean J, Ghemawat S, 2008. MapReduce: simplified data processing on large clusters. Commun ACM, 51(1): 107–113. https://doi.org/10.1145/1327452.1327492

    Article  Google Scholar 

  • Defferrard M, Bresson X, Vandergheynst P, 2016. Convolutional neural networks on graphs with fast localized spectral filtering. https://arxiv.org/abs/1606.09375

  • Desikan P, Pathak N, Srivastava J, et al., 2005. Incremental page rank computation on evolving graphs. Special Interest Tracks and Posters of the 14th Int Conf on World Wide Web, p.1094–1095. https://doi.org/10.1145/1062745.1062885

  • Doekemeijer N, Varbanescu AL, 2014. A Survey of Parallel Graph Processing Frameworks. Technical Report No. PDS-2014–003, Delft University of Technology, the Netherlands.

    Google Scholar 

  • Dragojević A, Narayanan D, Hodson O, et al., 2014. FaRM: fast remote memory. Proc 11th USENIX Conf on Networked Systems Design and Implementation, p.401–414.

  • Duvenaud D, Maclaurin D, Aguilera-Iparraguirre J, et al., 2015. Convolutional networks on graphs for learning molecular fingerprints. Proc 28th Int Conf on Neural Information Processing Systems, p.2224–2232.

  • Ekanayake J, Li H, Zhang B, et al., 2010. Twister: a runtime for iterative MapReduce. Proc 19th ACM Int Symp on High Performance Distributed Computing, p.810–818.

  • Farkas IJ, Abel D, Palla G, et al., 2007. Weighted network modules. New J Phys, 9(6): 180. https://doi.org/10.1088/1367-2630/9/6/180

    Article  Google Scholar 

  • Garey MR, Johnson DS, Stockmeyer L, 1974. Some simplified NP-complete problems. Proc 6th Annual ACM Symp on Theory of Computing, p.47–63. https://doi.org/10.1145/800119.803884

  • Gonzalez JE, Low Y, Gu H, et al., 2012. PowerGraph: distributed graph-parallel computation on natural graphs. Proc 10th USENIX Conf on Operating Systems Design and Implementation, p.17–30.

  • Gonzalez JE, Xin RS, Dave A, et al., 2014. GraphX: graph processing in a distributed dataflow framework. Proc 11th USENIX Conf on Operating Systems Design and Implementation, p.599–613.

  • Han WS, Lee J, Lee JH, 2013a. TurboISO: towards ultrafast and robust subgraph isomorphism search in large graph databases. Proc Int Conf on Management of Data, p.337–348.

  • Han WS, Lee S, Park K, et al., 2013b. TurboGraph: a fast parallel graph engine handling billion-scale graphs in a single PC. Proc 19th ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining, p.77–85. https://doi.org/10.1145/2487575.2487581

  • Harish P, Vineet V, Narayanan P, 2009. Large graph algorithms for massively multithreaded architectures. Technical Report No. IIIT/TR/2009/74. Centre for Visual Information Technology, University of Hyderabad, India.

    Google Scholar 

  • Hirschberg DS, Chandra AK, Sarwate DV, 1979. Computing connected components on parallel computers. Commun ACM, 22(8): 461–464. https://doi.org/10.1145/359138.359141

    Article  MathSciNet  MATH  Google Scholar 

  • Ho LY, Li TH, Wu JJ, et al., 2013. Kylin: an efficient and scalable graph data processing system. IEEE Int Conf on Big Data, p.193–198. https://doi.org/10.1109/BigData.2013.6691574

  • Holder LB, Cook DJ, Djoko S, 1994. Substructure discovery in the SUBDUE system. Proc 3rd Int Conf on Knowledge Discovery and Data Mining, p.169–180.

  • Huan J, Wang W, Prins J, 2003. Efficient mining of frequent subgraphs in the presence of isomorphism. 3rd IEEE Int Conf on Data Mining, p.549–552. https://doi.org/10.1109/ICDM.2003.1250974

  • Huan J, Wang W, Prins J, et al., 2004. SPIN: mining maximal frequent subgraphs from graph databases. 10th Int Conf on Knowledge Discovery and Data Mining, p.581–586. https://doi.org/10.1145/1014052.1014123

  • Huang J, Abadi DJ, 2016. Leopard: lightweight edge oriented partitioning and replication for dynamic graphs. Proc VLDB Endowm, 9(7): 540–551. https://doi.org/10.14778/2904483.2904486

    Article  Google Scholar 

  • Inokuchi A, Washio T, Motoda H, 2000. An Apriori-based algorithm for mining frequent substructures from graph data. European Conf on Principles of Data Mining and Knowledge Discovery, p.13–23. https://doi.org/10.1007/3-540-45372-5_2

  • Jain N, Liao G, Willke TL, 2013. GraphBuilder: scalable graph ETL framework. 1st Int Workshop on Graph Data Management Experiences and Systems, Article 4. https://doi.org/10.1145/2484425.2484429

  • Kalavri V, Liagouris J, Hoffmann M, et al., 2018. Three steps is all you need: fast, accurate, automatic scaling decisions for distributed streaming dataflows. 13th USENIX Symp on Operating Systems Design and Implementation, p.783–798.

  • Kamvar SD, Haveliwala TH, Manning CD, et al., 2003. Extrapolation methods for accelerating PageRank computations. Proc 12th Int Conf on World Wide Web, p.261–270. https://doi.org/10.1145/775152.775190

  • Kang U, Tsourakakis CE, Faloutsos C, 2009. PEGASUS: a peta-scale graph mining system implementation and observations. 9th IEEE Int Conf on Data Mining, p.229–238. https://doi.org/10.1109/ICDM.2009.14

  • Kelley S, 2009. The existence and discovery of overlapping communities in large-scale networks. PhD Thesis, Rensselaer Polytechnic Institute, Troy, NY, USA.

    Google Scholar 

  • Kipf TN, Welling M, 2016a. Semi-supervised classification with graph convolutional networks. https://arxiv.org/abs/1609.02907

  • Kipf TN, Welling M, 2016b. Variational graph auto-encoders. https://arxiv.org/abs/1611.07308

  • Kolountzakis MN, Miller GL, Peng R, et al., 2012. Efficient triangle counting in large graphs via degree-based vertex partitioning. Int Math, 8(1-2): 161–185. https://doi.org/10.1080/15427951.2012.625260

    MathSciNet  MATH  Google Scholar 

  • Kuramochi M, Karypis G, 2003. GREW: a scalable frequent subgraph discovery algorithm. 4th IEEE Int Conf on Data Mining, p.439–442. https://doi.org/10.1109/ICDM.2004.10024

  • Kuramochi M, Karypis G, 2004. An efficient algorithm for discovering frequent subgraphs. IEEE Trans Knowl Data Eng, 16(9): 1038–1051. https://doi.org/10.1109/TKDE.2004.33

    Article  Google Scholar 

  • Kutzkov K, Pagh R, 2014. Triangle counting in dynamic graph streams. Scandinavian Workshop on Algorithm Theory, p.306–318. https://doi.org/10.1007/978-3-319-08404-6_27

  • Kyrola A, Blelloch GE, Guestrin C, 2012. GraphChi: large-scale graph computation on just a PC. Proc USENIX Symp on Operating Systems Design and Implementation, p.31–46.

  • Lancichinetti A, Fortunato S, Kertész J, 2009. Detecting the overlapping and hierarchical community structure in complex networks. N J Phys, 11(3): 19–44.

    Article  Google Scholar 

  • Lang K, 2004. Finding good nearly balanced cuts in power law graphs. Yahoo Research Labs, CA, USA. http://www.optimization-online.org/db_file/2004/12/1023.pdf [Assessed on Sept. 16, 2019].

    Google Scholar 

  • Lee C, Reid F, Mcdaid A, et al., 2010. Detecting highly overlapping community structure by greedy clique expansion. 4th SNA-KDD Workshop on Social Network Mining and Analysis, p.1–10.

  • Leiserson CE, Schardl TB, 2010. A work-efficient parallel breadth-first search algorithm (or how to cope with the nondeterminism of reducers). Proc 22nd Annual ACM Symp on Parallelism in Algorithms and Architectures, p.303–314. https://doi.org/10.1145/1810479.1810534

  • Liu H, Huang HH, 2017. Graphene: fine-grained IO management for graph computing. Proc 15th USENIX Conf on File and Storage Technologies, p.285–300.

  • Lotker Z, Patt-Shamir B, Peleg D, 2006. Distributed MST for constant diameter graphs. Distr Comput, 18(6): 453–460. https://doi.org/10.1007/s00446-005-0127-6

    Article  MATH  Google Scholar 

  • Low Y, Gonzalez JE, Kyrola A, et al., 2010. GraphLab: a new framework for parallel machine learning. https://arxiv.org/abs/1408.2041

  • Low Y, Bickson D, Gonzalez J, et al., 2012. Distributed GraphLab: a framework for machine learning and data mining in the cloud. Proc VLDB Endowm, 5(8): 716–727. https://doi.org/10.14778/2212351.2212354

    Article  Google Scholar 

  • Ma H, Yang H, Lyu MR, et al., 2008. Mining social networks using heat diffusion processes for marketing candidates selection. Proc 17th ACM Conf on Information and Knowledge Management, p.233–242. https://doi.org/10.1145/1458082.1458115

  • Maass S, Min C, Kashyap S, et al., 2017. Mosaic: processing a trillion-edge graph on a single machine. Proc 20th European Conf on Computer Systems, p.527–543. https://doi.org/10.1145/3064176.3064191

  • Maheshwari A, Zeh N, 2001. I/O-efficient algorithms for graphs of bounded treewidth. Proc 12th Annual ACM-SIAM Symp on Discrete Algorithms, p.89–90.

  • Malewicz G, Austern MH, Bik AJ, et al., 2010. Pregel: a system for large-scale graph processing. Proc ACM SIGMOD Int Conf on Management of Data, p.135–146.

  • Matsumoto K, Nakasato N, Sedukhin SG, 2011. Blocked all-pairs shortest paths algorithm for hybrid CPU-GPU system. IEEE 13th Int Conf on High Performance Computing and Communications, p.145–152. https://doi.org/10.1109/HPCC.2011.28

  • McCune RR, Weninger T, Madey G, 2015. Thinking like a vertex: a survey of vertex-centric frameworks for large-scale distributed graph processing. ACM Comput Surv, 48(2): 25. https://doi.org/10.1145/2818185

    Article  Google Scholar 

  • Miao X, 2015. DynaDiffuse: a dynamic diffusion model for continuous time constrained influence maximization. Proc 29th AAAI Conf on Artificial Intelligence, p.346–352.

  • Mihalcea R, 2004. Graph-based ranking algorithms for sentence extraction, applied to text summarization. Proc ACL on Interactive Poster and Demonstration Sessions, Article 20. https://doi.org/10.3115/1219044.1219064

  • Murray DG, McSherry F, Isaacs R, et al., 2013. Naiad: a timely dataflow system. Proc 24th ACM Symp on Operating Systems Principles, p.439–455.

  • Nanongkai D, 2014. Distributed approximation algorithms for weighted shortest paths. Proc 46th Annual ACM Symp on Theory of Computing, p.565–573. https://doi.org/10.1145/2591796.2591850

  • Nguyen D, Lenharth A, Pingali K, 2013. A lightweight infrastructure for graph analytics. Proc 24th ACM Symp on Operating Systems Principles, p.456–471. https://doi.org/10.1145/2517349.2522739

  • Niepert M, Ahmed M, Kutzkov K, 2016. Learning convolutional neural networks for graphs. https://arxiv.org/abs/1605.05273

  • Nuutila E, Soisalon-Soininen E, 1994. On finding the strongly connected components in a directed graph. Inform Process Lett, 49(1): 9–14. https://doi.org/10.1016/0020-0190(94)90047-7

    Article  MathSciNet  MATH  Google Scholar 

  • Pan SR, Hu RQ, Long GD, et al., 2018 Adversarially regularized graph autoencoder for graph embedding. https://arxiv.org/abs/1802.04407

  • Power R, Li JY, 2010. Piccolo: building fast, distributed programs with partitioned tables. Proc 9th USENIX Conf on Operating Systems Design and Implementation, p.293–306.

  • Psorakis I, Roberts S, Ebden M, et al., 2011. Overlapping community detection using Bayesian non-negative matrix factorization. Phys Rev E, 83(2): 066114. https://doi.org/10.1103/PhysRevE.83.066114

    Article  Google Scholar 

  • Rahimian F, Payberah AH, Girdzijauskas S, et al., 2014. Distributed vertex-cut partitioning. IFIP Int Conf on Distributed Applications and Interoperable Systems, p.186–200. https://doi.org/10.1007/978-3-662-43352-2_15

  • Ren XG, Wang JH, 2015. Exploiting vertex relationships in speeding up subgraph isomorphism over large graphs. Proc VLDB Endowm, 8(5): 617–628. https://doi.org/10.14778/2735479.2735493

    Article  Google Scholar 

  • Rodriguez MA, 2015. The Gremlin graph traversal machine and language (invited talk). Proc 15th Symp on Database Programming Languages, p.1–10. https://doi.org/10.1145/2815072.2815073

  • Roy A, Mihailovic I, Zwaenepoel W, 2013. X-Stream: edge-centric graph processing using streaming partitions. Proc 24th ACM Symp on Operating Systems Principles, p.472–488. https://doi.org/10.1145/2517349.2522740

  • Roy A, Bindschaedler L, Malicevic J, et al., 2015. Chaos: scale-out graph processing from secondary storage. Proc 25th Symp on Operating Systems Principles, p.410–424. https://doi.org/10.1145/2815400.2815408

  • Sabrin KM, Lin Z, Chau DHP, et al., 2013. MMap: Mining Billion-Scale Graphs on a PC with Fast, Minimalist Approach via Memory Mapping. Technical Report No. GT-CSE-2013–04, Georgia Institute of Technology, Atlanta, USA.

    Google Scholar 

  • Sakr S, Bajaber F, Barnawi A, et al., 2015. Big data processing systems: state-of-the-art and open challenges. Int Conf on Cloud Computing, p.1–8.

  • Sarma AD, Molla AR, Pandurangan G, et al., 2013. Fast distributed PageRank computation. Int Conf on Distributed Computing and Networking, p.11–26. https://doi.org/10.1007/978-3-642-35668-1_2

  • Scarselli F, Gori M, Tsoi AC, et al., 2009. The graph neural network model. IEEE Trans Neur Netw, 20(1): 61–80. https://doi.org/10.1109/TNN.2008.2005605

    Article  Google Scholar 

  • Schloegel K, Karypis G, Kumar V, 2000. Parallel multilevel algorithms for multi-constraint graph partitioning. Proc 6th Int European Conf on Parallel Processing, p.296–310. https://doi.org/10.1007/3-540-44520-X_39

  • Seo S, Yoon EJ, Kim J, et al., 2010. HAMA: an efficient matrix computation with the MapReduce framework. IEEE Second Int Conf on Cloud Computing Technology and Science, p.721–726. https://doi.org/10.1109/CloudCom.2010.17

  • Shang HC, Zhang Y, Lin XM, et al., 2008. Taming verification hardness: an efficient algorithm for testing subgraph isomorphism. Proc VLDB Endowm, 1(1): 364–375. https://doi.org/10.14778/1453856.1453899

    Article  Google Scholar 

  • Shao B, Wang HX, Li YT, 2013. Trinity: a distributed graph engine on a memory cloud. Proc ACM SIGMOD Int Conf on Management of Data, p.505–516. https://doi.org/10.1145/2463676.2467799

  • Shen HW, Cheng XQ, Cai K, et al., 2008. Detect overlapping and hierarchical community structure in networks. Phys A, 388(8):1706–1712. https://doi.org/10.1016/j.physa.2008.12.021

    Article  Google Scholar 

  • Shen YY, Chen G, Jagadish HV, et al., 2014. Fast failure recovery in distributed graph processing systems. Proc VLDB Endowm, 8(4): 437–448. https://doi.org/10.14778/2735496.2735506

    Article  Google Scholar 

  • Shi JX, Yao YY, Chen R, et al., 2016. Fast and concurrent RDF queries with RDMA-based distributed graph exploration. Proc 12th USENIX Conf on Operating Systems Design and Implementation, p.317–332.

  • Shun JL, Blelloch GE, 2013. Ligra: a lightweight graph processing framework for shared memory. ACM SIGPLAN Not, 48(8): 135–146. https://doi.org/10.1145/2442516.2442530

    Article  Google Scholar 

  • Simmhan Y, Kumbhare A, Wickramaarachchi C, et al., 2014. GoFFish: a sub-graph centric framework for large-scale graph analytics. European Conf on Parallel Processing, p.451–462. https://doi.org/10.1007/978-3-319-09873-9_38

  • Stanton I, Kliot G, 2012. Streaming graph partitioning for large distributed graphs. Proc 18th ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining, p.1222–1230. https://doi.org/10.1145/2339530.2339722

  • Sundaram N, Satish N, Patwary MMA, et al., 2015. Graph-Mat: high performance graph analytics made productive. Proc VLDB Endowm, 8(11): 1214–1225. https://doi.org/10.14778/2809974.2809983

    Article  Google Scholar 

  • Taleb Y, Stutsman R, Antoniu G, et al., 2018. Tailwind: fast and atomic RDMA-based replication. USENIX Annual Technical Conf, p.850–863.

  • Tangwongsan K, Pavan A, Tirthapura S, 2013. Parallel triangle counting in massive streaming graphs. Proc 22nd ACM Int Conf on Information and Knowledge Management, p.781–786. https://doi.org/10.1145/2505515.2505741

  • Tian YY, Balmin A, Corsten SA, et al., 2013. From “think like a vertex” to “think like a graph.” Proc VLDB Endowm, 7(3):193–204. https://doi.org/10.14778/2732232.2732238

    Article  Google Scholar 

  • Ullmann JR, 1976. An algorithm for subgraph isomorphism. JACM, 23(1): 31–42. https://doi.org/10.1145/321921.321925

    Article  MathSciNet  Google Scholar 

  • Valiant LG, 1990. A bridging model for parallel computation. Commun ACM, 33(8): 103–111. https://doi.org/10.1145/79173.79181

    Article  Google Scholar 

  • Vaswani A, Shazeer N, Parmar N, et al., 2017. Attention is all you need. https://arxiv.org/abs/1706.03762

  • Veličković P, Cucurull G, Casanova A, et al., 2017. Graph attention networks. https://arxiv.org/abs/1710.10903

  • Vora K, Xu GH, Gupta R, 2016. Load the edges you need: a generic I/O optimization for disk-based graph processing. USENIX Annual Technical Conf, p.507–522.

  • Vora K, Gupta R, Xu GQ, 2017. KickStarter: fast and accurate computations on streaming graphs via trimmed approximations. Proc 22nd Int Conf on Architectural Support for Programming Languages and Operating Systems, p.237–251. https://doi.org/10.1145/3037697.3037748

  • Wang DX, Cui P, Zhu WW, 2016. Structural deep network embedding. 22nd ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining, p.1225–1234. https://doi.org/10.1145/2939672.2939753

  • Wang K, Xu GH, Su Z, et al., 2015. GraphQ: graph query processing with abstraction refinement-scalable and programmable analytics over very large graphs on a single PC. USENIX Annual Technical Conf, p.387–401.

  • Wang K, Hussain A, Zuo ZQ, et al., 2017. Graspan: a single-machine disk-based graph system for interprocedural static analyses of large-scale systems code. ACM SIGPLAN Not, 52(4): 389–404. https://doi.org/10.1145/3093336.3037744

    Article  Google Scholar 

  • Wang K, Zuo ZQ, Thorpe J, et al., 2018. RStream: marrying relational algebra with streaming for efficient graph mining on a single machine. Proc 12th USENIX Conf on Operating Systems Design and Implementation, p.763–782.

  • Wang P, Zhang K, Chen R, et al., 2014. Replication-based fault-tolerance for large-scale graph processing. 44th Annual IEEE/IFIP Int Conf on Dependable Systems and Networks, p.562–573. https://doi.org/10.1109/DSN.2014.58

  • Washio T, Motoda H, 2003. State of the art of graph-based data mining. ACM SIGKDD Explor Newsl, 5(1): 59–68. https://doi.org/10.1145/959242.959249

    Article  Google Scholar 

  • Xie CN, Chen R, Guan HB, et al., 2015. SYNC or ASYNC: time to fuse for distributed graph-parallel computation. ACM SIGPLAN Not, 50(8): 194–204. https://doi.org/10.1145/2858788.2688508

    Article  Google Scholar 

  • Xie WL, Wang GZ, Bindel D, et al., 2013. Fast iterative graph computation with block updates. Proc VLDB Endowm, 6(14): 2014–2025. https://doi.org/10.14778/2556549.2556581

    Article  Google Scholar 

  • Yan D, Cheng J, Lu Y, et al., 2014. Blogel: a block-centric framework for distributed computation on real-world graphs. Proc VLDB Endowm, 7(14): 1981–1992. https://doi.org/10.14778/2733085.2733103

    Article  Google Scholar 

  • Yan XF, Han JW, 2002. gSpan: graph-based substructure pattern mining. Proc IEEE Int Conf on Data Mining, p.721–724. https://doi.org/10.1109/ICDM.2002.1184038

  • Yan XF, Han JW, 2003. CloseGraph: mining closed frequent graph patterns. Proc ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining, p.286–295. https://doi.org/10.1145/956750.956784

  • Yoo A, Chow E, Henderson K, et al., 2005. A scalable distributed parallel breadth-first search algorithm on BlueGene/L. Proc ACM/IEEE Conf on Supercomputing, Article 25. https://doi.org/10.1109/SC.2005.4

  • Yuan PP, Zhang WY, Xie CF, et al., 2014. Fast iterative graph computation: a path centric approach. Proc Int Conf for High Performance Computing, Networking, Storage and Analysis, p.401–412.

  • Zaharia M, Chowdhury M, Franklin MJ, et al., 2010. Spark: cluster computing with working sets. Proc 2nd USENIX Conf on Hot Topics in Cloud Computing, Article 10.

  • Zhang KY, Chen R, Chen HB, 2015. NUMA-aware graph-structured analytics. ACM SIGPLAN Not, 50(8): 183–193. https://doi.org/10.1145/2858788.2688507

    Article  Google Scholar 

  • Zhang MX, Wu YW, Chen K, et al., 2016. Exploring the hidden dimension in graph processing. Proc 12th USENIX Conf on Operating Systems Design and Implementation, p.285–300.

  • Zhang S, Wang RS, Zhang XS, 2007. Identification of overlapping community structure in complex networks using fuzzy c-means clustering. Phys A, 374(1): 483–490. https://doi.org/10.1016/j.physa.2006.07.023

    Article  Google Scholar 

  • Zhang Y, Liao XF, Jin H, et al., 2018. CGraph: a correlations-aware approach for efficient concurrent iterative graph processing. USENIX Annual Technical Conf, p.1–12.

  • Zhang YH, Chen R, Chen HB, 2017. Sub-millisecond stateful stream querying over fast-evolving linked data. Proc 26th Symp on Operating Systems Principles, p.614–630. https://doi.org/10.1145/3132747.3132777

  • Zhang YM, Li DS, Guo CX, et al., 2017a. CubicRing: exploiting network proximity for distributed in-memory key-value store. IEEE/ACM Trans Netw, 25(4): 2040–2053. https://doi.org/10.1109/TNET.2017.2669215

    Article  Google Scholar 

  • Zhang YM, Li DS, Zhang CX, et al., 2017b. GraphA: efficient partitioning and storage for distributed graph computation. IEEE Trans Serv Comput, online. https://doi.org/10.1109/TSC.2017.2778737

  • Zhang YM, Li DS, Liu L, 2019. Leveraging glocality for fast failure recovery in distributed RAM storage. ACM Trans Stor, 15(1): 3. https://doi.org/10.1145/3289604

    Google Scholar 

  • Zhao Y, Yoshigoe K, Xie M, et al., 2014. LightGraph: lighten communication in distributed graph-parallel processing. IEEE Int Congress on Big Data, p.717–724. https://doi.org/10.1109/BigData.Congress.2014.106

  • Zhou C, Gao J, Sun B, et al., 2014. MOCgraph: scalable distributed graph processing using message online computing. Proc VLDB Endowm, 8(4): 377–388. https://doi.org/10.14778/2735496.2735501

    Article  Google Scholar 

  • Zhu G, Lin X, Zhu K, et al., 2012. TreeSpan: efficiently computing similarity all-matching. Proc ACM SIG-MOD Int Conf on Management of Data, p.529–540. https://doi.org/10.1145/2213836.2213896

  • Zhu XW, Han WT, Chen WG, 2015. GridGraph: large-scale graph processing on a single machine using 2-level hierarchical partitioning. USENIX Annual Technical Conf, p.375–386.

  • Zhu XW, Chen WG, Zheng WM, et al., 2016. Gemini: a computation-centric distributed graph processing system. USENIX Symposium on Operating Systems Design and Implementation, p.301–316.

Download references

Author information

Authors and Affiliations

  1. Science and Technology on Parallel and Distributed Processing Laboratory, National University of Defense Technology, Changsha, 410000, China

    Ning Liu, Dong-sheng Li, Yi-ming Zhang & Xiong-lve Li

Authors
  1. Ning Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dong-sheng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi-ming Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiong-lve Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Dong-sheng LI guided the research. Ning LIU designed the research and drafted the manuscript. Xiong-lve LI helped organize the manuscript. Dong-sheng LI and Yi-ming ZHANG revised and edited the final version.

Corresponding author

Correspondence to Dong-sheng Li.

Additional information

Compliance with ethics guidelines

Ning LIU, Dong-sheng LI, Yi-ming ZHANG, and Xionglve LI declare that they have no conflict of interest.

Project supported by the National Key Program of China (No. 2018YFB2101100), the National Natural Science Foundation of China (Nos. 61932001 and 61872376), and the Major State Research Development Program of China (No. 2016YFB0201305)

Dr. Dong-sheng LI, corresponding author of this invited review article, received the BS degree (with honors) and PhD degree (with honors) in computer science from College of Computer Science, National University of Defense Technology (NUDT), Changsha, China, in 1999 and 2005, respectively. He was awarded the prize of National Excellent Doctoral Dissertation by the Ministry of Education of China in 2008. He is now a full professor at the National Lab for Parallel and Distributed Processing, NUDT. He is a corresponding expert of Frontiers of Information Technology & Electronic Engineering. His research interests include parallel and distributed computing, cloud computing, and large-scale data management.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, N., Li, Ds., Zhang, Ym. et al. Large-scale graph processing systems: a survey. Front Inform Technol Electron Eng 21, 384–404 (2020). https://doi.org/10.1631/FITEE.1900127

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1631/FITEE.1900127

Key words

CLC number

Search

Navigation