research-article

iPregel: A Combiner-Based In-Memory Shared Memory Vertex-Centric Framework

Authors:

Ludovic A. R. Capelli,

Timothy A. K. ZakianAuthors Info & Claims

ICPP Workshops '18: Workshop Proceedings of the 47th International Conference on Parallel Processing

Article No.: 33, Pages 1 - 10

https://doi.org/10.1145/3229710.3229719

Published: 13 August 2018 Publication History

Abstract

The expressiveness of the vertex-centric programming model introduced by Pregel attracted great attention. Over the years, numerous frameworks emerged, abiding by the same programming model, while relying on widely different architectural designs. The vast majority of existing vertex-centric frameworks exploits distributed memory parallelism or out-of-core computations. To our knowledge, only one vertex-centric framework is designed upon in-memory storage and shared memory parallelism. Unfortunately, while built on a faster architecture than that of other vertex-centric frameworks, it did not prove to significantly outperform other existing solutions.

In this paper we present iPregel: another in-memory shared memory vertex-centric framework. The optimisations developed and presented in this paper particularly target three hotspots of vertex-centric calculations: selecting active vertices, routing messages to their recipient and updating recipients inbox. We compare iPregel against the state-of-the-art in-memory distributed memory framework Pregel+ on three of the most common vertex-centric applications: PageRank, Hashmin and the Single-Source Shortest Path. Experiments demonstrate that the single-node framework iPregel is faster than its distributed memory counterpart until at least 11 nodes are used. Further experiments show that iPregel completes a PageRank application with an order of magnitude less memory than popular vertex-centric frameworks.

References

[1]

I. Abdelaziz, R. Harbi, S. Salihoglu, and P. Kalnis. 2017. Combining Vertex-Centric Graph Processing with SPARQL for Large-Scale RDF Data Analytics. IEEE Transactions on Parallel and Distributed Systems 28, 12 (Dec 2017), 3374--3388.

Digital Library

[2]

Ibrahim Abdelaziz, Razen Harbi, Semih Salihoglu, Panos Kalnis, and Nikos Mamoulis. 2015. Spartex: A vertex-centric framework for RDF data analytics. Proceedings of the VLDB Endowment 8, 12 (2015), 1880--1883.

Digital Library

[3]

Ballmer Alex, Walters Benjamin, and Raicu Ioan. {n. d.}. FemtoGraph: A Pregel Based Shared-memory Graph Processing Library. ({n. d.}). Poster at SC'16.

[4]

Yingyi Bu. 2013. Pregelix: Dataflow-based Big Graph Analytics. In Proceedings of the 4th Annual Symposium on Cloud Computing (SOCC '13). ACM, New York, NY, USA, Article 54, 2 pages.

Digital Library

[5]

Avery Ching, Sergey Edunov, Maja Kabiljo, Dionysios Logothetis, and Sambavi Muthukrishnan. 2015. One trillion edges: Graph processing at facebook-scale. Proceedings of the VLDB Endowment 8, 12 (2015), 1804--1815.

Digital Library

[6]

Disa Mhembere Da Zheng, Randal Burns, Joshua Vogelstein, Carey E Priebe, and Alexander S Szalay. 2015. FlashGraph: Processing billion-node graphs on an array of commodity SSDs. In Proceedings of the 13th USENIX Conference on File and Storage Technologies. 45--58.

Digital Library

[7]

L. Dagum and R. Menon. 1998. OpenMP: an industry standard API for shared-memory programming. IEEE Computational Science and Engineering 5, 1 (Jan 1998), 46--55.

Digital Library

[8]

The Center for Discrete Mathematics and Theoretical Computer Science (DIMACS). 2006. 9th DIMACS Implementation Challenge. http://www.dis.uniromal.it/challenge9/download.shtml. (2006).

[9]

Vasiliki Kalavri, Vladimir Vlassov, and Seif Haridi. 2016. High-Level Programming Abstractions for Distributed Graph Processing. (07 2016). arXiv:1607.02646 https://arxiv.org/abs/1607.02646

[10]

Arijit Khan. 2016. Vertex-Centric Graph Processing: The Good, the Bad, and the Ugly. (12 2016). arXiv:1612.07404 https://arxiv.org/abs/1612.07404

[11]

Jérome Kunegis. 2013. KONECT: The Koblenz Network Collection. In Proceedings of the 22Nd International Conference on World Wide Web (WWW '13 Companion). ACM, New York, NY, USA, 1343--1350.

Digital Library

[12]

Aapo Kyrola, Guy E Blelloch, and Carlos Guestrin. 2012. Graphchi: Large-scale graph computation on just a pc. USENIX.

Digital Library

[13]

Jure Leskovec and Andrej Krevl. 2014. SNAP Datasets: Stanford Large Network Dataset Collection. http://snap.stanford.edu/data. (June 2014).

[14]

Yucheng Low, Joseph Gonzalez, Aapo Kyrola, Danny Bickson, Carlos Guestrin, and Joseph M Hellerstein. 2010. Graphlab: A new framework for parallel machine learning. arXiv preprint. arXiv preprint arXiv:1006.4990 1 (2010).

Digital Library

[15]

Grzegorz Malewicz, Matthew H. Austern, Aart J.C Bik, James C. Dehnert, Ilan Horn, Naty Leiser, and Grzegorz Czajkowski. 2010. Pregel: A System for Large-scale Graph Processing. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data (SIGMOD '10). ACM, New York, NY, USA, 135--146.

Digital Library

[16]

Louise Quick, Paul Wilkinson, and David Hardcastle. 2012. Using Pregel-like Large Scale Graph Processing Frameworks for Social Network Analysis. In Proceedings of the 2012 International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2012) (ASONAM '12). IEEE Computer Society, Washington, DC, USA, 457--463.

Digital Library

[17]

Julian Shun and Guy E. Blelloch. 2013. Ligra. Proceedings of the 18th ACM SIGPLAN symposium on Principles and practice of parallel programming - PPoPP '13 (2013).

[18]

Leslie G Valiant. 1990. A bridging model for parallel computation. Commun. ACM 33, 8 (1990), 103--111.

Digital Library

[19]

Da Yan, James Cheng, Yi Lu, and Wilfred Ng. 2015. Effective techniques for message reduction and load balancing in distributed graph computation. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 1307--1317.

Digital Library

[20]

Da Yan, Yuzhen Huang, Miao Liu, Hongzhi Chen, James Cheng, Huanhuan Wu, and Chengcui Zhang. 2017. GraphD: Distributed Vertex-Centric Graph Processing Beyond the Memory Limit. IEEE Transactions on Parallel and Distributed Systems (2017).

Digital Library

Cited By

Capelli LBrown NBull J(2022)NVRAM as an Enabler to New Horizons in Graph ProcessingSN Computer Science10.1007/s42979-022-01317-43:5Online publication date: 20-Jul-2022
https://doi.org/10.1007/s42979-022-01317-4
Ramezani MKandemir MSivasubramaniam A(2022)GraphGuess: Approximate Graph Processing System with Adaptive CorrectionEuro-Par 2022: Parallel Processing10.1007/978-3-031-12597-3_18(285-300)Online publication date: 22-Aug-2022
https://dl.acm.org/doi/10.1007/978-3-031-12597-3_18
Capelli LBrown NBull J(2019)iPregel: Strategies to Deal with an Extreme Form of Irregularity in Vertex-Centric Graph Processing2019 IEEE/ACM 9th Workshop on Irregular Applications: Architectures and Algorithms (IA3)10.1109/IA349570.2019.00013(45-50)Online publication date: Nov-2019
https://doi.org/10.1109/IA349570.2019.00013
Show More Cited By

Index Terms

iPregel: A Combiner-Based In-Memory Shared Memory Vertex-Centric Framework
1. Computing methodologies
  1. Parallel computing methodologies
    1. Parallel algorithms
      1. Shared memory algorithms
    2. Parallel programming languages

Recommendations

iPregel: Vertex-centric programmability vs memory efficiency and performance, why choose?
Highlights
- iPregel is up to 2300 ×  faster and 100 ×  more memory efficient than FemtoGraph.
Abstract
The vertex-centric programming model, designed to improve the programmability in graph processing application writing, has attracted great attention over the years. Multiple shared memory frameworks that have implemented the vertex-...
Pimiento: A Vertex-Centric Graph-Processing Framework on a Single Machine
Algorithms and Architectures for Parallel Processing
Abstract
Here, we describe a method for handling large graphs with data sizes exceeding memory capacity using minimal hardware resources. This method (called Pimiento) is a vertex-centric graph-processing framework on a single machine and represents a semi-...
NVRAM as an Enabler to New Horizons in Graph Processing
Abstract
From the world wide web, to genomics, to traffic analysis, graphs are central to many scientific, engineering, and societal endeavours. Therefore an important question is what hardware technologies are most appropriate to invest in and use for ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICPP Workshops '18: Workshop Proceedings of the 47th International Conference on Parallel Processing

August 2018

409 pages

ISBN:9781450365239

DOI:10.1145/3229710

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In-Cooperation

University of Oregon: University of Oregon

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 August 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICPP '18 Comp

ICPP '18 Comp: 47th International Conference on Parallel Processing Companion

August 13 - 16, 2018

OR, Eugene, USA

Acceptance Rates

Overall Acceptance Rate 91 of 313 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
108
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)1

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Capelli LBrown NBull J(2022)NVRAM as an Enabler to New Horizons in Graph ProcessingSN Computer Science10.1007/s42979-022-01317-43:5Online publication date: 20-Jul-2022
https://doi.org/10.1007/s42979-022-01317-4
Ramezani MKandemir MSivasubramaniam A(2022)GraphGuess: Approximate Graph Processing System with Adaptive CorrectionEuro-Par 2022: Parallel Processing10.1007/978-3-031-12597-3_18(285-300)Online publication date: 22-Aug-2022
https://dl.acm.org/doi/10.1007/978-3-031-12597-3_18
Capelli LBrown NBull J(2019)iPregel: Strategies to Deal with an Extreme Form of Irregularity in Vertex-Centric Graph Processing2019 IEEE/ACM 9th Workshop on Irregular Applications: Architectures and Algorithms (IA3)10.1109/IA349570.2019.00013(45-50)Online publication date: Nov-2019
https://doi.org/10.1109/IA349570.2019.00013
Capelli LHu ZZakian TBrown NBull J(2019)iPregelParallel Computing10.1016/j.parco.2019.04.00586:C(45-56)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.1016/j.parco.2019.04.005

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents