research-article

Balanced label propagation for partitioning massive graphs

Authors:

Lars BackstromAuthors Info & Claims

WSDM '13: Proceedings of the sixth ACM international conference on Web search and data mining

Pages 507 - 516

https://doi.org/10.1145/2433396.2433461

Published: 04 February 2013 Publication History

Abstract

Partitioning graphs at scale is a key challenge for any application that involves distributing a graph across disks, machines, or data centers. Graph partitioning is a very well studied problem with a rich literature, but existing algorithms typically can not scale to billions of edges, or can not provide guarantees about partition sizes.

In this work we introduce an efficient algorithm, balanced label propagation, for precisely partitioning massive graphs while greedily maximizing edge locality, the number of edges that are assigned to the same shard of a partition. By combining the computational efficiency of label propagation --- where nodes are iteratively relabeled to the same 'label' as the plurality of their graph neighbors --- with the guarantees of constrained optimization --- guiding the propagation by a linear program constraining the partition sizes --- our algorithm makes it practically possible to partition graphs with billions of edges.

Our algorithm is motivated by the challenge of performing graph predictions in a distributed system. Because this requires assigning each node in a graph to a physical machine with memory limitations, it is critically necessary to ensure the resulting partition shards do not overload any single machine.

We evaluate our algorithm for its partitioning performance on the Facebook social graph, and also study its performance when partitioning Facebook's 'People You May Know' service (PYMK), the distributed system responsible for the feature extraction and ranking of the friends-of-friends of all active Facebook users. In a live deployment, we observed average query times and average network traffic levels that were 50.5% and 37.1% (respectively) when compared to the previous naive random sharding.

References

[1]

L. Backstrom, D. Huttenlocher, J. Kleinberg, X. Lan. Group formation in large social networks: membership, growth, and evolution. In KDD, 44--54, 2006.

Digital Library

[2]

L. Backstrom, E. Sun, C. Marlow. Find me if you can: improving geographical prediction with social and spatial proximity. In WWW, 61--70, 2010.

Digital Library

[3]

M.J. Barber, J.W. Clark. Detecting network communities by propagating labels under constraints. Physical Review E, 80:026129, 2009.

[4]

M. Berkelaar. The lpsolve package. http://lpsolve.sourceforge.net, 2011.

[5]

S.P. Boyd, L. Vandenberghe. Convex optimization. Cambridge University Press, 2004.

[6]

U. Feige, R. Krauthgamer. A polylogarithmic approximation of the minimum bisection. In FOCS, 105--115, 2000.

Digital Library

[7]

S. Fortunato. Community detection in graphs. Physics Reports, 486:75--174, 2010.

[8]

M.R. Garey, D.S. Johnson, L. Stockmeyer. Some simplified NP-complete graph problems. Theoretical computer science, 1(3):237--267, 1976.

[9]

S. Gregory. Finding overlapping communities in networks by label propagation New Journal of Physics, 12:103018, 2010.

[10]

G. Karypis, V. Kumar. Multilevel k-way partitioning scheme for irregular graphs. J. Parallel Distrib. Computing 48(1):96--129, 1998.

Digital Library

[11]

J. Leskovec, K. Lang, A. Dasgupta, M. Mahoney. Community structure in large networks: natural cluster sizes and the absence of large well-defined clusters. Internet Mathematics 6(1):29--123, 2009.

[12]

D. Liben-Nowell, J. Novak, R. Kumar, P. Raghavan, A. Tomkins. Geographic routing in social networks. PNAS, 102:11623, 2005.

[13]

P.J. Mucha, T. Richardson, K. Macon, M.A. Porter, and J.P. Onnela. Community structure in time-dependent, multiscale, and multiplex networks. Science, 328:876, 2010.

[14]

M.E.J. Newman, M. Girvan. Finding and evaluating community structure in networks. Physical Review E, 69:026113, 2004.

[15]

H.N. Fowler. Plato: Phaedrus, Loeb Classical Library, 1971.

[16]

U.N. Raghavan, R. Albert, S. Kumara. Near linear time algorithm to detect community structures in large-scale networks. Physical Review E, 76:036106, 2007.

[17]

I. Stanton, G. Kilot. Streaming graph partitioning for large distributed graphs. KDD, 1222--1230, 2012.

Digital Library

[18]

A. Thusoo, JS Sarma, N Jain, Z Shao, P Chakka, N Zhang, S Antony,H Liu, R Murthy. Hive -- a petabyte scale data warehouse using hadoop. In ICDE, 996--1005, 2010.

[19]

J. Ugander, B. Karrer, L. Backstorm, C. Marlow. The anatomy of the Facebook social graph. Arxiv preprint arXiv:1111.4503, 2011.

[20]

X. Zhu, Z. Ghahramani. Learning from labeled and unlabeled data with label propagation. Carnegie Mellon University CALD Tech Report Carnegie Mellon University-CALD-02--107, 2002.

Cited By

Chen YOu FLiu QWu GChen KDeng MChen MXu R(2024)Dual Clustering-Based Method for Geospatial Knowledge Graph PartitioningApplied Sciences10.3390/app14221070414:22(10704)Online publication date: 19-Nov-2024
https://doi.org/10.3390/app142210704
Kung PFan ZZhao TLiu YLai ZShi JWu YYu JShah NVenkataraman GHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Improving Embedding-Based Retrieval in Friend Recommendation with ANN Query ExpansionProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3661367(2930-2934)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3661367
Chen YMou ZLin BZhang TGao F(2024)Complete Coverage Path Planning for Data Collection with Multiple UAVs2024 IEEE Wireless Communications and Networking Conference (WCNC)10.1109/WCNC57260.2024.10570581(01-06)Online publication date: 21-Apr-2024
https://doi.org/10.1109/WCNC57260.2024.10570581
Show More Cited By

Index Terms

Balanced label propagation for partitioning massive graphs
1. Mathematics of computing
  1. Discrete mathematics
    1. Graph theory
      1. Graph algorithms

Recommendations

Restreaming graph partitioning: simple versatile algorithms for advanced balancing
KDD '13: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining

Partitioning large graphs is difficult, especially when performed in the limited models of computation afforded to modern large scale computing systems. In this work we introduce restreaming graph partitioning and develop algorithms that scale similarly ...
Sampling-based Label Propagation for Balanced Graph Partitioning
ICPE '22: Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering

In this experience paper, we present new sampling-based algorithms for balanced graph partitioning based on the Label Propagation (LP) approach. The purpose is to define new heuristics to extend the multi-objective and scalable Balanced GRAph ...
Partitioning of a graph into induced subgraphs not containing prescribed cliques
Abstract
Let K p be a complete graph of order p ≥ 2. A K p-free k-coloring of a graph H is a partition of V ( H ) into V 1 , V 2 … , V k such that H [ V i ] does not contain K p for each i ≤ k. In 1977 Borodin and Kostochka conjectured that any graph H ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WSDM '13: Proceedings of the sixth ACM international conference on Web search and data mining

February 2013

816 pages

ISBN:9781450318693

DOI:10.1145/2433396

General Chairs:
Stefano Leonardi
Sapienza University of Rome, Italy
,
Alessandro Panconesi
Sapienza University of Rome, Italy
,
Program Chairs:
Paolo Ferragina
University of Pisa, Italy
,
Aristides Gionis
Yahoo! Research, Barcelona, Spain

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 February 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

WSDM 2013

Sponsor:

WSDM 2013: Sixth ACM International Conference on Web Search and Data Mining

February 4 - 8, 2013

Rome, Italy

Acceptance Rates

Overall Acceptance Rate 498 of 2,863 submissions, 17%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

120
Total Citations
View Citations
1,301
Total Downloads

Downloads (Last 12 months)55
Downloads (Last 6 weeks)4

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen YOu FLiu QWu GChen KDeng MChen MXu R(2024)Dual Clustering-Based Method for Geospatial Knowledge Graph PartitioningApplied Sciences10.3390/app14221070414:22(10704)Online publication date: 19-Nov-2024
https://doi.org/10.3390/app142210704
Kung PFan ZZhao TLiu YLai ZShi JWu YYu JShah NVenkataraman GHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Improving Embedding-Based Retrieval in Friend Recommendation with ANN Query ExpansionProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3661367(2930-2934)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3661367
Chen YMou ZLin BZhang TGao F(2024)Complete Coverage Path Planning for Data Collection with Multiple UAVs2024 IEEE Wireless Communications and Networking Conference (WCNC)10.1109/WCNC57260.2024.10570581(01-06)Online publication date: 21-Apr-2024
https://doi.org/10.1109/WCNC57260.2024.10570581
Ye CLi YHe BLi ZSun J(2024)Large-Scale Graph Label Propagation on GPUsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2023.333632936:10(5234-5248)Online publication date: Oct-2024
https://doi.org/10.1109/TKDE.2023.3336329
Ding ZKong DZhang ZXie XXu J(2024)ClusPar: A Game-Theoretic Approach for Efficient and Scalable Streaming Edge PartitioningIEEE Transactions on Computers10.1109/TC.2024.3475568(1-14)Online publication date: 2024
https://doi.org/10.1109/TC.2024.3475568
Clarkson JTheodorakis GWebber J(2024)BIFROST: A Future Graph Database Runtime2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00448(5605-5613)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00448
Yang ZJiang ZYang WGao S(2024)Exact Vertex Migration Model of Graph Partitioning Based on Mixed 0–1 Linear Programming and Iteration AlgorithmJournal of the Operations Research Society of China10.1007/s40305-023-00534-9Online publication date: 24-Jan-2024
https://doi.org/10.1007/s40305-023-00534-9
Shang HShi XJiang B(2023)Network A/B Testing: Nonparametric Statistical Significance Test Based on Cluster-Level PermutationJournal of Data Science10.6339/23-JDS1112(523-537)Online publication date: 25-Jul-2023
https://doi.org/10.6339/23-JDS1112
Ugander JYin H(2023)Randomized graph cluster randomizationJournal of Causal Inference10.1515/jci-2022-001411:1Online publication date: 25-May-2023
https://doi.org/10.1515/jci-2022-0014
Çatalyürek ÜDevine KFaraj MGottesbüren LHeuer TMeyerhenke HSanders PSchlag SSchulz CSeemaier DWagner D(2023)More Recent Advances in (Hyper)Graph PartitioningACM Computing Surveys10.1145/357180855:12(1-38)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3571808
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten