research-article

Constant Time Graph Neural Networks

Authors:

Hisashi KashimaAuthors Info & Claims

ACM Transactions on Knowledge Discovery from Data (TKDD), Volume 16, Issue 5

Article No.: 92, Pages 1 - 31

https://doi.org/10.1145/3502733

Published: 09 March 2022 Publication History

Abstract

The recent advancements in graph neural networks (GNNs) have led to state-of-the-art performances in various applications, including chemo-informatics, question-answering systems, and recommender systems. However, scaling up these methods to huge graphs, such as social networks and Web graphs, remains a challenge. In particular, the existing methods for accelerating GNNs either are not theoretically guaranteed in terms of the approximation error or incurred at least a linear time computation cost. In this study, we reveal the query complexity of the uniform node sampling scheme for Message Passing Neural Networks, including GraphSAGE, graph attention networks (GATs), and graph convolutional networks (GCNs). Surprisingly, our analysis reveals that the complexity of the node sampling method is completely independent of the number of the nodes, edges, and neighbors of the input and depends only on the error tolerance and confidence probability while providing a theoretical guarantee for the approximation error. To the best of our knowledge, this is the first article to provide a theoretical guarantee of approximation for GNNs within constant time. Through experiments with synthetic and real-world datasets, we investigated the speed and precision of the node sampling scheme and validated our theoretical results.

References

[1]

Albert-Laszlo Barabasi and Reka Albert. 1999. Emergence of scaling in random networks. Science 286, 5439 (1999), 509–512.

[2]

Igor I. Baskin, Vladimir A. Palyulin, and Nikolai S. Zefirov. 1997. A neural device for searching direct correlations between structures and properties of chemical compounds. Journal of Chemical Information and Computer Sciences 37, 4 (1997), 715–721.

[3]

Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2014. Spectral networks and locally connected networks on graphs. In Proceedings of the 2nd International Conference on Learning Representations.

[4]

Bernard Chazelle, Ronitt Rubinfeld, and Luca Trevisan. 2005. Approximating the minimum spanning tree weight in sublinear time. SIAM Journal on Computing 34, 6 (2005), 1370–1379.

Digital Library

[5]

Jie Chen, Tengfei Ma, and Cao Xiao. 2018. FastGCN: Fast learning with graph convolutional networks via importance sampling. In Proceedings of the 6th International Conference on Learning Representations.

[6]

Jianfei Chen, Jun Zhu, and Le Song. 2018. Stochastic training of graph convolutional networks with variance reduction. In Proceedings of the 35th International Conference on Machine Learning. 941–949.

[7]

Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. 2019. Cluster-GCN: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, New York, NY, 257–266.

Digital Library

[8]

Artur Czumaj and Christian Sohler. 2004. Estimating the weight of metric minimum spanning trees in sublinear-time. In Proceedings of the 36th Annual ACM Symposium on Theory of Computing. ACM, New York, NY, 175–183.

Digital Library

[9]

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of the 29th Advances in Neural Information Processing Systems, NeurIPS. 3837–3845.

[10]

Paul Erdős and Alfréd Rényi. 1959. On random graphs I. Publicationes Mathematicae 6 (1959), 290–297.

[11]

Wenqi Fan, Yao Ma, Qing Li, Yuan He, Yihong Eric Zhao, Jiliang Tang, and Dawei Yin. 2019. Graph neural networks for social recommendation. In Proceedings of the World Wide Web Conference. ACM, New York, NY, 417–426.

Digital Library

[12]

Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning. PMLR, 1263–1272.

Digital Library

[13]

Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. PMLR, 249–256.

[14]

Marco Gori, Gabriele Monfardini, and Franco Scarselli. 2005. A new model for learning in graph domains. In Proceedings of the International Joint Conference on Neural Networks. 729–734.

[15]

William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Proceedings of the 30st Advances in Neural Information Processing Systems, NeurIPS. 1025–1035.

[16]

Kohei Hayashi and Yuichi Yoshida. 2016. Minimizing quadratic functions in constant time. In Proceedings of the 29th Advances in Neural Information Processing Systems, NeurIPS. 2217–2225.

[17]

Kohei Hayashi and Yuichi Yoshida. 2017. Fitting low-rank tensors in constant time. In Proceedings of the Advances in Neural Information Processing Systems 30, NeurIPS. 2470–2478.

[18]

Wassily Hoeffding. 1963. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association 58, 301 (March 1963), 13–30.

[19]

Wen-bing Huang, Tong Zhang, Yu Rong, and Junzhou Huang. 2018. Adaptive sampling towards fast graph representation learning. In Proceedings of the 31st Advances in Neural Information Processing Systems, NeurIPS.

[20]

Piotr Indyk. 1999. A sublinear time approximation scheme for clustering in metric spaces. In Proceedings of the 40th Annual Symposium on Foundations of Computer Science. IEEE, 154–159.

Digital Library

[21]

Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of the 5th International Conference on Learning Representations.

[22]

Nina Mishra, Daniel Oblinger, and Leonard Pitt. 2001. Sublinear time approximate clustering. In Proceedings of the 12th Annual Symposium on Discrete Algorithms. SIAM, 439–447.

[23]

Huy N. Nguyen and Krzysztof Onak. 2008. Constant-time approximation algorithms via local improvements. In Proceedings of the 49th Annual Symposium on Foundations of Computer Science. IEEE, 327–336.

Digital Library

[24]

Namyong Park, Andrey Kan, Xin Luna Dong, Tong Zhao, and Christos Faloutsos. 2019. Estimating node importance in knowledge graphs using graph neural networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, New York, NY, 596–606.

Digital Library

[25]

Michal Parnas and Dana Ron. 2007. Approximating the minimum vertex cover in sublinear time and a connection to distributed algorithms. Theoretical Computer Science 381, 1–3 (2007), 183–196.

Digital Library

[26]

Ronitt Rubinfeld and Madhu Sudan. 1996. Robust characterizations of polynomials with applications to program testing. SIAM Journal on Computing 25, 2 (1996), 252–271.

Digital Library

[27]

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2009. The graph neural network model. IEEE Transactions on Neural Networks and Learning Systems 20, 1 (2009), 61–80.

Digital Library

[28]

Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In Proceedings of the Semantic Web - 15th International Conference, European Semantic Web Conference. Springer, 593–607.

[29]

Alessandro Sperduti and Antonina Starita. 1997. Supervised neural networks for the classification of structures. IEEE Transactions on Neural Networks and Learning Systems 8, 3 (1997), 714–735.

Digital Library

[30]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 30th Advances in Neural Information Processing Systems : Annual Conference on Neural Information Processing Systems 2017, NeurIPS. 5998–6008.

[31]

Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In Proceedings of the 6th International Conference on Learning Representations.

[32]

Hongwei Wang, Miao Zhao, Xing Xie, Wenjie Li, and Minyi Guo. 2019. Knowledge graph convolutional networks for recommender systems. In Proceedings of the World Wide Web Conference. ACM, New York, NY, 3307–3313.

Digital Library

[33]

Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019. KGAT: Knowledge graph attention network for recommendation. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, New York, NY, 950–958.

Digital Library

[34]

Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. 2018. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, New York, NY, 974–983.

Digital Library

[35]

Yuichi Yoshida, Masaki Yamamoto, and Hiro Ito. 2009. An improved constant-time approximation algorithm for maximum matchings. In Proceedings of the 41st Annual ACM Symposium on Theory of Computing. ACM, New York, NY, 225–234.

Digital Library

[36]

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. 2017. Deep sets. In Proceedings of the 30th Advances in Neural Information Processing Systems, NeurIPS. 3391–3401.

[37]

Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. 2018. An end-to-end deep learning architecture for graph classification. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. AAAI Press, 4438–4445.

[38]

Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu. 2019. Layer-dependent importance sampling for training deep and large graph convolutional networks. In Proceedings of the 32nd Advances in Neural Information Processing Systems, NeurIPS. 11247–11256.

Cited By

Chen YYe ZZhao HWang Y(2023)Feature-Based Graph Backdoor Attack in the Node Classification TaskInternational Journal of Intelligent Systems10.1155/2023/54183982023Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1155/2023/5418398
Zhao HYang XDeng C(2023)Parameter-Agnostic Deep Graph ClusteringACM Transactions on Knowledge Discovery from Data10.1145/363378318:3(1-20)Online publication date: 27-Nov-2023
https://dl.acm.org/doi/10.1145/3633783

Index Terms

Constant Time Graph Neural Networks

Recommendations

Toward the analysis of graph neural networks
ICSE-NIER '22: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results

Graph Neural Networks (GNNs) have recently emerged as an effective framework for representing and analyzing graph-structured data. GNNs have been applied to many real-world problems such as knowledge graph analysis, social networks recommendation, and ...
Large-Scale Graph Neural Networks: The Past and New Frontiers
KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Graph Neural Networks (GNNs) have gained significant attention in recent years due to their ability to model complex relationships between entities in graph-structured data such as social networks, protein structures, and knowledge graphs. However, due ...
Beyond low-pass filtering on large-scale graphs via Adaptive Filtering Graph Neural Networks
Abstract
Graph Neural Networks (GNNs) have emerged as a crucial deep learning framework for graph-structured data. However, existing GNNs suffer from the scalability limitation, which hinders their practical implementation in industrial settings. Many ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Knowledge Discovery from Data

ACM Transactions on Knowledge Discovery from Data Volume 16, Issue 5

October 2022

532 pages

ISSN:1556-4681

EISSN:1556-472X

DOI:10.1145/3514187

Editor:
Charu Aggarwal
IBM T.J. Watson Research, USA

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 March 2022

Accepted: 01 November 2021

Revised: 01 September 2021

Received: 01 March 2021

Published in TKDD Volume 16, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

JSPS KAKENHI
JST PRESTO program

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
338
Total Downloads

Downloads (Last 12 months)58
Downloads (Last 6 weeks)5

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen YYe ZZhao HWang Y(2023)Feature-Based Graph Backdoor Attack in the Node Classification TaskInternational Journal of Intelligent Systems10.1155/2023/54183982023Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1155/2023/5418398
Zhao HYang XDeng C(2023)Parameter-Agnostic Deep Graph ClusteringACM Transactions on Knowledge Discovery from Data10.1145/363378318:3(1-20)Online publication date: 27-Nov-2023
https://dl.acm.org/doi/10.1145/3633783

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents