Clenshaw Graph Neural Networks
Authors: 
Yuhe Guo and Zhewei WeiAuthors Info & Claims
Yuhe Guo and Zhewei WeiAuthors Info & ClaimsAugust 2023
Pages 614 - 625
Abstract
Graph Convolutional Networks (GCNs), which use a message-passing paradigm with stacked convolution layers, are foundational spatial methods for learning graph representations. Polynomial filters, which have an advantage on heterophilous graphs, are motivated differently from the spectral perspective of graph convolutions. Recent spatial GCN models use various residual connection techniques to alleviate the model degradation problem such as over-smoothing and gradient vanishing. However, current residual connections do not effectively harness the full potential of polynomial filters, which are commonly employed in the spectral domain of GNNs.
In this paper, we introduce ClenshawGCN, a GNN model that injects the characteristic of spectral models into spatial models by a simple residual connection submodule: the Clenshaw residual connection, which is essentially a second-order negative residual combined with an initial residual. We show that a ClenshawGCN implicitly simulates an arbitrary polynomial filter under the Chebyshev basis, since the iteration process of stacked (spatial) convolutions equipped with Clenshaw residuals can be interpreted by Clenshaw Summation Algorithm. In addition, we conduct comprehensive experiments to demonstrate the superiority of our model over spatial and spectral GNN models. Our implementation is at https://github.com/yuziGuo/KDDClenshawGNN.
Supplementary Material
References
[1]
Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. 2019. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In international conference on machine learning. PMLR, 21--29.
[2]
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 2623--2631.
[3]
Filippo Maria Bianchi, Daniele Grattarola, Lorenzo Livi, and Cesare Alippi. 2021. Graph neural networks with convolutional arma filters. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).
[4]
Ben Chamberlain, James Rowbottom, Maria I Gorinova, Michael Bronstein, Stefan Webb, and Emanuele Rossi. 2021. Grand: Graph neural diffusion. In International Conference on Machine Learning. PMLR, 1407--1418.
[5]
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. 2020. Simple and deep graph convolutional networks. In ICML. PMLR, 1725--1735.
[6]
Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. 2021. Adaptive Universal Generalized PageRank Graph Neural Network. In ICLR.
[7]
Charles W Clenshaw. 1955. A note on the summation of Chebyshev series. Math. Comp., Vol. 9, 51 (1955), 118--120.
[8]
Mark Craven, Andrew McCallum, Dan PiPasquo, Tom Mitchell, and Dayne Freitag. 1998. Learning to extract symbolic knowledge from the World Wide Web. Technical Report. Carnegie-mellon univ pittsburgh pa school of computer Science.
[9]
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. (6 2016). http://arxiv.org/abs/1606.09375
[10]
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. 2015. Convolutional networks on graphs for learning molecular fingerprints. Advances in neural information processing systems, Vol. 28 (2015).
[11]
Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. 2019. Graph neural networks for social recommendation. In The world wide web conference. 417--426.
[12]
Matthias Fey and Jan E. Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds.
[13]
David K. Hammond, Pierre Vandergheynst, and Rémi Gribonval. 2009. Wavelets on Graphs via Spectral Graph Theory. CoRR, Vol. abs/0912.3848 (2009). showeprint[arXiv]0912.3848 http://arxiv.org/abs/0912.3848
[14]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[15]
Mingguo He, Zhewei Wei, Zengfeng Huang, and Hongteng Xu. 2021. BernNet: Learning Arbitrary Graph Spectral Filters via Bernstein Approximation. Advances in Neural Information Processing Systems, Vol. 34 (6 2021), 14239--14251. http://arxiv.org/abs/2106.10994
[16]
Mingguo He, Zhewei Wei, and Ji-Rong Wen. 2022. Convolutional Neural Networks on Graphs with Chebyshev Approximation, Revisited. arXiv preprint arXiv:2202.03580 (2022).
[17]
William George Horner. 1819. XXI. A new method of solving numerical equations of all orders, by continuous approximation. Philosophical Transactions of the Royal Society of London 109 ( 1819), 308--335.
[18]
Forrest Iandola, Matt Moskewicz, Sergey Karayev, Ross Girshick, Trevor Darrell, and Kurt Keutzer. 2014. Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprint arXiv:1404.1869 (2014).
[19]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[20]
Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In ICLR.
[21]
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. 2019a. Predict then propagate: Graph neural networks meet personalized pagerank. In ICLR.
[22]
Johannes Klicpera, Stefan Weißenberger, and Stephan Günnemann. 2019b. Diffusion improves graph learning. arXiv preprint arXiv:1911.05485 (2019).
[23]
Guohao Li, Matthias Müller, Ali Thabet, and Bernard Ghanem. 2019. DeepGCNs: Can GCNs Go as Deep as CNNs? (4 2019). http://arxiv.org/abs/1904.03751
[24]
Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI conference on artificial intelligence.
[25]
Xiang Li, Renyu Zhu, Yao Cheng, Caihua Shan, Siqiang Luo, Dongsheng Li, and Weining Qian. 2022. Finding Global Homophily in Graph Neural Networks When Meeting Heterophily. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (Eds.). PMLR, 13242--13256.
[26]
Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser-Nam Lim. 2021. Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods. Advances in Neural Information Processing Systems, Vol. 34 (10 2021), 20887--20902. http://arxiv.org/abs/2110.14446
[27]
Xiaorui Liu, Jiayuan Ding, Wei Jin, Han Xu, Yao Ma, Zitao Liu, and Jiliang Tang. 2021. Graph Neural Networks with Adaptive Residual. NIPS (2021). https://github.com/lxiaorui/AirGNN.
[28]
Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-Wen Chang, and Doina Precup. 2021a. Is Heterophily A Real Nightmare For Graph Neural Networks To Do Node Classification? (9 2021). http://arxiv.org/abs/2109.05641
[29]
Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-Wen Chang, and Doina Precup. 2021b. Is Heterophily A Real Nightmare For Graph Neural Networks To Do Node Classification? arXiv preprint arXiv:2109.05641 (2021).
[30]
Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-Wen Chang, and Doina Precup. 2022. Revisiting heterophily for graph neural networks. arXiv preprint arXiv:2210.07606 (2022).
[31]
John C Mason and David C Handscomb. 2002. Chebyshev polynomials. Chapman and Hall/CRC.
[32]
Miller McPherson, Lynn Smith-Lovin, and James M Cook. 2001. Birds of a feather: Homophily in social networks. Annual review of sociology (2001), 415--444.
[33]
Sunil K Narang, Akshay Gadde, and Antonio Ortega. 2013. Signal processing techniques for interpolation in graph structured data. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 5445--5449.
[34]
Hoang Nt and Takanori Maehara. 2019. Revisiting graph neural networks: All we have is low-pass filters. arXiv preprint arXiv:1905.09550 (2019).
[35]
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation ranking: Bringing order to the web. Technical Report. Stanford InfoLab.
[36]
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. 2020. Geom-GCN: Geometric Graph Convolutional Networks. In ICLR.
[37]
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. 2020. DropEdge: Towards Deep Graph Convolutional Networks on Node Classification. In ICLR.
[38]
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. 2021. E (n) equivariant graph neural networks. In International conference on machine learning. PMLR, 9323--9332.
[39]
Michael Sejr Schlichtkrull, Nicola De Cao, and Ivan Titov. 2020. Interpreting graph neural networks for nlp with differentiable edge masking. arXiv preprint arXiv:2010.00577 (2020).
[40]
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008. Collective classification in network data. AI magazine, Vol. 29, 3 (2008), 93--93.
[41]
David I Shuman, Benjamin Ricaud, and Pierre Vandergheynst. 2013. Vertex-Frequency Analysis on Graphs. (7 2013). http://arxiv.org/abs/1307.5708
[42]
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. 2013. On the importance of initialization and momentum in deep learning. In International conference on machine learning. PMLR, 1139--1147.
[43]
Jie Tang, Jimeng Sun, Chi Wang, and Zi Yang. 2009. Social influence analysis in large-scale networks. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. 807--816.
[44]
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903 (2017).
[45]
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, et al. 2017. Graph attention networks. stat, Vol. 1050, 20 (2017), 10--48550.
[46]
Guangtao Wang, Rex Ying, Jing Huang, and Jure Leskovec. 2019. Improving graph attention networks with large margin-based constraints. arXiv preprint arXiv:1910.11945 (2019).
[47]
Xiyuan Wang and Muhan Zhang. 2022. How Powerful are Spectral Graph Neural Networks. (5 2022). http://arxiv.org/abs/2205.11172
[48]
Wikipedia. 2023 a. Clenshaw algorithm - Wikipedia, The Free Encyclopedia. http://en.wikipedia.org/w/index.php?title=Clenshaw%20algorithm&oldid=1089015914. [Online; accessed 03-February-2023].
[49]
Wikipedia. 2023 b. Horner's method - Wikipedia, The Free Encyclopedia. http://en.wikipedia.org/w/index.php?title=Horner's%20method&oldid=1135871092. [Online; accessed 03-February-2023].
[50]
Lingfei Wu, Yu Chen, Kai Shen, Xiaojie Guo, Hanning Gao, Shucheng Li, Jian Pei, and Bo Long. 2021. Graph neural networks for natural language processing: A survey. arXiv preprint arXiv:2106.06090 (2021).
[51]
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. 2020. Graph neural networks in recommender systems: a survey. ACM Computing Surveys (CSUR) (2020).
[52]
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. 2018. Representation Learning on Graphs with Jumping Knowledge Networks. In ICML.
[53]
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. 2016. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning. PMLR, 40--48.
[54]
Wentao Zhang, Zeang Sheng, Ziqi Yin, Yuezihan Jiang, Yikuan Xia, Jun Gao, Zhi Yang, and Bin Cui. 2022. Model Degradation Hinders Deep Graph Neural Networks. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2493--2503. https://doi.org/10.1145/3534678.3539374
[55]
Wentao Zhang, Ziqi Yin, Zeang Sheng, Wen Ouyang, Xiaosen Li, Yangyu Tao, Zhi Yang, and Bin Cui. 2021. Graph Attention Multi-Layer Perceptron. CoRR, Vol. abs/2108.10097 (2021). showeprint[arXiv]2108.10097 https://arxiv.org/abs/2108.10097
[56]
Jialin Zhao, Yuxiao Dong, Ming Ding, Evgeny Kharlamov, and Jie Tang. 2021. Adaptive Diffusion in Graph Neural Networks. Advances in Neural Information Processing Systems, Vol. 34 (2021), 23321--23333.
[57]
Xin Zheng, Yixin Liu, Shirui Pan, Miao Zhang, Di Jin, and Philip S Yu. 2022. Graph neural networks for graphs with heterophily: A survey. arXiv preprint arXiv:2202.07082 (2022).
[58]
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. 2020. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, Vol. 33 (2020), 7793--7804.
[59]
Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. 2021. Interpreting and unifying graph neural networks with an optimization framework. In Proceedings of the Web Conference 2021. 1215--1226.
Index Terms
- Clenshaw Graph Neural Networks
Recommendations
Difference Residual Graph Neural Networks
MM '22: Proceedings of the 30th ACM International Conference on MultimediaGraph Neural Networks have been widely employed for multimodal fusion and embedding. To overcome over-smoothing issue, residual connections, which are designed for alleviating vanishing gradient problem in NNs, are adopted in Graph Neural Networks (GNNs)...
Evolving Super Graph Neural Networks for Large-Scale Time-Series Forecasting
Advances in Knowledge Discovery and Data MiningAbstractGraph Recurrent Neural Networks (GRNN) excel in time-series prediction by modeling complicated non-linear relationships among time-series. However, most GRNN models target small datasets that only have tens of time-series or hundreds of time-...
Short-term residential load forecasting using Graph Convolutional Recurrent Neural Networks
AbstractThe abundance of energy consumption data collected by smart meters has inspired researchers to employ deep neural networks to solve the existing problems in the power industry, such as Short-Term Load Forecasting (STLF). Most studies ...
Comments
Information & Contributors
Information
Published In
August 2023
5996 pages
ISBN:9798400701030
DOI:10.1145/3580305
- General Chairs:
- Ambuj Singh,
- Yizhou Sun,
- Program Chairs:
- Leman Akoglu,
- Dimitrios Gunopulos,
- Xifeng Yan,
- Ravi Kumar,
- Fatma Ozcan,
- Jieping Ye
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 August 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Alibaba Group through Alibaba Innovative Research Program
- major key project of PCL
- National Natural Science Foundation of China
- Beijing Natural Science Foundation
- Beijing Outstanding Young Scientist Program
- Huawei-Renmin University joint program on Information Retrieval
Conference
KDD '23: The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
August 6 - 10, 2023
CA, Long Beach, USA
Acceptance Rates
Overall Acceptance Rate 1,133 of 8,635 submissions, 13%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 418Total Downloads
- Downloads (Last 12 months)418
- Downloads (Last 6 weeks)38
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in
Access Granted