research-article

Graph-adaptive Rectified Linear Unit for Graph Neural Networks

Authors:

Irwin KingAuthors Info & Claims

WWW '22: Proceedings of the ACM Web Conference 2022

Pages 1331 - 1339

https://doi.org/10.1145/3485447.3512159

Published: 25 April 2022 Publication History

Abstract

Graph Neural Networks (GNNs) have achieved remarkable success by extending traditional convolution to learning on non-Euclidean data. The key to the GNNs is adopting the neural message-passing paradigm with two stages: aggregation and update. The current design of GNNs considers the topology information in the aggregation stage. However, in the updating stage, all nodes share the same updating function. The identical updating function treats each node embedding as i.i.d. random variables and thus ignores the implicit relationships between neighborhoods, which limits the capacity of the GNNs. The updating function is usually implemented with a linear transformation followed by a non-linear activation function. To make the updating function topology-aware, we inject the topological information into the non-linear activation function and propose Graph-adaptive Rectified Linear Unit (GReLU), which is a new parametric activation function incorporating the neighborhood information in a novel and efficient way. The parameters of GReLU are obtained from a hyperfunction based on both node features and the corresponding adjacent matrix. To reduce the risk of overfitting and the computational cost, we decompose the hyperfunction as two independent components for nodes and features respectively. We conduct comprehensive experiments to show that our plug-and-play GReLU method is efficient and effective given different GNN backbones and various downstream tasks.

References

[1]

Filippo Maria Bianchi, Daniele Grattarola, Lorenzo Livi, and Cesare Alippi. 2019. Graph neural networks with convolutional arma filters. arXiv preprint arXiv:1901.01343(2019).

[2]

Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. 2020. Dynamic ReLU. arXiv preprint arXiv:2003.10027(2020).

[3]

Yankai Chen, Menglin Yang, Yingxue Zhang, Mengchen Zhao, Ziqiao Meng, Jianye Hao, and Irwin King. 2022. Modeling Scale-free Graphs with Hyperbolic Geometry for Knowledge-aware Recommendation. In The Fifteenth ACM International Conference on Web Search and Data Mining.

[4]

Yankai Chen, Yaming Yang, Yujing Wang, Jing Bai, Xiangchen Song, and Irwin King. 2022. Attentive Knowledge-aware Graph Convolutional Networks with Collaborative Guidance for Personalized Recommendation. In The 38th IEEE International Conference on Data Engineering.

[5]

Yankai Chen, Jie Zhang, Yixiang Fang, Xin Cao, and Irwin King. 2020. Efficient community search over large directed graphs: An augmented index-based approach. In IJCAI. 3544–3550.

[6]

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289(2015).

[7]

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems 29 (2016), 3844–3852.

[8]

Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. 2000. Incorporating second-order functional knowledge for better option pricing. Advances in neural information processing systems 13 (2000), 472–478.

[9]

Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. 2020. MAGNN: metapath aggregated graph neural network for heterogeneous graph embedding. In Proceedings of The Web Conference 2020. 2331–2341.

Digital Library

[10]

Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. 2013. Maxout networks. In International conference on machine learning. PMLR, 1319–1327.

[11]

Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Advances in neural information processing systems. 1024–1034.

[12]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision. 1026–1034.

Digital Library

[13]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.

[14]

Kevin Jarrett, Koray Kavukcuoglu, Marc’Aurelio Ranzato, and Yann LeCun. 2009. What is the best multi-stage architecture for object recognition?. In 2009 IEEE 12th international conference on computer vision. IEEE, 2146–2153.

[15]

Dongkwan Kim and Alice H. Oh. 2021. How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. https://openreview.net/forum?id=Wi5KUNlqWty

[16]

Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907(2016).

[17]

Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-normalizing neural networks. Advances in neural information processing systems 30 (2017), 971–980.

[18]

Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. 2018. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05997(2018).

[19]

Piotr Koniusz, Hongguang Zhang, and Fatih Porikli. 2018. A Deeper Look at Power Normalizations. In CVPR. 5774–5783.

[20]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature 521, 7553 (2015), 436–444.

[21]

Dongho Lee, Byungkook Oh, Seungmin Seo, and Kyong-Ho Lee. 2020. News Recommendation with Topic-Enriched Knowledge Graphs. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 695–704.

Digital Library

[22]

Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2015. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493(2015).

[23]

Nicholas Lim, Bryan Hooi, See-Kiong Ng, Xueou Wang, Yong Liang Goh, Renrong Weng, and Jagannadan Varadarajan. 2020. STP-UDGAT: Spatial-Temporal-Preference User Dimensional Graph Attention Network for Next POI Recommendation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 845–854.

Digital Library

[24]

J. Mairal, P. Koniusz, Z. Harchaoui, and C. Schmid. 2014. Convolutional Kernel Networks. NIPS (2014).

[25]

Zaiqiao Meng, Shangsong Liang, Hongyan Bao, and Xiangliang Zhang. 2019. Co-embedding attributed networks. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. 393–401.

Digital Library

[26]

Diganta Misra. 2019. Mish: A self regularized non-monotonic neural activation function. arXiv preprint arXiv:1908.08681(2019).

[27]

Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In ICML.

[28]

Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. 2016. Understanding and improving convolutional neural networks via concatenated rectified linear units. In international conference on machine learning. PMLR, 2217–2225.

[29]

Zixing Song, Ziqiao Meng, Yifei Zhang, and Irwin King. 2021. Semi-supervised Multi-label Learning for Graph-structured Data. In CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021, Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong (Eds.). ACM, 1723–1733. https://doi.org/10.1145/3459637.3482391

Digital Library

[30]

Zixing Song, Xiangli Yang, Zenglin Xu, and Irwin King. 2021. Graph-based Semi-supervised Learning: A Comprehensive Review. CoRR abs/2102.13303(2021). arXiv:2102.13303https://arxiv.org/abs/2102.13303

[31]

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15, 1 (2014), 1929–1958.

Digital Library

[32]

Ke Sun, Piotr Koniusz, and Zhen Wang. 2020. Fisher-Bures Adversary Graph Convolutional Networks. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference(Proceedings of Machine Learning Research, Vol. 115), Ryan P. Adams and Vibhav Gogate (Eds.). PMLR, 465–475. http://proceedings.mlr.press/v115/sun20a.html

[33]

Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903(2017).

[34]

Yaqing Wang, Fenglong Ma, and Jing Gao. 2020. Efficient Knowledge Graph Validation via Cross-Graph Representation Learning. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 1595–1604.

Digital Library

[35]

Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. 2019. Simplifying graph convolutional networks. In International conference on machine learning. PMLR, 6861–6871.

[36]

Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu. 2021. A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems 32, 1(2021), 4–24. https://doi.org/10.1109/TNNLS.2020.2978386

[37]

Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. 2015. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853(2015).

[38]

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018. How powerful are graph neural networks?arXiv preprint arXiv:1810.00826(2018).

[39]

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2019. How Powerful are Graph Neural Networks?. In International Conference on Learning Representations. https://openreview.net/forum?id=ryGs6iA5Km

[40]

Menglin Yang, Ziqiao Meng, and Irwin King. 2020. FeatureNorm: L2 Feature Normalization for Dynamic Graph Embedding. In 2020 IEEE International Conference on Data Mining (ICDM). IEEE, 731–740.

[41]

Menglin Yang, Min Zhou, Marcus Kalander, Zengfeng Huang, and Irwin King. 2021. Discrete-time Temporal Network Embedding via Implicit Hierarchical Learning in Hyperbolic Space. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 1975–1985.

Digital Library

[42]

Hao Zhu and Piotr Koniusz. 2020. Simple spectral graph convolution. In International Conference on Learning Representations.

[43]

Hao Zhu and Piotr Koniusz. 2021. Refine: Random range finder for network embedding. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 3682–3686.

Digital Library

[44]

Hao Zhu, Ke Sun, and Peter Koniusz. 2021. Contrastive laplacian eigenmaps. Advances in Neural Information Processing Systems 34 (2021).

[45]

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. 2020. Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/58ae23d878a47004366189884c2f8440-Abstract.html

Cited By

Khan MMello GHabib LEngelstad PYazidi A(2024)HITS-based Propagation Paradigm for Graph Neural NetworksACM Transactions on Knowledge Discovery from Data10.1145/363877918:4(1-23)Online publication date: 13-Feb-2024
https://dl.acm.org/doi/10.1145/3638779
Zhang YZhu HSong ZChen YFu XMeng ZKoniusz PKing IBaeza-Yates RBonchi F(2024)Geometric View of Soft Decorrelation in Self-Supervised LearningProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671914(4338-4349)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671914
Gong ZLi BWang CChen JZhao P(2024)BF-SAM: enhancing SAM through multi-modal fusion for fine-grained building function identificationInternational Journal of Geographical Information Science10.1080/13658816.2024.2399142(1-27)Online publication date: 5-Sep-2024
https://doi.org/10.1080/13658816.2024.2399142
Show More Cited By

Index Terms

Graph-adaptive Rectified Linear Unit for Graph Neural Networks

Index terms have been assigned to the content through auto-classification.

Recommendations

A novel softplus linear unit for deep convolutional neural networks

Current improvements in the performance of deep neural networks are partly due to the proposition of rectified linear units. A ReLU activation function outputs zero for negative component, inducing the death of some neurons and a bias shift of the ...
Toward the analysis of graph neural networks
ICSE-NIER '22: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results

Graph Neural Networks (GNNs) have recently emerged as an effective framework for representing and analyzing graph-structured data. GNNs have been applied to many real-world problems such as knowledge graph analysis, social networks recommendation, and ...
Graph Neural Networks: Foundation, Frontiers and Applications
KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

The field of graph neural networks (GNNs) has seen rapid and incredible strides over the recent years. Graph neural networks, also known as deep learning on graphs, graph representation learning, or geometric deep learning, have become one of the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '22: Proceedings of the ACM Web Conference 2022

April 2022

3764 pages

ISBN:9781450390965

DOI:10.1145/3485447

Editors:
Frédérique Laforest
INSA Lyon, France
,
Raphaël Troncy
EURECOM, France
,
Elena Simperl
King’s College London, UK
,
Deepak Agarwal
Pinterest, USA
,
Aristides Gionis
KTH Royal Institute of Technology, Sweden
,
Ivan Herman
W3C / retired
,
Lionel Médini
Université Lyon 1, France

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Key Research and Development Program of China
CUHK 2300174, Collaborative Research Fund (CRF)

Conference

WWW '22

Sponsor:

SIGWEB

WWW '22: The ACM Web Conference 2022

April 25 - 29, 2022

Virtual Event, Lyon, France

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
347
Total Downloads

Downloads (Last 12 months)75
Downloads (Last 6 weeks)11

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Khan MMello GHabib LEngelstad PYazidi A(2024)HITS-based Propagation Paradigm for Graph Neural NetworksACM Transactions on Knowledge Discovery from Data10.1145/363877918:4(1-23)Online publication date: 13-Feb-2024
https://dl.acm.org/doi/10.1145/3638779
Zhang YZhu HSong ZChen YFu XMeng ZKoniusz PKing IBaeza-Yates RBonchi F(2024)Geometric View of Soft Decorrelation in Self-Supervised LearningProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671914(4338-4349)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671914
Gong ZLi BWang CChen JZhao P(2024)BF-SAM: enhancing SAM through multi-modal fusion for fine-grained building function identificationInternational Journal of Geographical Information Science10.1080/13658816.2024.2399142(1-27)Online publication date: 5-Sep-2024
https://doi.org/10.1080/13658816.2024.2399142
Wang KGe TZhao LJiang W(2024) Research on SO 3 prediction method in thermal power plant flue gas based on machine learning E3S Web of Conferences10.1051/e3sconf/202453603010536(03010)Online publication date: 10-Jun-2024
https://doi.org/10.1051/e3sconf/202453603010
Song ZZhang YKing IOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Optimal block-wise asymmetric graph construction for graph-based semi-supervised learningProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3669237(71135-71149)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3669237
Zhang YZhu HChen YSong ZKoniusz PKing IOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Mitigating the popularity bias of graph collaborative filteringProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3669074(67533-67550)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3669074
Song ZZhang YKing IOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)No change, no gainProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3668180(47511-47526)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3668180
Song ZZhang YKing IFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Towards Fair Financial Services for All: A Temporal GNN Approach for Individual Fairness on Transaction NetworksProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615091(2331-2341)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615091
Yang MZhou MPan LKing ISingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)κHGCN: Tree-likeness Modeling via Continuous and Discrete Curvature LearningProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599532(2965-2977)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599532
Zhang YChen YSong ZKing ISingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)Contrastive Cross-scale Graph Knowledge SynergyProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599286(3422-3433)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599286
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents