research-article

Designing the Topology of Graph Neural Networks: A Novel Feature Fusion Perspective

Authors:

Zhiqiang HeAuthors Info & Claims

WWW '22: Proceedings of the ACM Web Conference 2022

Pages 1381 - 1391

https://doi.org/10.1145/3485447.3512185

Published: 25 April 2022 Publication History

Abstract

In recent years, Graph Neural Networks (GNNs) have shown superior performance on diverse real-world applications. To improve the model capacity, besides designing aggregation operations, GNN topology design is also very important. In general, there are two mainstream GNN topology design manners. The first one is to stack aggregation operations to obtain the higher-level features but easily got performance drop as the network goes deeper. Secondly, the multiple aggregation operations are utilized in each layer which provides adequate and independent feature extraction stage on local neighbors while are costly to obtain the higher-level information. To enjoy the benefits while alleviating the corresponding deficiencies of these two manners, we learn to design the topology of GNNs in a novel feature fusion perspective which is dubbed F2GNN. To be specific, we provide a feature fusion perspective in designing GNN topology and propose a novel framework to unify the existing topology designs with feature selection and fusion strategies. Then we develop a neural architecture search method on top of the unified framework which contains a set of selection and fusion operations in the search space and an improved differentiable search algorithm. The performance gains on diverse datasets, five homophily and three heterophily ones, demonstrate the effectiveness of F2GNN. We further conduct experiments to show that F2GNN can improve the model capacity while alleviating the deficiencies of existing GNN topology design manners, especially alleviating the over-smoothing problem, by utilizing different levels of features adaptively. 1

References

[1]

Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. 2019. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In ICML. PMLR, 21–29.

[2]

Aleksandar Bojchevski and Stephan Günnemann. 2018. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. ICLR.

[3]

Shaofei Cai, Liang Li, Jincan Deng, Beichen Zhang, Zheng-Jun Zha, Li Su, and Qingming Huang. 2021. Rethinking Graph Neural Network Search from Message-passing. CVPR (2021).

[4]

Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. 2020. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In AAAI, Vol. 34. 3438–3445.

[5]

Jiamin Chen, Jianliang Gao, Yibo Chen, Moctard Babatounde Oloulade, Tengfei Lyu, and Zhao Li. 2021. GraphPAS: Parallel Architecture Search for Graph Neural Networks. In SIGIR. 2182–2186.

[6]

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. 2020. Simple and deep graph convolutional networks. In ICML. PMLR, 1725–1735.

[7]

Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. 2019. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In ICCV. 1294–1303.

[8]

Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. 2021. Progressive darts: Bridging the optimization gap for nas in the wild. International Journal of Computer Vision 129, 3 (2021), 638–655.

Digital Library

[9]

Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. 2020. Principal Neighbourhood Aggregation for Graph Nets. In NeurIPS, Vol. 33. 13260–13271.

[10]

Corinna Cortes, Xavier Gonzalvo, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. 2017. Adanet: Adaptive structural learning of artificial neural networks. In ICML. PMLR, 874–883.

[11]

Yuhui Ding, Quanming Yao, Huan Zhao, and Tong Zhang. 2021. Diffmg: Differentiable meta graph search for heterogeneous graph neural networks. In KDD. 279–288.

[12]

Lun Du, Xiaozhou Shi, Qiang Fu, Hengyu Liu, Shi Han, and Dongmei Zhang. 2022. GBK-GNN: Gated Bi-Kernel Graph Neural Networks for Modeling Both Homophily and Heterophily. In TheWebConf.

[13]

Wenzheng Feng, Jie Zhang, Yuxiao Dong, Yu Han, Huanbo Luan, Qian Xu, Qiang Yang, Evgeny Kharlamov, and Jie Tang. 2020. Graph Random Neural Networks for Semi-Supervised Learning on Graphs. NeurIPS 33(2020).

[14]

Matthias Fey and Jan E. Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric.

[15]

Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. 2020. Graphnas: Graph neural architecture search with reinforcement learning. In IJCAI.

[16]

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural Message Passing for Quantum Chemistry. In ICML. 1263–1272.

[17]

Yu-Chao Gu, Li-Juan Wang, Yun Liu, Yi Yang, Yu-Huan Wu, Shao-Ping Lu, and Ming-Ming Cheng. 2021. Dots: Decoupling operation and topology in differentiable architecture search. In CVPR. 12311–12320.

[18]

Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. 2020. Single path one-shot neural architecture search with uniform sampling. In ECCV. Springer, 544–560.

[19]

Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In NeurIPS. 1024–1034.

[20]

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020. Open Graph Benchmark: Datasets for Machine Learning on Graphs. In NeurIPS. 22118–22133.

[21]

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In CVPR. 4700–4708.

[22]

Anees Kazi, Shayan Shekarforoush, S Arvind Krishna, Hendrik Burwinkel, Gerome Vivar, Karsten Kortüm, Seyed-Ahmad Ahmadi, Shadi Albarqouni, and Nassir Navab. 2019. InceptionGCN: receptive field aware graph convolutional network for disease prediction. In IPMI. Springer, 73–85.

[23]

Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. ICLR.

[24]

Dawei Leng, Jinjiang Guo, Lurong Pan, Jie Li, and Xinyu Wang. 2021. Enhance Information Propagation for Graph Neural Network by Heterogeneous Aggregations. arXiv preprint arXiv:2102.04064(2021).

[25]

Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. 2019. Deepgcns: Can gcns go as deep as cnns?. In ICCV. 9267–9276.

[26]

Guohao Li, Guocheng Qian, Itzel C Delgadillo, Matthias Muller, Ali Thabet, and Bernard Ghanem. 2020. Sgas: Sequential greedy architecture search. In CVPR. 1620–1630.

[27]

Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018. Deeper insights into graph convolutional networks for semi-supervised learning. In AAAI, Vol. 32.

[28]

Yaoman Li and Irwin King. 2020. AutoGraph: Automated Graph Neural Network. In ICONIP. 189–201.

[29]

Yanxi Li, Zean Wen, Yunhe Wang, and Chang Xu. 2021. One-shot Graph Neural Architecture Search with Dynamic Search Space. In AAAI.

[30]

Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2019. DARTS: Differentiable architecture search. ICLR.

[31]

Meng Liu, Hongyang Gao, and Shuiwang Ji. 2020. Towards deeper graph neural networks. In KDD. 338–348.

[32]

Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image-based recommendations on styles and substitutes. In SIGIR. 43–52.

[33]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, 2019. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS. 8026–8037.

[34]

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. 2020. Geom-gcn: Geometric graph convolutional networks. ICLR.

[35]

Yijian Qin, Xin Wang, Zeyang Zhang, and Wenwu Zhu. 2021. Graph Differentiable Architecture Search with Structure Learning. NeurIPS 34.

[36]

Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. 2020. Dropedge: Towards deep graph convolutional networks on node classification. ICLR.

[37]

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008. Collective classification in network data. AI magazine 29, 3 (2008), 93–93.

[38]

Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. 2018. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868(2018).

[39]

Diego Valsesia, Giulia Fracastoro, and Enrico Magli. 2020. Don’t stack layers in graph neural networks, wire them randomly.

[40]

Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. ICLR.

[41]

Zhili Wang, Shimin Di, and Lei Chen. 2021. AutoGEL: An Automated Graph Neural Network with Explicit Link Information. NeurIPS 34(2021).

[42]

Zhenyi Wang, Huan Zhao, and Chuan Shi. 2022. Profiling the Design Space for Graph Neural Networks based Collaborative Filtering. In WSDM.

[43]

Lanning Wei, Huan Zhao, Quanming Yao, and Zhiqiang He. 2021. Pooling architecture search for graph classification. In CIKM. 2091–2100.

[44]

Lingxi Xie, Xin Chen, Kaifeng Bi, Longhui Wei, Yuhui Xu, Lanfei Wang, Zhengsu Chen, An Xiao, Jianlong Chang, Xiaopeng Zhang, 2021. Weight-sharing neural architecture search: A battle to shrink the optimization gap. ACM Computing Surveys (CSUR) 54, 9 (2021), 1–37.

Digital Library

[45]

Sirui Xie, Shoukang Hu, Xinjiang Wang, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. 2021. Understanding the wiring evolution in differentiable neural architecture search. In AISTATS. 874–882.

[46]

Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. 2019. Exploring randomly wired neural networks for image recognition. In ICCV. 1284–1293.

[47]

Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. 2018. SNAS: stochastic neural architecture search. ICLR.

[48]

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2019. How powerful are graph neural networks?ICLR.

[49]

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. 2018. Representation Learning on Graphs with Jumping Knowledge Networks. In ICML. 5453–5462.

[50]

Jiaxuan You, Zhitao Ying, and Jure Leskovec. 2020. Design space for graph neural networks. NeurIPS 33.

[51]

Kun Yuan, Quanquan Li, Shaopeng Guo, Dapeng Chen, Aojun Zhou, Fengwei Yu, and Ziwei Liu. 2021. Differentiable Dynamic Wirings for Neural Networks. In ICCV. 327–336.

[52]

Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. 2020. Graphsaint: Graph sampling based inductive learning method. ICLR.

[53]

Yongqi Zhang and Quanming Yao. 2022. Knowledge Graph Reasoning with Relational Directed Graph. In TheWebConf.

[54]

Yongqi Zhang, Quanming Yao, and Lei Chen. 2020. Interstellar: Searching Recurrent Architecture for Knowledge Graph Embedding. In NeurIPS.

[55]

Yongqi Zhang, Quanming Yao, Wenyuan Dai, and Lei Chen. 2020. AutoSF: Searching Scoring Functions for Knowledge Graph Embedding. In ICDE. IEEE, 433–444.

[56]

Ziwei Zhang, Xin Wang, and Wenwu Zhu. 2021. Automated Machine Learning on Graphs: A Survey. In IJCAI. 4704–4712.

[57]

Huan Zhao, Lanning Wei, and Quanming Yao. 2020. Simplifying Architecture Search for Graph Neural Network.

[58]

Huan Zhao, Quanming Yao, and Weiwei Tu. 2021. Search to aggregate neighborhood for graph neural network. In ICDE.

[59]

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. 2020. Beyond homophily in graph neural networks: Current limitations and effective designs. In NeurIPS.

[60]

Barret Zoph and Quoc V Le. 2017. Neural architecture search with reinforcement learning. ICLR.

Cited By

Gao JWu ZAl-Sabri ROloulade BChen J(2024)AutoDDI: Drug–Drug Interaction Prediction With Automated Graph Neural NetworkIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2024.334957028:3(1773-1784)Online publication date: Mar-2024
https://doi.org/10.1109/JBHI.2024.3349570
Li HDi SChen LZhou X(2024) E 2 GCL: Efficient and Expressive Contrastive Learning on Graph Neural Networks 2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00071(859-873)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00071
Wang YChen JLi QHe CGao J(2024)Graph neural architecture search with heterogeneous message-passing mechanismsKnowledge and Information Systems10.1007/s10115-024-02090-x66:7(4283-4308)Online publication date: 12-Apr-2024
https://dl.acm.org/doi/10.1007/s10115-024-02090-x
Show More Cited By

Index Terms

Designing the Topology of Graph Neural Networks: A Novel Feature Fusion Perspective
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Information systems

Index terms have been assigned to the content through auto-classification.

Recommendations

Pooling Architecture Search for Graph Classification
CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

Graph classification is an important problem with applications across many domains, like chemistry and bioinformatics, for which graph neural networks (GNNs) have been state-of-the-art (SOTA) methods. GNNs are designed to learn node-level representation ...
Feature Selection and Extraction for Graph Neural Networks
ACM SE '20: Proceedings of the 2020 ACM Southeast Conference

Graph Neural Networks (GNNs) have been a latest hot research topic in data science, due to the fact that they use the ubiquitous data structure graphs as the underlying elements for constructing and training neural networks. In a GNN, each node has ...
GraphPAS: Parallel Architecture Search for Graph Neural Networks
SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Graph neural architecture search has received a lot of attention as Graph Neural Networks (GNNs) has been successfully applied on the non-Euclidean data recently. However, exploring all possible GNNs architectures in the huge search space is too time-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '22: Proceedings of the ACM Web Conference 2022

April 2022

3764 pages

ISBN:9781450390965

DOI:10.1145/3485447

Editors:
Frédérique Laforest
INSA Lyon, France
,
Raphaël Troncy
EURECOM, France
,
Elena Simperl
King’s College London, UK
,
Deepak Agarwal
Pinterest, USA
,
Aristides Gionis
KTH Royal Institute of Technology, Sweden
,
Ivan Herman
W3C / retired
,
Lionel Médini
Université Lyon 1, France

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '22

Sponsor:

SIGWEB

WWW '22: The ACM Web Conference 2022

April 25 - 29, 2022

Virtual Event, Lyon, France

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
591
Total Downloads

Downloads (Last 12 months)154
Downloads (Last 6 weeks)11

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gao JWu ZAl-Sabri ROloulade BChen J(2024)AutoDDI: Drug–Drug Interaction Prediction With Automated Graph Neural NetworkIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2024.334957028:3(1773-1784)Online publication date: Mar-2024
https://doi.org/10.1109/JBHI.2024.3349570
Li HDi SChen LZhou X(2024) E 2 GCL: Efficient and Expressive Contrastive Learning on Graph Neural Networks 2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00071(859-873)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00071
Wang YChen JLi QHe CGao J(2024)Graph neural architecture search with heterogeneous message-passing mechanismsKnowledge and Information Systems10.1007/s10115-024-02090-x66:7(4283-4308)Online publication date: 12-Apr-2024
https://dl.acm.org/doi/10.1007/s10115-024-02090-x
纪雷(2023)Analysis of Security Issues in Embedding Knowledge GraphSoftware Engineering and Applications10.12677/SEA.2023.12608212:06(844-850)Online publication date: 2023
https://doi.org/10.12677/SEA.2023.126082
Wang ZDi SChen LSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)A Message Passing Neural Network Space for Better Capturing Data-dependent Receptive FieldsProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599243(2489-2501)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599243
Wang CJiao LZhao JLi LLiu XLiu FYang S(2023)Bi-Level Multiobjective Evolutionary Learning: A Case Study on Multitask Graph Neural Topology SearchIEEE Transactions on Evolutionary Computation10.1109/TEVC.2023.325526328:1(208-222)Online publication date: 10-Mar-2023
https://dl.acm.org/doi/10.1109/TEVC.2023.3255263
Liu YLiu J(2023)A surrogate evolutionary neural architecture search algorithm for graph neural networksApplied Soft Computing10.1016/j.asoc.2023.110485144:COnline publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1016/j.asoc.2023.110485
Wang ZYang FXu QWang YYan HXie M(2023)Capacity estimation of lithium-ion batteries based on data aggregation and feature fusion via graph neural networkApplied Energy10.1016/j.apenergy.2023.120808336(120808)Online publication date: Apr-2023
https://doi.org/10.1016/j.apenergy.2023.120808
Xu ZWei LZhao HYing RYao QTu WGuyon I(2022)Bridging the Gap of AutoGraph Between Academia and Industry: Analyzing AutoGraph Challenge at KDD Cup 2020Frontiers in Artificial Intelligence10.3389/frai.2022.9051045Online publication date: 16-Jun-2022
https://doi.org/10.3389/frai.2022.905104

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents