research-article

Genetic-GNN: : Evolutionary architecture search for Graph Neural Networks

Authors:

Jianxun LiuAuthors Info & Claims

Volume 247, Issue C

https://doi.org/10.1016/j.knosys.2022.108752

Published: 08 July 2022 Publication History

Abstract

Neural architecture search (NAS) has seen significant attention throughout the computational intelligence research community and has pushed forward the state-of-the-art of many neural models to address grid-like data such as texts and images. However, little work has been done on Graph Neural Network (GNN) models dedicated to unstructured network data. Given the huge number of choices and combinations of components such as aggregators and activation functions, determining the suitable GNN model for a specific problem normally necessitates tremendous expert knowledge and laborious trials. In addition, the moderate change of hyperparameters such as the learning rate and dropout rate would dramatically impact the learning capacity of a GNN model. In this paper, we propose a novel framework through the evolution of individual models in a large GNN architecture searching space. Instead of simply optimizing the model structures, an alternating evolution process is performed between GNN model structures and hyperparameters to dynamically approach the optimal fit of each other. Experiments and validations demonstrate that evolutionary NAS is capable of matching existing state-of-the-art reinforcement learning methods for both transductive and inductive graph representation learning and node classification.

References

[1]

Zhang D., Yin J., Zhu X., Zhang C., Network representation learning: A survey, IEEE Trans. Big Data (2018).

[2]

Cui P., Wang X., Pei J., Zhu W., A survey on network embedding, IEEE Trans. Knowl. Data Eng. 31 (5) (2018) 833–852.

[3]

Wu Z., Pan S., Chen F., Long G., Zhang C., Philip S.Y., A comprehensive survey on graph neural networks, IEEE Trans. Neural Netw. Learn. Syst. (2020).

[4]

Kipf T.N., Welling M., Semi-supervised classification with graph convolutional networks, in: International Conference on Learning Representations (ICLR), 2017.

[5]

Z.-M. Chen, X.-S. Wei, P. Wang, Y. Guo, Multi-label image recognition with graph convolutional networks, in: Proc. of IEEE CVPR, 2019, pp. 5177–5186.

[6]

Sun M., Zhao S., Gilvary C., Elemento O., Zhou J., Wang F., Graph convolutional networks for computational drug development and discovery, Brief Bioinform. (2019).

[7]

Zhao L., Song Y., Zhang C., Liu Y., Wang P., Lin T., Deng M., Li H., T-gcn: A temporal graph convolutional network for traffic prediction, IEEE Trans. Intell. Transp. Syst. (2019).

[8]

Zhang Z., Cui P., Zhu W., Deep learning on graphs: A survey, IEEE Trans. Knowl. Data Eng. (2020).

Digital Library

[9]

Kriege N.M., Johansson F.D., Morris C., A survey on graph kernels, Appl. Netw. Sci. 5 (1) (2020) 1–42.

[10]

Hamilton W., Ying Z., Leskovec J., Inductive representation learning on large graphs, in: Advances in Neural Information Processing Systems, 2017, pp. 1024–1034.

[11]

He X., Zhao K., Chu X., AutoML: A survey of the state-of-the-art, Knowl.-Based Syst. 212 (2021).

[12]

Elsken T., Metzen J.H., Hutter F., Neural architecture search: A survey, 2018, arXiv preprint arXiv:1808.05377.

[13]

Ding Y., Yao Q., Zhang T., Propagation model search for graph neural networks, 2020, arXiv preprint arXiv:2010.03250.

[14]

X. Dong, Y. Yang, One-shot neural architecture search via self-evaluated template network, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 3681–3690.

[15]

Y. Chen, G. Meng, Q. Zhang, S. Xiang, C. Huang, L. Mu, X. Wang, Renas: Reinforced evolutionary neural architecture search, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 4787–4796.

[16]

Gao Y., Yang H., Zhang P., Zhou C., Hu Y., GraphNAS: Graph neural architecture search with reinforcement learning, 2019, arXiv:1904.09981.

[17]

Zhou K., Song Q., Huang X., Hu X., Auto-GNN: Neural architecture search of graph neural networks, 2019, arXiv:1909.03184.

[18]

Gotmare A., Keskar N.S., Xiong C., Socher R., A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation, in: International Conference on Learning Representations, 2018.

[19]

Min S., Gao Z., Peng J., Wang L., Qin K., Fang B., STGSN—A Spatial–Temporal Graph Neural Network framework for time-evolving social networks, Knowl.-Based Syst. 214 (2021).

[20]

Molaei S., Zare H., Veisi H., Deep learning approach on information diffusion in heterogeneous networks, Knowl.-Based Syst. 189 (2020).

[21]

Su C., Tong J., Zhu Y., Cui P., Wang F., Network embedding in biomedical data science, Brief. Bioinform. 21 (1) (2020) 182–197.

[22]

Chen C., Li K., Wei W., Zhou J.T., Zeng Z., Hierarchical graph neural networks for few-shot learning, IEEE Trans. Circuits Syst. Video Technol. (2021).

[23]

C. Chen, K. Li, S.G. Teo, X. Zou, K. Wang, J. Wang, Z. Zeng, Gated residual recurrent graph neural networks for traffic prediction, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 2019, pp. 485–492.

[24]

Micheli A., Neural network for graphs: A contextual constructive approach, IEEE Trans. Neural Netw. 20 (3) (2009) 498–511.

[25]

Defferrard M., Bresson X., Vandergheynst P., Convolutional neural networks on graphs with fast localized spectral filtering, in: Advances in Neural Information Processing Systems, 2016, pp. 3844–3852.

[26]

Niepert M., Ahmed M., Kutzkov K., Learning convolutional neural networks for graphs, in: International Conference on Machine Learning, 2016, pp. 2014–2023.

[27]

Zhang X., Chen W., Yan H., TLINE: Scalable transductive network embedding, in: Asia Information Retrieval Symposium, Springer, 2016, pp. 98–110.

[28]

Yang Z., Dong S., HAGERec: hierarchical attention graph convolutional network incorporating knowledge graph for explainable recommendation, Knowl.-Based Syst. 204 (2020).

[29]

Li Y., Tarlow D., Brockschmidt M., Zemel R., Gated graph sequence neural networks, 2015, arXiv preprint arXiv:1511.05493.

[30]

Atwood J., Towsley D., Diffusion-convolutional neural networks, in: Advances in Neural Information Processing Systems, 2016, pp. 1993–2001.

[31]

Veličković P., Cucurull G., Casanova A., Romero A., Lio P., Bengio Y., Graph attention networks, 2017, arXiv preprint arXiv:1710.10903.

[32]

C. Liu, L.-C. Chen, F. Schroff, H. Adam, W. Hua, A.L. Yuille, L. Fei-Fei, Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 82–92.

[33]

Wang Y., Yang Y., Chen Y., Bai J., Zhang C., Su G., Kou X., Tong Y., Yang M., Zhou L., TextNAS: A neural architecture search space tailored for text representation, in: AAAI, 2020, pp. 9242–9249.

[34]

Li S., Li W., Wen S., Shi K., Yang Y., Zhou P., Huang T., Auto-FERNet: A facial expression recognition network with architecture search, IEEE Trans. Netw. Sci. Eng. (2021).

[35]

Wistuba M., Rawat A., Pedapati T., A survey on neural architecture search, 2019, arXiv preprint arXiv:1905.01392.

[36]

Jaafra Y., Laurent J.L., Deruyver A., Naceur M.S., Reinforcement learning for neural architecture search: A review, Image Vis. Comput. 89 (2019) 57–66.

Digital Library

[37]

Lyu B., Wen S., Shi K., Huang T., Multiobjective reinforcement learning-based neural architecture search for efficient portrait parsing, IEEE Trans. Cybern. (2021).

[38]

Zoph B., Le Q.V., Neural architecture search with reinforcement learning, 2016, arXiv preprint arXiv:1611.01578.

[39]

Baker B., Gupta O., Naik N., Raskar R., Designing neural network architectures using reinforcement learning, 2016, arXiv preprint arXiv:1611.02167.

[40]

Cai H., Chen T., Zhang W., Yu Y., Wang J., Efficient architecture search by network transformation, in: Thirty-Second AAAI Conference on Artificial Intelligence, 2018.

[41]

He X., Zhao K., Chu X., AutoML: A survey of the state-of-the-art, 2019, arXiv preprint arXiv:1908.00709.

[42]

Bergstra J.S., Bardenet R., Bengio Y., Kégl B., Algorithms for hyper-parameter optimization, in: Advances in Neural Information Processing Systems, 2011, pp. 2546–2554.

[43]

Hutter F., Hoos H.H., Leyton-Brown K., Sequential model-based optimization for general algorithm configuration, in: International Conference on Learning and Intelligent Optimization, Springer, 2011, pp. 507–523.

[44]

Kandasamy K., Neiswanger W., Schneider J., Poczos B., Xing E.P., Neural architecture search with bayesian optimisation and optimal transport, in: Advances in Neural Information Processing Systems, 2018, pp. 2016–2025.

[45]

Ho K., Gilbert A., Jin H., Collomosse J., Neural architecture search for deep image prior, 2020, arXiv preprint arXiv:2001.04776.

[46]

L. Xie, A. Yuille, Genetic cnn, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1379–1388.

[47]

M. Suganuma, S. Shirakawa, T. Nagao, A genetic programming approach to designing convolutional neural network architectures, in: Proceedings of the Genetic and Evolutionary Computation Conference, 2017, pp. 497–504.

[48]

Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser Ł., Polosukhin I., Attention is all you need, in: Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.

[49]

Sheikh R.H., Raghuwanshi M.M., Jaiswal A.N., Genetic algorithm based clustering: a survey, in: 2008 First International Conference on Emerging Trends in Engineering and Technology, IEEE, 2008, pp. 314–319.

[50]

Srivastava N., Hinton G., Krizhevsky A., Sutskever I., Salakhutdinov R., Dropout: a simple way to prevent neural networks from overfitting, J. Mach. Learn. Res. 15 (1) (2014) 1929–1958.

Digital Library

[51]

H. Gao, Z. Wang, S. Ji, Large-scale learnable graph convolutional networks, in: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018, pp. 1416–1424.

[52]

Zhao H., Wei L., Yao Q., Simplifying architecture search for graph neural network, 2020, arXiv preprint arXiv:2008.11652.

[53]

Zhao Y., Wang D., Gao X., Mullins R., Lio P., Jamnik M., Probabilistic dual network architecture search on graphs, 2020, arXiv preprint arXiv:2003.09676.

[54]

Li Y., King I., AutoGraph: Automated graph neural network, in: International Conference on Neural Information Processing, Springer, 2020, pp. 189–201.

[55]

Nunes M., Pappa G.L., Neural architecture search in graph neural networks, in: Brazilian Conference on Intelligent Systems, Springer, 2020, pp. 302–317.

Cited By

Fan RWang WChu XTsafrir DMusuvathi MGupta RAbu-Ghazaleh N(2024)DTC-SpMM: Bridging the Gap in Accelerating General Sparse Matrix Multiplication with Tensor CoresProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 310.1145/3620666.3651378(253-267)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3620666.3651378
Odema MBouzidi HOuarnoughi HNiar SAl Faruque M(2023)MaGNAS: A Mapping-Aware Graph Neural Architecture Search Framework for Heterogeneous MPSoC DeploymentACM Transactions on Embedded Computing Systems10.1145/360938622:5s(1-26)Online publication date: 31-Oct-2023
https://dl.acm.org/doi/10.1145/3609386
Peng BDing YXia QYang Y(2023)Recurrent neural networks integrate multiple graph operators for spatial time series predictionApplied Intelligence10.1007/s10489-023-04632-253:21(26067-26078)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1007/s10489-023-04632-2
Show More Cited By

Index Terms

Genetic-GNN: Evolutionary architecture search for Graph Neural Networks
1. Computing methodologies
  1. Artificial intelligence
    1. Search methodologies
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Theory of computation

Index terms have been assigned to the content through auto-classification.

Recommendations

EGNN: Graph structure learning based on evolutionary computation helps more in graph neural networks
Abstract
In recent years, graph neural networks (GNNs) have been successfully applied in many fields due to their characteristics of neighborhood aggregation and have achieved state-of-the-art performance. While most GNNs process graph data, ...
Highlights
- A GNN model parameter evolution problem is studied within an evolutionary framework. The evolution process adheres to the principle of survival of the ...
Fitness landscape analysis of graph neural network architecture search spaces
GECCO '21: Proceedings of the Genetic and Evolutionary Computation Conference

Neural Architecture Search (NAS) is the name given to a set of methods designed to automatically configure the layout of neural networks. Their success on Convolutional Neural Networks inspired its use on optimizing other types of neural network ...
AKE-GNN: Effective Graph Learning with Adaptive Knowledge Exchange
CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

Graph Neural Networks (GNNs) have already been widely used in various graph mining tasks. However, recent works reveal that the learned weights (channels) in well-trained GNNs are highly redundant, which inevitably limits the performance of GNNs. Instead ...

Comments

Information & Contributors

Information

Published In

cover image Knowledge-Based Systems

Knowledge-Based Systems Volume 247, Issue C

Jul 2022

449 pages

ISSN:0950-7051

Issue’s Table of Contents

Elsevier B.V.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 08 July 2022

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Fan RWang WChu XTsafrir DMusuvathi MGupta RAbu-Ghazaleh N(2024)DTC-SpMM: Bridging the Gap in Accelerating General Sparse Matrix Multiplication with Tensor CoresProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 310.1145/3620666.3651378(253-267)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3620666.3651378
Odema MBouzidi HOuarnoughi HNiar SAl Faruque M(2023)MaGNAS: A Mapping-Aware Graph Neural Architecture Search Framework for Heterogeneous MPSoC DeploymentACM Transactions on Embedded Computing Systems10.1145/360938622:5s(1-26)Online publication date: 31-Oct-2023
https://dl.acm.org/doi/10.1145/3609386
Peng BDing YXia QYang Y(2023)Recurrent neural networks integrate multiple graph operators for spatial time series predictionApplied Intelligence10.1007/s10489-023-04632-253:21(26067-26078)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1007/s10489-023-04632-2
Liu YLiu JLi Y(2023)Automatic search of architecture and hyperparameters of graph convolutional networks for node classificationApplied Intelligence10.1007/s10489-022-04096-w53:9(11104-11119)Online publication date: 1-May-2023
https://dl.acm.org/doi/10.1007/s10489-022-04096-w

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents