Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3579654.3579738acmotherconferencesArticle/Chapter ViewAbstractPublication PagesacaiConference Proceedingsconference-collections
research-article

Adversarial Training on Weights for Graph Neural Networks

Published: 14 March 2023 Publication History

Abstract

Despite the fact that Graph Neural Networks (GNNs) have been extensively used for graph embedding representation, it is challenging to train well-performing GNNs on graphs with good generalization due to the limitation of overfitting. Previous research in Computer Vision (CV) has shown that the lack of generalization usually corresponds to the convergence of model parameters to sharp local minima. However, there is still a lack of related research in the field of graph analysis. In this paper, we investigate the loss landscape of models from the weight change perspective and show that the vanilla training method tends to cause GNNs to fall into sharp local minima with poor generalization. To tackle this problem, we propose a method named Adversarial Training on Weights (ATW) to flatten the weight loss landscape using adversarial training, thus improving the generalization of GNNs. Extensive experiments with multiple backbones on various datasets demonstrate the effectiveness of our method.

References

[1]
Anish Athalye, Nicholas Carlini, and David A. Wagner. 2018. Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. In ICML(Proceedings of Machine Learning Research, Vol. 80). PMLR, 274–283.
[2]
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2014. Spectral Networks and Locally Connected Networks on Graphs. In ICLR.
[3]
Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. 2021. When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations. CoRR abs/2106.01548(2021).
[4]
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. 2017. Sharp Minima Can Generalize For Deep Nets. In ICML(Proceedings of Machine Learning Research, Vol. 70). PMLR, 1019–1028.
[5]
David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gómez-Bombarelli, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. 2015. Convolutional Networks on Graphs for Learning Molecular Fingerprints. In NIPS. 2224–2232.
[6]
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2021. Sharpness-aware Minimization for Efficiently Improving Generalization. In ICLR. OpenReview.net.
[7]
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In ICLR (Poster).
[8]
William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive Representation Learning on Large Graphs. In NIPS. 1024–1034.
[9]
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yong-Dong Zhang, and Meng Wang. 2020. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In SIGIR. ACM, 639–648.
[10]
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020. Open Graph Benchmark: Datasets for Machine Learning on Graphs. In NeurIPS.
[11]
Xiao Huang, Jundong Li, and Xia Hu. 2017. Label Informed Attributed Network Embedding. In the Tenth ACM International Conference.
[12]
Alexander G. Ororbia II, Daniel Kifer, and C. Lee Giles. 2017. Unifying Adversarial Training Algorithms with Data Gradient Regularization. Neural Comput. 29, 4 (2017), 867–887.
[13]
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. 2020. Fantastic Generalization Measures and Where to Find Them. In ICLR. OpenReview.net.
[14]
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. 2017. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. In ICLR. OpenReview.net.
[15]
Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR (Poster). OpenReview.net.
[16]
Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, and Tom Goldstein. 2020. FLAG: Adversarial Data Augmentation for Graph Neural Networks. CoRR abs/2010.09891(2020).
[17]
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. Visualizing the Loss Landscape of Neural Nets. In Neural Information Processing Systems.
[18]
Chen Ma, Liheng Ma, Yingxue Zhang, Jianing Sun, Xue Liu, and Mark Coates. 2020. Memory Augmented Graph Neural Networks for Sequential Recommendation. In AAAI. AAAI Press, 5045–5052.
[19]
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards Deep Learning Models Resistant to Adversarial Attacks. In ICLR (Poster). OpenReview.net.
[20]
Takeru Miyato, Andrew M. Dai, and Ian J. Goodfellow. 2017. Adversarial Training Methods for Semi-Supervised Text Classification. In ICLR (Poster). OpenReview.net.
[21]
Andrew Slavin Ross and Finale Doshi-Velez. 2018. Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing Their Input Gradients. In AAAI. AAAI Press, 1660–1669.
[22]
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John P. Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. 2019. Adversarial training for free!. In NeurIPS. 3353–3364.
[23]
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In ICLR (Poster).
[24]
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. In ICLR (Poster). OpenReview.net.
[25]
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. 2020. Adversarial Weight Perturbation Helps Robust Generalization. In NeurIPS.
[26]
Weilin Xu, David Evans, and Yanjun Qi. 2018. Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks. In NDSS. The Internet Society.
[27]
Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. 2016. Revisiting Semi-Supervised Learning with Graph Embeddings. In ICML(JMLR Workshop and Conference Proceedings, Vol. 48). JMLR.org, 40–48.
[28]
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor K. Prasanna. 2020. GraphSAINT: Graph Sampling Based Inductive Learning Method. In ICLR. OpenReview.net.
[29]
Kaixiong Zhou, Ninghao Liu, Fan Yang, Zirui Liu, Rui Chen, Li Li, Soo-Hyun Choi, and Xia Hu. 2021. Adaptive Label Smoothing To Regularize Large-Scale Graph Training. arXiv preprint arXiv:2108.13555(2021).

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ACAI '22: Proceedings of the 2022 5th International Conference on Algorithms, Computing and Artificial Intelligence
December 2022
770 pages
ISBN:9781450398336
DOI:10.1145/3579654
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 March 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. adversarial training
  2. graph neural networks
  3. node classification

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • The National Natural Science Foundation of China
  • The Science and Technology Development Program of Jilin Province
  • The Interdisciplinary and Integrated Innovation of JLU

Conference

ACAI 2022

Acceptance Rates

Overall Acceptance Rate 173 of 395 submissions, 44%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 39
    Total Downloads
  • Downloads (Last 12 months)11
  • Downloads (Last 6 weeks)0
Reflects downloads up to 31 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media