short-paper

Calibrate Automated Graph Neural Network via Hyperparameter Uncertainty

Authors:

Zhiqiang TaoAuthors Info & Claims

CIKM '22: Proceedings of the 31st ACM International Conference on Information & Knowledge Management

Pages 4640 - 4644

https://doi.org/10.1145/3511808.3557556

Published: 17 October 2022 Publication History

Abstract

Automated graph learning has drawn widespread research attention due to its great potential to reduce human efforts when dealing with graph data, among which hyperparameter optimization (HPO) is one of the mainstream directions and has made promising progress. However, how to obtain reliable and trustworthy prediction results with automated graph neural networks (GNN) is still quite underexplored. To this end, we investigate automated GNN calibration by marrying uncertainty estimation to the HPO problem. Specifically, we propose a hyperparameter uncertainty-induced graph convolutional network (HyperU-GCN) with a bilevel formulation, where the upper-level problem explicitly reasons uncertainties by developing a probabilistic hypernetworks through a variational Bayesian lens, while the lower-level problem learns how the GCN weights respond to a hyperparameter distribution. By squeezing model uncertainty into the hyperparameter space, the proposed HyperU-GCN could achieve calibrated predictions in a similar way to Bayesian model averaging over hyperparameters. Extensive experimental results on six public datasets were provided in terms of node classification accuracy and expected calibration error (ECE), demonstrating the effectiveness of our approach compared with several state-of-the-art uncertainty-aware and calibrated GCN methods.

References

[1]

James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of machine learning research, Vol. 13, 2 (2012).

[2]

Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015. Weight uncertainty in neural network. In International conference on machine learning. PMLR, 1613--1622.

[3]

Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. 2018. Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning. PMLR, 1184--1193.

[4]

Stefan Falkner, Aaron Klein, and Frank Hutter. 2018. BOHB: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learning. PMLR, 1437--1446.

[5]

Guosheng Feng, Chunnan Wang, and Hongzhi Wang. 2021. Search For Deep Graph Neural Networks. arXiv preprint arXiv:2109.10047 (2021).

[6]

Matthias Feurer and Frank Hutter. 2019. Hyperparameter optimization. In Automated machine learning. Springer, Cham, 3--33.

[7]

Tiago M Fragoso, Wesley Bertoli, and Francisco Louzada. 2018. Bayesian model averaging: A systematic review and conceptual classification. International Statistical Review, Vol. 86, 1 (2018), 1--28.

[8]

Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. 2017. Forward and reverse gradient-based hyperparameter optimization. In International Conference on Machine Learning. PMLR, 1165--1173.

[9]

Yarin Gal et al. 2016. Uncertainty in deep learning. (2016).

[10]

Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning. PMLR, 1050--1059.

[11]

Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. 2020. Graph Neural Architecture Search. In IJCAI, Vol. 20. 1403--1409.

[12]

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning. PMLR, 1321--1330.

[13]

David Ha, Andrew Dai, and Quoc V Le. 2016. Hypernetworks. arXiv preprint arXiv:1609.09106 (2016).

[14]

Matthew D Hoffman and Matthew J Johnson. 2016. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, Vol. 1.

[15]

Durk P Kingma, Tim Salimans, and Max Welling. 2015. Variational dropout and the local reparameterization trick. Advances in neural information processing systems, Vol. 28 (2015).

[16]

Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).

[17]

Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).

[18]

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, Vol. 30 (2017).

[19]

Guohao Li, Matthias Mü ller, Ali K. Thabet, and Bernard Ghanem. 2019. DeepGCNs: Can GCNs Go As Deep As CNNs?. In 2019 IEEE/CVF International Conference on Computer Vision. IEEE, 9266--9275.

[20]

Lisha Li, Kevin G Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2017. Hyperband: Bandit-based configuration evaluation for hyperparameter optimization. In ICLR (Poster).

[21]

Jonathan Lorraine and David Duvenaud. 2018. Stochastic hyperparameter optimization through hypernetworks. arXiv preprint arXiv:1802.09419 (2018).

[22]

David JC MacKay. 1992. A practical Bayesian framework for backpropagation networks. Neural computation, Vol. 4, 3 (1992), 448--472.

[23]

Matthew MacKay, Paul Vicol, Jon Lorraine, David Duvenaud, and Roger Grosse. 2019. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. arXiv preprint arXiv:1903.03088 (2019).

[24]

Andrey Malinin and Mark Gales. 2018. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, Vol. 31 (2018).

[25]

Radford M Neal. 2012. Bayesian learning for neural networks. Vol. 118. Springer Science & Business Media.

[26]

Manfred Opper and Cédric Archambeau. 2009. The variational Gaussian approximation revisited. Neural computation, Vol. 21, 3 (2009), 786--792.

[27]

John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, Vol. 10, 3 (1999), 61--74.

[28]

Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. 2019. Dropedge: Towards deep graph convolutional networks on node classification. arXiv preprint arXiv:1907.10903 (2019).

[29]

Jürgen Schmidhuber. 1992. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, Vol. 4, 1 (1992), 131--139.

Digital Library

[30]

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008. Collective classification in network data. AI magazine, Vol. 29, 3 (2008), 93--93.

[31]

Murat Sensoy, Lance Kaplan, and Melih Kandemir. 2018. Evidential deep learning to quantify classification uncertainty. Advances in Neural Information Processing Systems, Vol. 31 (2018).

[32]

Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. 2018. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868 (2018).

[33]

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, Vol. 15, 1 (2014), 1929--1958.

[34]

Zhiqiang Tao, Yaliang Li, Bolin Ding, Ce Zhang, Jingren Zhou, and Yun Fu. 2020. Learning to Mutate with Hypergradient Guided Population. In NeurIPS.

[35]

Petar Velivc ković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903 (2017).

[36]

Xiao Wang, Hongrui Liu, Chuan Shi, and Cheng Yang. 2021. Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration. Advances in Neural Information Processing Systems, Vol. 34 (2021).

[37]

Xin Wang, Ziwei Zhang, and Wenwu Zhu. 2022a. Automated Graph Machine Learning: Approaches, Libraries and Directions. CoRR, Vol. abs/2201.01288 (2022). showeprint[arXiv]2201.01288 https://arxiv.org/abs/2201.01288

[38]

Xin Wang, Ziwei Zhang, and Wenwu Zhu. 2022b. Automated Graph Machine Learning: Approaches, Libraries and Directions. arXiv preprint arXiv:2201.01288 (2022).

[39]

Bianca Zadrozny and Charles Elkan. 2001. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, Vol. 1. Citeseer, 609--616.

[40]

Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. 694--699.

Digital Library

[41]

Huan Zhao, Lanning Wei, and Quanming Yao. 2020b. Simplifying architecture search for graph neural network. arXiv preprint arXiv:2008.11652 (2020).

[42]

Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. 2020a. Uncertainty aware semi-supervised learning on graph data. Advances in Neural Information Processing Systems, Vol. 33 (2020), 12827--12836.

[43]

Kaixiong Zhou, Qingquan Song, Xiao Huang, and Xia Hu. 2019. Auto-gnn: Neural architecture search of graph neural networks. arXiv preprint arXiv:1909.03184 (2019).

[44]

Ronghang Zhu, Zhiqiang Tao, Yaliang Li, and Sheng Li. 2021. Automated graph learning via population based self-tuning GCN. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2096--2100.

Digital Library

Cited By

Sun GBai YYang XFang YFu YTao ZChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Aligning Out-of-Distribution Web Images and Caption Semantics via Evidential LearningProceedings of the ACM on Web Conference 202410.1145/3589334.3645653(2271-2281)Online publication date: 13-May-2024
https://doi.org/10.1145/3589334.3645653
Shi WYang XZhao XChen HTao ZLi SFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Calibrate Graph Neural Networks under Out-of-Distribution Nodes via Deep Q-learningProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614797(2270-2279)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614797
Liu YZhao SWang XGeng LXiao ZLin J(2023)Self-Consistent Graph Neural Networks for Semi-Supervised Node ClassificationIEEE Transactions on Big Data10.1109/TBDATA.2023.32665909:4(1186-1197)Online publication date: 1-Aug-2023
https://doi.org/10.1109/TBDATA.2023.3266590

Index Terms

Calibrate Automated Graph Neural Network via Hyperparameter Uncertainty
1. Computing methodologies
  1. Machine learning
    1. Machine learning algorithms
2. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

Semi-Supervised Graph Neural Network with Probabilistic Modeling to Mitigate Uncertainty
ICISDM '20: Proceedings of the 2020 the 4th International Conference on Information System and Data Mining

Probabilistic Graph Neural Network (PGNN) introduced here proposes a machine learning application that combines deep learning on graphs with probabilistic modeling, to account for uncertainty in both inputs and outputs. Graph neural network models (GNNs)...
Hyperspherical Weight Uncertainty in Neural Networks
Advances in Intelligent Data Analysis XIX
Abstract
Bayesian neural networks learn a posterior probability distribution over the weights of the network to estimate the uncertainty in predictions. Parameterization of prior and posterior distribution as Gaussian in Monte Carlo Dropout, Bayes-by-...
Signed Bipartite Graph Neural Networks
CIKM '21: Proceedings of the 30th ACM International Conference on Information & Knowledge Management

Signed networks are such social networks having both positive and negative links. A lot of theories and algorithms have been developed to model such networks (e.g., balance theory). However, previous work mainly focuses on the unipartite signed networks ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '22: Proceedings of the 31st ACM International Conference on Information & Knowledge Management

October 2022

5274 pages

ISBN:9781450392365

DOI:10.1145/3511808

General Chairs:
Mohammad Al Hasan
Indiana University Purdue University, Indianapolis, USA
,
Li Xiong
Emory University, Atlanta, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

CIKM '22

Sponsor:

CIKM '22: The 31st ACM International Conference on Information and Knowledge Management

October 17 - 21, 2022

GA, Atlanta, USA

Acceptance Rates

CIKM '22 Paper Acceptance Rate 621 of 2,257 submissions, 28%;

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
199
Total Downloads

Downloads (Last 12 months)58
Downloads (Last 6 weeks)3

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sun GBai YYang XFang YFu YTao ZChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Aligning Out-of-Distribution Web Images and Caption Semantics via Evidential LearningProceedings of the ACM on Web Conference 202410.1145/3589334.3645653(2271-2281)Online publication date: 13-May-2024
https://doi.org/10.1145/3589334.3645653
Shi WYang XZhao XChen HTao ZLi SFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Calibrate Graph Neural Networks under Out-of-Distribution Nodes via Deep Q-learningProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614797(2270-2279)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614797
Liu YZhao SWang XGeng LXiao ZLin J(2023)Self-Consistent Graph Neural Networks for Semi-Supervised Node ClassificationIEEE Transactions on Big Data10.1109/TBDATA.2023.32665909:4(1186-1197)Online publication date: 1-Aug-2023
https://doi.org/10.1109/TBDATA.2023.3266590

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents