research-article

Variational Tobit Gaussian Process Regression

Authors:

Tobias M. Louw,

Theresa R. SmithAuthors Info & Claims

Statistics and Computing, Volume 33, Issue 3

https://doi.org/10.1007/s11222-023-10225-3

Published: 31 March 2023 Publication History

Abstract

We propose a variational inference-based framework for training a Gaussian process regression model subject to censored observational data. Data censoring is a typical problem encountered during the data gathering procedure and requires specialized techniques to perform inference since the resulting probabilistic models are typically analytically intractable. In this article we exploit the variational sparse Gaussian process inducing variable framework and local variational methods to compute an analytically tractable lower bound on the true log marginal likelihood of the probabilistic model which can be used to perform Bayesian model training and inference. We demonstrate the proposed framework on synthetically-produced, noise-corrupted observational data, as well as on a real-world data set, subject to artificial censoring. The resulting predictions are comparable to existing methods to account for data censoring, but provides a significant reduction in computational cost.

References

[1]

Alaa, A.M., van der Schaar, M.: Deep multi-task Gaussian processes for survival analysis with competing risks. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 2326–2334 (2017)

[2]

Allik B, Miller C, Piovoso MJ, et al. The Tobit Kalman filter: an estimator for censored measurements IEEE Trans. Control Syst. Technol. 2016 24 1 365-371

[3]

Amemiya T Tobit models: a survey J. Econom. 1984 24 1 3-61

[4]

Barrett JE and Coolen ACC Covariate dimension reduction for survival data via the Gaussian process latent variable model Stat. Med. 2016 35 8 1340-1353

[5]

Bishop CM Pattern Recognition and Machine Learning, Information science and statistics 2009 New York Springer

[6]

Blei DM, Kucukelbir A, and McAuliffe JD Variational inference: a review for statisticians J. Am. Stat. Assoc. 2017 112 518 859-877

[7]

Bui, T., Turner, R.: On the paper: Variational Learning of Inducing Variables in Sparse Gaussian Processes (Titsias, 2009). https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.716.942 &rep=rep1 &type=pdf (2014)

[8]

Campbell KR and Yau C Bayesian Gaussian process latent variable models for pseudotime inference in single-cell RNA-seq data bioRxiv 2015 2015 1-6

[9]

Chen N, Qian Z, Nabney IT, et al. Wind power forecasts using Gaussian processes and numerical weather prediction IEEE Trans. Power Syst. 2013 29 2 656-665

[10]

Csató L and Opper M Sparse on-line Gaussian processes Neural Comput. 2002 14 3 641-668

[11]

Damianou, A., Lawrence, N.D.: Deep Gaussian processes. In: Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, pp. 207–215 (2013)

[12]

Damianou AC, Titsias MK, and Lawrence ND Variational inference for latent variables and uncertain inputs in Gaussian processes J. Mach. Learn. Res. 2016 17 42 1-62

[13]

DWS: National Water Management System data extracted on 2019-07-17. Department of Water and Sanitation. www.dwa.gov.za/iwqs/wms/data/C_reg_WMS_nobor.htm (2019)

[14]

EPA: Indicators: Conductivity. United States Environmental Protection Agency. www.epa.gov/national-aquatic-resource-surveys/indicators-conductivity (2022)

[15]

Ertin, E.: Gaussian process models for censored sensor readings. In: 2007 IEEE/SP 14th Workshop on Statistical Signal Processing, pp. 665–669 (2007)

[16]

Galy-Fajou, T., Opper, M.: Adaptive inducing points selection For Gaussian processes. arXiv preprint arXiv:2107.10066v1 (2021)

[17]

Gammelli D, Peled I, Rodrigues F, et al. Estimating latent demand of shared mobility through censored Gaussian Processes Transp. Res. Part C Emerg. Technol. 2020 120 102775

[18]

Gammelli, D., Rolsted, K.P., Pacino, D., et al.: Generalized Multi-Output Gaussian Process Censored Regression. arXiv preprint arXiv:2009.04822v2 (2020b)

[19]

Gibbs MN and MacKay DJ Variational Gaussian process classifiers IEEE Trans. Neural Netw. 2000 11 6 1458-1464

[20]

Groot, P., Lucas, P.: Gaussian Process Regression with Censored Data Using Expectation Propagation. In: Proceedings of the 6th European Workshop on Probabilistic Graphical Models, pp. 115–122 (2012)

[21]

Hensman, J., Fusi, N., Lawrence, N.D.: Gaussian processes for big data. In: Uncertainty in Artificial Intelligence—Proceedings of the 29th Conference, UAI 2013, pp. 282–290 (2013)

[22]

Hensman, J., Matthews, A., Ghahramani, Z.: Scalable variational Gaussian process classification. In: Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, pp. 351–360 (2015) http://proceedings.mlr.press/v38/hensman15.pdf

[23]

Hoffman MD, Blei DM, Wang C, et al. Stochastic variational inference J. Mach. Learn. Res. 2013 14 1303-1347

[24]

Hutter, F., Hoos, H., Leyton-Brown, K.: Bayesian Optimization With Censored Response Data. arXiv preprint arXiv:1310.1947v1 (2013)

[25]

Jordan MI, Ghahramani Z, Jaakkola TS, et al. An introduction to variational methods for graphical models Mach. Learn. 1999 37 2 183-233

[26]

Lawrence, N.D.: Gaussian process latent variable models for visualisation of high dimensional data. In: Advances in Neural Information Processing Systems 16 (2004)

[27]

Lawrence, N.D.: Learning for Larger Datasets with the Gaussian Process Latent Variable Model. In: Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, pp. 243–250 (2007)

[28]

Lázaro-Gredilla, M.: Bayesian Warped Gaussian Processes. In: Advances in Neural Information Processing Systems (2012)

[29]

Lázaro-Gredilla M, Quiñnero-Candela J, Rasmussen CE, et al. Sparse Spectrum Gaussian Process Regression J. Mach. Learn. Res. 2010 11 63 1865-1881

[30]

Li, A.H., Bradic, J.: Censored quantile regression forest. In: International Conference on Artificial Intelligence and Statistics, pp. 2109–2119 (2020)

[31]

Logan JD Applied Mathematics 2006 3 New York Wiley

[32]

MacKay DJC Information Theory, Inference, and Learning Algorithms 2004 Cambridge Cambridge University Press

[33]

Minka, T.P.: Expectation Propagation for approximate Bayesian inference. In: Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, pp. 362–369 (2001a)

[34]

Minka, T.P.: A family of algorithms for approximate Bayesian inference. PhD thesis, Massachusetts Institute of Technology (2001b)

[35]

Nickisch H and Rasmussen CE Approximations for binary gaussian process classification J. Mach. Learn. Res. 2008 9 67 2035-2078

[36]

Pishro-Nik H Introduction to Probability, Statistics, and Random Processes 2014 London Kappa Research LLC

[37]

Quiñonero-Candela J and Rasmussen CE A unifying view of sparse approximate gaussian process regression J. Mach. Learn. Res. 2005 6 65 1939-1959

[38]

Rao, A., Monteiro, J., Mourao-Miranda, J.: Prediction of clinical scores from neuroimaging data with censored likelihood Gaussian processes. In: 2016 International Workshop on Pattern Recognition in Neuroimaging (PRNI), pp. 1–4 (2016).

[39]

Rasmussen CE and Williams CK Gaussian Processes for Machine Learning, Adaptive Computation and Machine Learning 2006 Cambridge MIT Press

[40]

Saul, A.D., Hensman, J., Vehtari, A., et al.: Chained Gaussian Processes. In: Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, pp. 1431–1440 (2016).

[41]

Seeger, M., Williams, C. K.I., Lawrence, N.D.: Fast Forward Selection to Speed Up Sparse Gaussian Process Regression. In: Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics (2003)

[42]

Snelson, E., Ghahramani, Z.: Sparse Gaussian processes using pseudo-inputs. In: Advances in Neural Information Processing Systems vol. 18 (2005)

[43]

Snelson E, Rasmussen CE, and Ghahramani Z Warped Gaussian processes Adv. Neural. Inf. Process. Syst. 2004 16 337-344

[44]

Titsias, M.: Variational Model Selection for Sparse Gaussian Process Regression. Tech. rep., University of Manchester, UK https://www2.aueb.gr/users/mtitsias/papers/sparseGPv2.pdf (2008)

[45]

Titsias, M.: Variational Learning of Inducing Variables in Sparse Gaussian Processes. In: Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, pp. 567–574 (2009)

[46]

Titsias, M., Lawrence, N.D.: Bayesian Gaussian process latent variable model. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 844–851 (2010)

[47]

Tobin J Estimation of relationships for limited dependent variables Econometrica 1958 26 1 24-36

[48]

Uhrenholt, A.K., Charvet, V., Jensen, B.S.: Probabilistic selection of inducing points in sparse Gaussian processes. In: Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, pp. 1035–1044 (2021).

[49]

Urtasun, R., Fleet, D.J., Fua, P.: 3D people tracking with Gaussian process dynamical models. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 238–245 (2006).

[50]

Vanhatalo J, Riihimäki J, Hartikainen J, et al. GPstuff: Bayesian Modeling with Gaussian Processes J. Mach. Learn. Res. 2013 14 1175-1179

[51]

Wang JM, Fleet DJ, and Hertzmann A Gaussian process dynamical models for human motion IEEE Trans. Pattern Anal. Mach. Intell. 2008 30 2 283-298

[52]

Wu, W. C.-H., Yeh, M.-Y., Chen, M.-S.: Deep Censored Learning of the Winning Price in the Real Time Bidding. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining pp. 2526–2535 (2018).

[53]

Zhang, J., Zhu, Z., Zou, J.: Supervised Gaussian process latent variable model based on Gaussian mixture model. In: 2017 International Conference on Security, Pattern Analysis, and Cybernetics, pp. 124–129 (2017).

Cited By

O’Neill E(2024)Type I Tobit Bayesian Additive Regression Trees for censored outcome regressionStatistics and Computing10.1007/s11222-024-10434-434:4Online publication date: 24-May-2024
https://dl.acm.org/doi/10.1007/s11222-024-10434-4
Dănăilă VBuiu C(2024)A deep learning approach to censored regressionPattern Analysis & Applications10.1007/s10044-024-01216-927:1Online publication date: 28-Feb-2024
https://dl.acm.org/doi/10.1007/s10044-024-01216-9

Recommendations

Generalized multi-output Gaussian process censored regression
Highlights
- Censored data as defining characteristic of numerous domains in science.
- Heteroscedastic Multi-Output Gaussian Process formulated for censored regression.
- Generalization of arbitrary likelihood functions enabled by devising a ...
Abstract
When modelling censored observations (i.e. data in which the value of a measurement or observation is un-observable beyond a given threshold), a typical approach in current regression methods is to use a censored-Gaussian (i.e. Tobit) model to ...
Variational Inference on Infinite Mixtures of Inverse Gaussian, Multinomial Probit and Exponential Regression
ICMLA '14: Proceedings of the 2014 13th International Conference on Machine Learning and Applications

We introduce a new class of methods and inference techniques for infinite mixtures of Inverse Gaussian, Multinomial Probit and Exponential Regression, models that belong to the widely applicable framework of Generalized Linear Model (GLM). We ...
Variational inference in nonconjugate models

Mean-field variational methods are widely used for approximate posterior inference in many probabilistic models. In a typical application, mean-field methods approximately compute the posterior with a coordinate-ascent optimization algorithm. When the ...

Comments

Information & Contributors

Information

Published In

cover image Statistics and Computing

Statistics and Computing Volume 33, Issue 3

Jun 2023

359 pages

ISSN:0960-3174

Issue’s Table of Contents

© The Author(s) 2023.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 31 March 2023

Accepted: 21 February 2023

Received: 01 September 2022

Author Tags

Qualifiers

Research-article

Funding Sources

School of Data Science and Computational Thinking, Stellenbosch University
Engineering and Physical Sciences Research Council

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

O’Neill E(2024)Type I Tobit Bayesian Additive Regression Trees for censored outcome regressionStatistics and Computing10.1007/s11222-024-10434-434:4Online publication date: 24-May-2024
https://dl.acm.org/doi/10.1007/s11222-024-10434-4
Dănăilă VBuiu C(2024)A deep learning approach to censored regressionPattern Analysis & Applications10.1007/s10044-024-01216-927:1Online publication date: 28-Feb-2024
https://dl.acm.org/doi/10.1007/s10044-024-01216-9

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents