Article

Bayesian regression with input noise for high dimensional data

Authors:

Stefan SchaalAuthors Info & Claims

ICML '06: Proceedings of the 23rd international conference on Machine learning

Pages 937 - 944

https://doi.org/10.1145/1143844.1143962

Published: 25 June 2006 Publication History

Abstract

This paper examines high dimensional regression with noise-contaminated input and output data. Goals of such learning problems include optimal prediction with noiseless query points and optimal system identification. As a first step, we focus on linear regression methods, since these can be easily cast into nonlinear learning problems with locally weighted learning approaches. Standard linear regression algorithms generate biased regression estimates if input noise is present and suffer numerically when the data contains redundancy and irrelevancy. Inspired by Factor Analysis Regression, we develop a variational Bayesian algorithm that is robust to ill-conditioned data, automatically detects relevant features, and identifies input and output noise -- all in a computationally efficient way. We demonstrate the effectiveness of our techniques on synthetic data and on a system identification task for a rigid body dynamics model of a robotic vision head. Our algorithm performs 10 to 70% better than previously suggested methods.

References

[1]

An, C. H., Atkeson, C. G., & Hollerbach, J. M. (1988). Model-based control of a robot manipulator. MIT Press.

Digital Library

[2]

Atkeson, C. G., Moore, A., & Schaal, S. (1997). Locally weighted learning. In Artificial intelligence review, vol. 11, 11--73. Kluwer.

Digital Library

[3]

Dempster, A., Laird, N., & Rubin, D. (1977). Maximum likelihood from incomplete data via the em algorithm. Journal of Royal Statistical Society. Series B, 39, 1--38.

[4]

Derksen, S., & Keselman, H. (1992). Backward, forward and stepwise automated subset selection algorithms: Frequency of obtaining authentic and noise variables. British Journal of Mathematical and Statistical Psychology, 45, 265--282.

[5]

Draper, N. R., & Smith, H. (1981). Applied regression analysis. Wiley.

[6]

D'Souza, A., Vijayakumar, S., & Schaal, S. (2004). The bayesian backfitting relevance vector machine. In Proceedings of the 21st international conference on machine learning. ACM Press.

Digital Library

[7]

Ghahramani, Z., & Beal, M. (2000). Graphical models and variational methods. In D. Saad and M. Opper (Eds.), Advanced mean field methods - theory and practice. MIT Press.

[8]

Golub, G. H., & Van Loan, C. (198). Matrix computations. John Hopkins University Press.

[9]

Hastie, T. J., & Tibshirani, R. J. (1990). Generalized additive models. No. 43 in Monographs on Statistics and Applied Probability. Chapman and Hall.

[10]

Hollerbach, J. M., & Wampler, C. W. (1996). The calibration index and the role of input noise in robot calibration. In G. Giralt and G. Hirzinger (Eds.), Robotics research: The seventh international symposium, 558--568. Springer.

[11]

Massey, W. (1965). Principal component regression in exploratory statistical research. Journal of the American Statistical Association, 60, 234--246.

[12]

Neal, R. (1994). Bayesian learning for neural networks. Doctoral dissertation, Dept. of Computer Science, University of Toronto.

Digital Library

[13]

Rao, Y. N., & Principe, J. (2002). Efficient total least squares method for system modeling using minor component analysis. In Proceedings of international workshop on neural networks for signal processing, 259--268. IEEE.

[14]

Strassen, V. (1969). Gaussian elimination is not optimal. Num Mathematik, 13, 354--356.

Digital Library

[15]

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of Royal Statistical Society, Series B, 58, 267--288.

[16]

Van Huffel, S., & Vanderwalle, J. (1991). The total least squares problem: Computational aspects and analysis. Society for Industrial and Applied Mathematics.

[17]

Wold, H. (1975). Soft modeling by latent variables: The nonlinear iterative partial least squares approach. In J. Gani (Ed.), Perspectives in probability and statistics, papers in honor of s. m. bartlett. Academic Press.

Cited By

Cheng MWu YHuang C(2020)Hybrid Gaussian Process Inference Model for Construction Management Decision MakingInternational Journal of Information Technology & Decision Making10.1142/S021962202050021219:04(1015-1036)Online publication date: 17-Jul-2020
https://doi.org/10.1142/S0219622020500212
Zhou HWang XSu J(2013)A General Self-Adaptive Reputation System Based on the Kalman FeedbackProceedings of the 2013 International Conference on Service Sciences10.1109/ICSS.2013.28(7-12)Online publication date: 11-Apr-2013
https://dl.acm.org/doi/10.1109/ICSS.2013.28
Wang XSu JHu XWu CZhou H(2013)Trust Model for Cloud Systems with Self Variance EvaluationSecurity, Privacy and Trust in Cloud Systems10.1007/978-3-642-38586-5_10(283-309)Online publication date: 4-Sep-2013
https://doi.org/10.1007/978-3-642-38586-5_10
Show More Cited By

Index Terms

Bayesian regression with input noise for high dimensional data
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Factorization methods
        Canonical correlation analysis
2. Mathematics of computing
  1. Probability and statistics
    1. Statistical paradigms
      1. Regression analysis
      2. Statistical graphics

Recommendations

Scalable subspace logistic regression models for high dimensional data
APWeb'12: Proceedings of the 14th Asia-Pacific international conference on Web Technologies and Applications

Although massive, high dimensional data in the real world provide more information for logistic regression classification, yet it also means a huge challenge for us to build models accurately and efficiently. In this paper, we propose a scalable ...
Enhanced algorithm for high-dimensional data classification

Graphical abstractIllustration of the decision hyperplanes generated by TSSVM, MCVSVM, and LMLP on an artificial dataset. Display Omitted HighlightsIn the case of the singularity of the within-class scatter matrix, the drawbacks of both MCVSVM and LMLP ...
Bayesian Shrinkage Methods for High Dimensional Regression

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICML '06: Proceedings of the 23rd international conference on Machine learning

June 2006

1154 pages

ISBN:1595933832

DOI:10.1145/1143844

Program Chairs:
William Cohen,
Andrew Moore

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 June 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Acceptance Rates

ICML '06 Paper Acceptance Rate 140 of 548 submissions, 26%;

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
587
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cheng MWu YHuang C(2020)Hybrid Gaussian Process Inference Model for Construction Management Decision MakingInternational Journal of Information Technology & Decision Making10.1142/S021962202050021219:04(1015-1036)Online publication date: 17-Jul-2020
https://doi.org/10.1142/S0219622020500212
Zhou HWang XSu J(2013)A General Self-Adaptive Reputation System Based on the Kalman FeedbackProceedings of the 2013 International Conference on Service Sciences10.1109/ICSS.2013.28(7-12)Online publication date: 11-Apr-2013
https://dl.acm.org/doi/10.1109/ICSS.2013.28
Wang XSu JHu XWu CZhou H(2013)Trust Model for Cloud Systems with Self Variance EvaluationSecurity, Privacy and Trust in Cloud Systems10.1007/978-3-642-38586-5_10(283-309)Online publication date: 4-Sep-2013
https://doi.org/10.1007/978-3-642-38586-5_10
Wang XLiu LSu J(2012)RLMIEEE Transactions on Services Computing10.1109/TSC.2010.565:1(131-143)Online publication date: 1-Jan-2012
https://dl.acm.org/doi/10.1109/TSC.2010.56
Ristovski KVucetic SObradovic Z(2012)Uncertainty Analysis of Neural-Network-Based Aerosol RetrievalIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2011.216612050:2(409-414)Online publication date: Feb-2012
https://doi.org/10.1109/TGRS.2011.2166120
Mordohai PMedioni G(2010)Dimensionality Estimation, Manifold Learning and Function Approximation using Tensor VotingThe Journal of Machine Learning Research10.5555/1756006.175601811(411-450)Online publication date: 1-Mar-2010
https://dl.acm.org/doi/10.5555/1756006.1756018
Ting JD'Souza AVijayakumar SSchaal S(2010)Efficient learning and feature selection in high-dimensional regressionNeural Computation10.1162/neco.2009.02-08-70222:4(831-886)Online publication date: 1-Apr-2010
https://dl.acm.org/doi/10.1162/neco.2009.02-08-702
Chen BKifer DLeFevre KMachanavajjhala A(2009)Privacy-Preserving Data PublishingFoundations and Trends in Databases10.1561/19000000082:1–2(1-167)Online publication date: 1-Jan-2009
https://dl.acm.org/doi/10.1561/1900000008
Wang XOu WSu J(2009)A reputation inference model based on linear hidden markov process2009 ISECS International Colloquium on Computing, Communication, Control, and Management10.1109/CCCM.2009.5270424(354-357)Online publication date: Aug-2009
https://doi.org/10.1109/CCCM.2009.5270424
Ting JTheodorou ESchaal S(2007)A Kalman filter for robust outlier detection2007 IEEE/RSJ International Conference on Intelligent Robots and Systems10.1109/IROS.2007.4399158(1514-1519)Online publication date: Oct-2007
https://doi.org/10.1109/IROS.2007.4399158
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents