Article

On-line prediction with kernels and the complexity approximation principle

Authors:

Alex Gammerman,

Yuri Kalnishkan,

Vladimir VovkAuthors Info & Claims

UAI '04: Proceedings of the 20th conference on Uncertainty in artificial intelligence

Pages 170 - 176

Published: 07 July 2004 Publication History

Abstract

The paper describes an application of Aggregating Algorithm to the problem of regression. It generalizes earlier results concerned with plain linear regression to kernel techniques and presents an on-line algorithm which performs nearly as well as any oblivious kernel predictor. The paper contains the derivation of an estimate on the performance of this algorithm. The estimate is then used to derive an application of the Complexity Approximation Principle to kernel methods.

References

[1]

{Aro50} N. Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society., 68:337--404, 1950.

[2]

{CBFH+97} N. Cesa-Bianchi, Y. Freund, D. Haussler, D. P. Helmbold, R. E. Schapire, and M. K. Warmuth. How to use expert advice. Journal of the ACM, 44(3):427--485, 1997.

Digital Library

[3]

{CST00} Nello Cristianini and John Shaw-Taylor. An Introduction to Support Vector Machines. Cambridge University Press, 2000.

[4]

{HS81} H. V. Henderson and S. R. Searle. On deriving the inverse of a sum of matrices. SIAM Review, 23(1), 1981.

Digital Library

[5]

{Kre63} M. Krein. Hermitian-positive kernels on homogeneous spaces. American Mathematical Society Translations: Series 2, 34:69--164, 1963.

[6]

{LV97} M. Li and P. Vitányi. An Introduction to Kolmogorov Complexity and Its Applications. Springer, New York, 1997.

Digital Library

[7]

{LW94} N. Littlestone and M. K. Warmuth. The weighted majority algorithm. Information and Computation, 108:212--261, 1994.

Digital Library

[8]

{PTVF94} William H. Press Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery. Numerical Recipies in C. Cambridge University Press, 2nd edition, 1994.

[9]

{SGV98} C. Saunders, A. Gammerman, and V. Vovk. Ridge regression learning algorithm in dual variables. In Proceedings of the 15th International Conference on Machine Learning, pages 515--521, 1998.

Digital Library

[10]

{SS02} B. Schölkopf and A. J. Smola. Learning with Kernels. MIT Press, 2002.

[11]

{VG99} V. Vovk and A. Gammerman. Complexity approximation principle. The Computer Journal, 42(4):318 322, 1999.

[12]

{Vov90} V. Vovk. Aggregating strategies. In M. Fulk and J. Case, editors, Proceedings of the 3rd Annual Workshop on Computational Learning Theory, pages 371--383, San Mateo, CA, 1990. Morgan Kaufmann.

Digital Library

[13]

{Vov98} V. Vovk. Competitive on-line linear regression. In Michael I. Jordan, Michael J. Kearns, and Sara A. Solla, editors, Advances in Neural Information Processing Systems, volume 10. The MIT Press, 1998.

Digital Library

[14]

{Vov01a} V. Vovk. Competitive on-line statistics. International Statistical Review, 69(2):213--248, 2001.

[15]

{Vov01b} V. Vovk. Probability theory for the Brier game. Theoretical Computer Science, 261:57--79, 2001.

Digital Library

[16]

{VW98} V. Vovk and C. J. H. C. Watkins. Universal portfolio selection. In Proceedings of the 11th Annual Conference on Computational Learning Theory, pages 12--23, 1998.

Digital Library

[17]

{Wah90} G. Wahba. Spline Models for Observational Data. SIAM, Philadelphia, 1990.

Cited By

Li JLiao SKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Nearly optimal algorithms with sublinear computational complexity for online kernel regressionProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619223(19743-19766)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3619223
Yang PLi PDemartini GZuccon GCulpepper JHuang ZTong H(2021)Adversarial Kernel Sampling on Class-imbalanced Data StreamsProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482227(2352-2362)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3459637.3482227
Calandriello DLazaric AValko M(2017)Second-order kernel online convex optimization with adaptive sketchingProceedings of the 34th International Conference on Machine Learning - Volume 7010.5555/3305381.3305448(645-653)Online publication date: 6-Aug-2017
https://dl.acm.org/doi/10.5555/3305381.3305448
Show More Cited By

Index Terms

On-line prediction with kernels and the complexity approximation principle

Index terms have been assigned to the content through auto-classification.

Recommendations

Refinement of operator-valued reproducing kernels

This paper studies the construction of a refinement kernel for a given operator-valued reproducing kernel such that the vector-valued reproducing kernel Hilbert space of the refinement kernel contains that of the given kernel as a subspace. The study is ...
Refinement of operator-valued reproducing kernels

This paper studies the construction of a refinement kernel for a given operator-valued reproducing kernel such that the vector-valued reproducing kernel Hilbert space of the refinement kernel contains that of the given kernel as a subspace. The study is ...
Positive Kernels and Stochastic Integrals

Let $(\Omega ,\mathcal{F},\mathbb{P})$ be a probability space with a Brownian motion $\{ \beta (t):t \geqq 0\} $. Let S be the space of functions $a(t) \in C^1 [0,\infty )$ such that $e^{ - \lambda t} a(t) \in L^1 (0,\infty )$ for all $\lambda > 0$. A ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

UAI '04: Proceedings of the 20th conference on Uncertainty in artificial intelligence

July 2004

657 pages

ISBN:0974903906

Conference Chair:
Christopher Meek
Microsoft Research
,
Program Chairs:
Max Chickering
Microsoft Reasearch
,
Joseph Halpern
Cornell University

Sponsors

Alberta Ingenuity Centre for Machine Learning
Sun Microsystems of Canada
Hewlett-Packard Laboratories
Information Extraction and Transportation
Informatics Circle of Research Excellence
Yahoo! Research Labs
IBMR: IBM Research
Intel: Intel
Microsoft Research: Microsoft Research
Pacific Institute of Mathematical Sciences
Boeing
University of Alberta: University of Alberta
Northrop Grumman Corporation

Publisher

AUAI Press

Arlington, Virginia, United States

Publication History

Published: 07 July 2004

Check for updates

Qualifiers

Article

Conference

UAI '04

Sponsor:

IBMR
Intel
Microsoft Research
University of Alberta

UAI '04: Uncertainty in artificial intelligence

July 7 - 11, 2004

Banff, Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
161
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li JLiao SKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Nearly optimal algorithms with sublinear computational complexity for online kernel regressionProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619223(19743-19766)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3619223
Yang PLi PDemartini GZuccon GCulpepper JHuang ZTong H(2021)Adversarial Kernel Sampling on Class-imbalanced Data StreamsProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482227(2352-2362)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3459637.3482227
Calandriello DLazaric AValko M(2017)Second-order kernel online convex optimization with adaptive sketchingProceedings of the 34th International Conference on Machine Learning - Volume 7010.5555/3305381.3305448(645-653)Online publication date: 6-Aug-2017
https://dl.acm.org/doi/10.5555/3305381.3305448
Chervonenkis AChernova SZykova T(2011)Applications of kernel ridge estimation to the problem of computing the aerodynamical characteristics of a passenger plane (in comparison with results obtained with artificial neural networks)Automation and Remote Control10.1134/S000511791105013472:5(1061-1067)Online publication date: 1-May-2011
https://dl.acm.org/doi/10.1134/S0005117911050134
Chernov AZhdanov F(2010)Prediction with expert advice under discounted lossProceedings of the 21st international conference on Algorithmic learning theory10.5555/1893193.1893222(255-269)Online publication date: 6-Oct-2010
https://dl.acm.org/doi/10.5555/1893193.1893222
Busuttil SKalnishkan Y(2007)Weighted Kernel Regression for Predicting Changing DependenciesProceedings of the 18th European conference on Machine Learning10.1007/978-3-540-74958-5_50(535-542)Online publication date: 17-Sep-2007
https://dl.acm.org/doi/10.1007/978-3-540-74958-5_50
Dani VMadani OPennock DSanghai SGalebach B(2006)An empirical comparison of algorithms for aggregating expert predictionsProceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence10.5555/3020419.3020433(106-113)Online publication date: 13-Jul-2006
https://dl.acm.org/doi/10.5555/3020419.3020433
Vovk V(2006)On-Line regression competitive with reproducing kernel hilbert spacesProceedings of the Third international conference on Theory and Applications of Models of Computation10.1007/11750321_43(452-463)Online publication date: 15-May-2006
https://dl.acm.org/doi/10.1007/11750321_43

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten