article

Multiplicative Updates for Nonnegative Quadratic Programming

Authors:

Lawrence K. Saul,

Daniel D. LeeAuthors Info & Claims

Neural Computation, Volume 19, Issue 8

Pages 2004 - 2031

https://doi.org/10.1162/neco.2007.19.8.2004

Published: 01 August 2007 Publication History

Abstract

Many problems in neural computation and statistical learning involve optimizations with nonnegativity constraints. In this article, we study convex problems in quadratic programming where the optimization is confined to an axis-aligned region in the nonnegative orthant. For these problems, we derive multiplicative updates that improve the value of the objective function at each iteration and converge monotonically to the global minimum. The updates have a simple closed form and do not involve any heuristics or free parameters that must be tuned to ensure convergence. Despite their simplicity, they differ strikingly in form from other multiplicative updates used in machine learning. We provide complete proofs of convergence for these updates and describe their application to problems in signal processing and pattern recognition.

References

[1]

Allen, J. B., & Berkley, D. A. (1979). Image method for efficiently simulating small-room acoustics. Journal of the Acoustical Society of America, 65, 943-950.

[2]

Bauer, E., Koller, D., & Singer, Y. (1997). Update rules for parameter estimation in Bayesian networks. In Proceedings of the Thirteenth Annual Conference on Uncertainty in AI (pp. 3-13). San Francisco: Morgan Kaufmann.

[3]

Bertsekas, D. P. (1999). Nonlinear programming (2nd ed.). Belmont, MA: Athena Scientific.

[4]

Cristianini, N., & Shawe-Taylor, J. (2000). An introduction to support vector machines. Cambridge: Cambridge University Press.

[5]

Darroch, J. N., & Ratcliff, D. (1972). Generalized iterative scaling for log-linear models. Annals of Mathematical Statistics, 43, 1470-1480.

[6]

Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society B, 39, 1-38.

[7]

Diego, J. M., Tegmark, M., Protopapas, P., & Sandvik, H. B. (2007). Combined reconstruction of weak and strong lensing data with WSLAP. Monthly Notices of the Royal Astronomical Society, 375, 958-970.

[8]

Friess, T., Cristianini, N., & Campbell, C. (1998). The Kernel-Adatron algorithm: A fast and simple learning procedure for support vector machines. In Proceedings of the Fifteenth International Conference on Machine Learning (pp. 188-196). San Francisco: Morgan Kaufmann.

[9]

Kivinen, J., & Warmuth, M. (1997). Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, 132, 1-63.

Digital Library

[10]

Lee, D. D., & Seung, H. S. (1999). Learning the parts of objects with nonnegative matrix factorization. Nature, 401, 788-791.

[11]

Lee, D. D., & Seung, H. S. (2001). Algorithms for non-negative matrix factorization. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems, 13 (pp. 556-562). Cambridge, MA: MIT Press.

[12]

Lin, Y., Lee, D. D., & Saul, L. K. (2004). Nonnegative deconvolution for time of arrival estimation. In Proceedings of the International Conference of Speech, Acoustics, and Signal Processing (ICASSP-2004) (Vol. 2, pp. 377-380). Piscataway, NJ: IEEE.

[13]

Platt, J. (1999). Fast training of support vector machines using sequential minimal optimization. In B. Schölkopf, C. J. C. Burges, & A. J. Smola (Eds.), Advances in kernel methods--Support vector learning (pp. 185-208). Cambridge, MA: MIT Press.

Digital Library

[14]

Saul, L. K., Sha, F., & Lee, D. D. (2003). Statistical signal processing with nonnegativity constraints. In Proceedings of the Eighth European Conference on Speech Communication and Technology (Vol. 2, pp. 1001-1004). Geneva, Switzerland.

[15]

Schölkopf, B., Sung, K., Burges, C., Girosi, E, Niyogi, P., Poggio, T., & Vapnik, V. (1997). Comparing support vector machines with gaussian kernels to radial basis function classiers. IEEE Transactions on Signal Processing, 45, 2758-2765.

Digital Library

[16]

Serafini, T., Zanghirati, G., & Zanni, L. (2005). Gradient projection methods for quadratic programs and applications in training support vector machines. Optimization Methods and Software, 20, 353-378.

[17]

Sha, F., Saul, L. K., & Lee, D. D. (2003a). Multiplicative updates for nonnegative quadratic programming in support vector machines. In S. Becker, S. Thrun, & K. Obermayer (Eds.), Advances in neural information processing systems, 15 (pp. 897- 904). Cambridge, MA: MIT Press.

[18]

Sha, F., Saul, L. K., & Lee, D. D. (2003b). Multiplicative updates for large margin classifiers. In Proceedings of the Sixteenth Annual Conference on Computational Learning Theory (COLT-03) (pp. 188-202). Berlin: Springer.

[19]

Vapnik, V. (1998). Statistical learning theory. New York: Wiley.

[20]

Wright, S. J. (1997). Primal-dual interior point methods. Philadelphia, PA: SIAM.

[21]

Zangwill, W. J. (1969). Nonlinear programming: A unified approach. Englewood Cliffs, NJ: Prentice Hall.

Cited By

Tan PTan ZJiang YZhou Z(2024)Towards enabling learnware to handle heterogeneous feature spacesMachine Language10.1007/s10994-022-06245-1113:4(1839-1860)Online publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1007/s10994-022-06245-1
Ding YPeng QSong ZChen H(2023)Variable selection and regularization via arbitrary rectangle-range generalized elastic netStatistics and Computing10.1007/s11222-023-10240-433:3Online publication date: 26-Apr-2023
https://dl.acm.org/doi/10.1007/s11222-023-10240-4
Ma SZhang LHu WZhang YWu JLi X(2018)Self-representative manifold concept factorization with adaptive neighbors for clusteringProceedings of the 27th International Joint Conference on Artificial Intelligence10.5555/3304889.3305013(2539-2545)Online publication date: 13-Jul-2018
https://dl.acm.org/doi/10.5555/3304889.3305013
Show More Cited By

Index Terms

Multiplicative Updates for Nonnegative Quadratic Programming

Recommendations

Mixed-integer quadratic programming

This paper considers mixed-integer quadratic programs in which the objective function is quadratic in the integer and in the continuous variables, and the constraints are linear in the variables of both types. The generalized Benders' decomposition is a ...
Conic approximation to nonconvex quadratic programming with convex quadratic constraints

In this paper, a conic reformulation and approximation is proposed for solving a nonconvex quadratic programming problem subject to several convex quadratic constraints. The original problem is transformed into a linear conic programming problem, which ...
Nonconvex quadratically constrained quadratic programming: best D.C. decompositions and their SDP representations

We propose in this paper a general D.C. decomposition scheme for constructing SDP relaxation formulations for a class of nonconvex quadratic programs with a nonconvex quadratic objective function and convex quadratic constraints. More specifically, we ...

Comments

Information & Contributors

Information

Published In

cover image Neural Computation

Neural Computation Volume 19, Issue 8

August 2007

295 pages

ISSN:0899-7667

Issue’s Table of Contents

Publisher

MIT Press

Cambridge, MA, United States

Publication History

Published: 01 August 2007

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

39
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Tan PTan ZJiang YZhou Z(2024)Towards enabling learnware to handle heterogeneous feature spacesMachine Language10.1007/s10994-022-06245-1113:4(1839-1860)Online publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1007/s10994-022-06245-1
Ding YPeng QSong ZChen H(2023)Variable selection and regularization via arbitrary rectangle-range generalized elastic netStatistics and Computing10.1007/s11222-023-10240-433:3Online publication date: 26-Apr-2023
https://dl.acm.org/doi/10.1007/s11222-023-10240-4
Ma SZhang LHu WZhang YWu JLi X(2018)Self-representative manifold concept factorization with adaptive neighbors for clusteringProceedings of the 27th International Joint Conference on Artificial Intelligence10.5555/3304889.3305013(2539-2545)Online publication date: 13-Jul-2018
https://dl.acm.org/doi/10.5555/3304889.3305013
Dessein APapadakis NRouas J(2018)Regularized optimal transport and the rot mover's distanceThe Journal of Machine Learning Research10.5555/3291125.329114019:1(590-642)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.5555/3291125.3291140
Zhan KShi JWang JWang HXie Y(2018)Adaptive structure concept factorization for multiview clusteringNeural Computation10.1162/neco_a_0105530:4(1080-1103)Online publication date: 1-Apr-2018
https://dl.acm.org/doi/10.1162/neco_a_01055
Ye JJin Z(2018)Hyper-graph regularized discriminative concept factorization for data representationSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-017-2636-122:13(4417-4429)Online publication date: 1-Jul-2018
https://dl.acm.org/doi/10.1007/s00500-017-2636-1
Xu LZhang CSingh SMarkovitch S(2017)Bridging video content and commentsProceedings of the Thirty-First AAAI Conference on Artificial Intelligence10.5555/3298239.3298473(1611-1617)Online publication date: 4-Feb-2017
https://dl.acm.org/doi/10.5555/3298239.3298473
Li JWang F(2017)Towards Unsupervised Gene SelectionIEEE/ACM Transactions on Computational Biology and Bioinformatics10.1109/TCBB.2016.259154514:3(514-521)Online publication date: 1-May-2017
https://dl.acm.org/doi/10.1109/TCBB.2016.2591545
Ye JJin Z(2017)Graph-Regularized Local Coordinate Concept Factorization for Image RepresentationNeural Processing Letters10.1007/s11063-017-9598-246:2(427-449)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1007/s11063-017-9598-2
Iosifidis AGabbouj M(2016)Multi-class Support Vector Machine classifiers using intrinsic and penalty graphsPattern Recognition10.1016/j.patcog.2016.02.00255:C(231-246)Online publication date: 1-Jul-2016
https://dl.acm.org/doi/10.1016/j.patcog.2016.02.002
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents