Abstract
A popular and classical method for finding the best rank one approximation of a real tensor is the higher order power method (HOPM). It is known in the literature that the iterative sequence generated by HOPM converges globally, while the convergence rate can be superlinear, linear or sublinear. In this paper, we examine the convergence rate of HOPM in solving the best rank one approximation problem of real tensors. We first show that the iterative sequence of HOPM always converges globally and provide an explicit eventual sublinear convergence rate. The sublinear convergence rate estimate is in terms of the dimension and the order of the underlying tensor space. Then, we examine the concept of nondegenerate singular vector tuples and show that, if the sequence of HOPM converges to a nondegenerate singular vector tuple, then the global convergence rate is R-linear. We show that, for almost all tensors (in the sense of Lebesgue measure), all the singular vector tuples are nondegenerate, and so, the HOPM “typically” exhibits global R-linear convergence rate. Moreover, without any regularity assumption, we establish that the sequence generated by HOPM always converges globally and R-linearly for orthogonally decomposable tensors with order at least 3. We achieved this by showing that each nonzero singular vector tuple of an orthogonally decomposable tensor with order at least 3 is nondegenerate.
Similar content being viewed by others
Notes
Here, a property holds for almost all tensors means that the set of tensors for which the property does not hold is a set of Lebesgue measure zero.
In [34], this notion was referred as completely orthogonally decomposable tensors.
While finalizing a first version of this work, Professor Bernd Sturmfels kindly pointed out to the authors that the total number of nonzero real singular vector tuples as well as the characterization of the set of nonzero singular vector tuples for an orthogonally decomposable tensor, has also been recently derived in [53] using algebraic geometry tools. It is worth noting that our derivation is more elementary. Moreover, our main concern here is the nondegeneracy, which is not considered in [53].
References
Ammar, A., Chinesta, F., Falcó, A.: On the convergence of a greedy rank-one update algorithm for a class of linear systems. Arch. Comput. Methods Eng. 17, 473–486 (2010)
Anandkumar, A., Ge, R., Hsu, D., Kakade, S.M., Telgarsky, M.: Tensor decompositions for learning latent variable models. J. Mach. Learn. Res. 15, 2773–2832 (2014)
Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka–Łojasiewicz inequality. Math. Oper. Res. 35, 438–457 (2010)
Beck, A.: First-order methods in optimization. In: MOSSIAM Series on Optimization, 25, xii+475 pp. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA; Mathematical Optimization Society, Philadelphia (2017)
Bolte, J., Daniilidis, A., Ley, O., Mazet, L.: Characterizations of Łojasiewicz inequalities and applications: subgradient flows, talweg, convexity. Trans. Am. Math. Soc. 362, 3319–3363 (2010)
Boothby, W.M.: An Introduction to Differentiable Manifolds and Riemannian Geometry. Academic Press, New York (1975)
Bott, R., Tu, L.W.: Differential Forms in Algebraic Topology. Springer, New York (1982)
Bro, R.: PARAFAC, tutorial and applications. Chemom. Intell. Lab. Syst. 38, 149–171 (1997)
Cardoso, J.-F., Comon, P.: Independent component analysis, a survey of some algebraic methods. In: The IEEE International Symposium on Circuits and Systems, vol. 2, pp. 93–96. IEEE, New York (1996)
Cardoso, J.-F., Souloumiac, A.: Jacobi angles for simultaneous diagonalization. SIAM J. Matrix Anal. Appl. 17, 161–164 (1996)
Carroll, J.D., Pruzansky, S.: The CANDECOMP-CANDELINC family of models and methods for multidimensional data analysis. In: Law, H.G., Snyder, C.W., Hattie, J.A., McDonald, R.P. (eds.) Research Methods for Multimode Data Analysis, pp. 372–402. Praeger, New York (1984)
Comon, P.: MA identification using fourth order cumulants. Signal Process. 26, 381–388 (1992)
Comon, P.: Independent component analysis, a new concept? Signal Process. 36, 287–314 (1994)
Comon, P., Mourrain, B.: Decomposition of quantics in sums of powers of linear forms. Signal Process. 53, 93–107 (1996)
D’ Acunto, D., Kurdyka, K.: Explicit bounds for the Łojasiewicz exponent in the gradient inequality for polynomials. Ann. Polon. Math 87, 51–61 (2005)
De Lathauwer, L.: Signal processing based on multilinear algebra. Ph.D. thesis, Katholieke Universiteit Leuven, Leuven, Belgium (1997)
De Lathauwer, L., De Moor, B., Vandewalle, J.: A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl. 21, 1253–1278 (2000)
De Lathauwer, L., De Moor, B., Vandewalle, J.: On the best rank-1 and rank-\((R_1, R_2,\dots, R_N)\) approximation of higher-order tensors. SIAM J. Matrix Anal. Appl. 21, 1324–1342 (2000)
De Lathauwer, L., Comon, P., De Moor, B., Vandewalle, J.: Higher-order power method: application in independent component analysis. In: Proceedings of the International Symposium on Nonlinear Theory and its Applications (NOLTA’95), pp. 91–96. Las Vegas, NV (1995)
de Silva, V., Lim, L.-H.: Tensor rank and the ill-posedness of the best low-rank approximation problem. SIAM J. Matrix Anal. Appl. 30, 1084–1127 (2008)
do Carmo, M.P.: Riemannian Geometry. Springer, Berlin (1992)
Edelman, A., Arias, T., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20, 303–353 (1998)
Espig, M., Hackbusch, W., Khachatryan, A.: On the convergence of alternating least squares optimization in tensor format representations (2015). arXiv:1506.00062v1
Espig, M., Khachatryan, A.: Convergence of alternating least squares optimization for rank-one approximation to higher order tensors (2015). arXiv:1503.05431
Frankel, P., Garrigos, G., Peypouquet, J.: Splitting methods with variable metric for Kurdyka–Łojasiewicz functions and general convergence rates. J. Optim. Theory Appl. 165, 874–900 (2015)
Friedland, S., Ottaviani, G.: The number of singular vector tuples and uniqueness of best rank-one approximation of tensors. Found. Comput. Math. 14, 1209–1242 (2014)
Gelfand, I.M., Kapranov, M.M., Zelevinsky, A.V.: Discriminants, Resultants and Multidimensional Determinants. Birkhäuser, Boston (1994)
Golub, G.H., Van Loan, C.F.: Matrix Computations, 4th edn. Johns Hopkins University Press, Baltimore (2013)
Grasedyck, L., Kressner, D., Tobler, C.: A literature survey of low-rank tensor approximation techniques. GAMM-Mitteilungen 36, 53–78 (2013)
Hackbusch, W.: Tensor Spaces and Numerical Tensor Calculus. Springer, Berlin (2012)
Hillar, C.J., Lim, L.-H.: Most tensor problems are NP-hard. J. ACM 60(45), 1–39 (2013)
Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press, New York (1985)
Kofidis, E., Regalia, P.: On the best rank-1 approximation of higher-order supersymmetric tensors. SIAM J. Matrix Anal. Appl. 23, 863–884 (2002)
Kolda, T.G.: Orthogonal tensor decompositions. SIAM J. Matrix Anal. Appl. 23, 243–255 (2001)
Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51, 455–500 (2009)
Kroonenberg, P.M.: Three-Mode Principal Component Analysis. DSWO Press, Leiden (1983)
Landsberg, J.M.: Tensors: Geometry and Applications. AMS, Providence (2012)
Lang, S.: Algebra, 3rd edn. Springer, Berlin (2004)
Li, G., Mordukhovich, B.S., Pham, T.S.: New fractional error bounds for polynomial systems with applications to Hölderian stability in optimization and spectral theory of tensors. Math. Program. 153, 333–362 (2015)
Li, G., Mordukhovich, B.S., Nghia, T.T.A., Pham, T.S.: Error bounds for parametric polynomial systems with applications to higher-order stability analysis and convergence rates. Math. Program. 168, 313–346 (2018)
Li, G., Pong, T.K.: Calculus of the exponent of Kurdyka–Łojasiewicz inequality and its applications to linear convergence of first-order methods. Found. Comput. Math. (2017). https://doi.org/10.1007/s10208-017-9366-8
Li, G., Qi, L.Q., Yu, G.: Semismoothness of the maximum eigenvalue function of a symmetric tensor and its application. Linear Algebra Appl. 438, 813–833 (2013)
Lim, L.-H.: Singular values and eigenvalues of tensors: a variational approach. In Proceedings of the 1st IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing, pp. 129–132 (2005)
Łojasiewicz, S.: Une propriété topologique des sous-ensembles analytiques réels, Les Équations aux Dérivées Partielles. Éditions du centre National de la Recherche Scientifique, Paris, pp. 87–89 (1963)
Mohlenkamp, M.J.: Musings on multilinear fitting. Linear Algebra Appl. 438, 834–852 (2013)
Morse, M.: The Calculus of Variations in the Large. American Mathematical Society, Providence (1934)
Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, Dordrecht (2004)
Nie, J., Wang, L.: Semidefinite relaxations for best rank-1 tensor approximations. SIAM J. Matrix Anal. Appl. 35, 1155–1179 (2014)
Ortega, J.M., Rheinboldt, W.C.: Iterative Solution of Nonlinear Equations in Several Variables. Springer, Berlin (1970)
Qi, L.: Eigenvalues of a real supersymmetric tensor. J. Symb. Comput. 40, 1302–1324 (2005)
Qi, L.: The best rank-one approximation ratio of a tensor space. SIAM J. Matrix Anal. Appl. 32, 430–442 (2011)
Robeva, E.: Orthogonal decomposition of symmetric tensors. SIAM J. Matrix Anal. Appl. 37, 86–102 (2016)
Robeva, E., Seigal, A.: Singular vectors of orthogonally decomposable tensors. Linear and Multilinear Algebra (2017). https://doi.org/10.1080/03081087.2016.1277508
Rockafellar, R.T., Wets, R.: Variational Analysis. Grundlehren der Mathematischen Wissenschaften, vol. 317. Springer, Berlin (1998)
Schneider, R., Uschmajew, A.: Convergence results for projected line-search methods on varieties of low-rank matrices via Łojasiewicz inequality. SIAM J. Optim. 25, 622–646 (2015)
Shafarevich, I.R.: Basic Algebraic Geometry. Springer, Berlin (1977)
Sommese, A.J., Wampler II, C.W.: The Numerical Solution of Systems of Polynomials Arising in Engineering and Science. World Scientific, Hackensack (2005)
Uschmajew, A.: Local convergence of the alternating least squares algorithm for canonical tensor approximation. SIAM J. Matrix Anal. Appl. 33, 639–652 (2012)
Uschmajew, A.: A new convergence proof for the high-order power method and generalizations. Pacific J. Optim. 11, 309–321 (2015)
Wang, L., Chu, M.: On the global convergence of the alternating least squares method for rank-one approximation to generic tensors. SIAM J. Matrix Anal. Appl. 23, 1058–1072 (2014)
Yang, Y., Feng, Y., Huang, X., Suykens, J.A.K.: Rank-1 tensor properties with applications to a class of tensor optimization problems. SIAM J. Optim. 26, 171–196 (2016)
Yang, Y., Feng, Y., Suykens, J.A.K.: A rank-one tensor updating algorithm for tensor completion. IEEE Signal Process. Lett. 22, 1633–1637 (2015)
Zhang, T., Golub, G.H.: Rank-one approximation to high order tensors. SIAM J. Matrix Anal. Appl. 23, 534–550 (2001)
Zhang, X., Ling, C., Qi, L.: The best rank-1 approximation of a symmetric tensor and related spherical optimization problems. SIAM J. Matrix Anal. Appl. 33, 806–821 (2012)
Acknowledgements
The authors are grateful to the referees and the editor for their constructive comments and helpful suggestions which have contributed to the final presentation of the paper. The authors would also like to thank Dr. Yang Qi (University of Chicago) for pointing out the reference [7]. The first author’s work is partially supported by National Science Foundation of China (Grant No. 11771328), Young Elite Scientists Sponsorship Program by Tianjin, and Innovation Research Foundation of Tianjin University (Grant Nos. 2017XZC-0084 and 2017XRG-0015). The second author is partially supported by a Future fellowship from Australian Research Council (FT130100038) and a discovery project from Australian Research Council (DP180100745).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hu, S., Li, G. Convergence rate analysis for the higher order power method in best rank one approximations of tensors. Numer. Math. 140, 993–1031 (2018). https://doi.org/10.1007/s00211-018-0981-3
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00211-018-0981-3