Abstract
In this paper, a new stochastic quasi-Newton method (SQN) is proposed which has a different approximation of the Hessian inverse matrix \(H_k\). The modified quasi-Newton Broyden–Fletcher–Goldfarb–Shanno (BFGS) formula which has a better approximation to Hessian matrix has not only the gradient variation but also the function value. Because of the special nature of the sum function, the mini-batch setting is built in the algorithm, and less compution cost can be guaranteed. The number of iterations reduce to at most \(O(\varepsilon ^{-\frac{1}{1-\beta }})\). The convergence analysis is established in this paper. The numerical experiments show that this algorithm is competitive to other algorithms.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’, pp. 177–186 (2010)
Byrd, R.H., Hansen, S.L., Nocedal, J., et al.: A stochastic quasi-Newton method for large-scale optimization. SIAM J. Optim. 26, 1008–1031 (2012)
Cotter, A., Shamir, O., Srebro, N., Sridharan, K.: Better mini-batch algorithms via accelerated gradient methods. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 1647–1655 (2011)
Durrett, R.: Probability: Theory and Examples. Cambridge University Press, London (2010)
Fercoq, O., RichtSrik, P.: Accelerated, parallel, and proximal coordinate descent. SIAM J. Optim. 25, 1997–2023 (2015)
Ghadimi, S., Lan, G.: Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM J. Optim. 23, 2341–2368 (2013)
Hassan, B., Mohammed, T.: A new variants of quasi-newton equation based on the quadratic function for unconstrained optimization. Indones. J. Electr. Eng. Comput. Sci. 2, 701–708 (2020)
Lan, G.: An optimal method for stochastic convex optimization. Technical report, Georgia Institute of Technology (2009)
Li, M., Zhang, T., Chen, Y., et al.: Efficient mini-batch training for stochastic optimization. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 661–670 (2014)
Mason, L., Baxter, J., Bartlett, P., Frean, M.: Boosting algorithms as gradient descent in function space. Adv. Neural Inf. Process. Syst. 12, 512–518 (1999)
Mokhtari, A., Eisen, M., Ribeiro, A.: IQN: An incremental quasi-Newton method with local superlinear convergence rate. SIAM J. Optim. 28, 1670–1698 (2018)
Nemirovski, A., Juditsky, A., Lan, G., Shapiro, A.: Robust stochastic approximation approach to stochastic programming. SIAM J. Optim. 19(4), 1574–1609 (2009)
Powell, M.J.D.: Some global convergence properties of a variable metric algorithm for minimization without exact line searches. In: Cottle, R.W., Lemke, C.E. (eds.) Nonlinear Programming, Philadelphia (1976)
Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 22, 400–407 (1951)
Schmidt, M., Roux, N., Bach, F.: Minimizing finite sums with the stochastic average gr1adient. Math. Program. 162, 83–112 (2017)
Shang, F., Zhou, K., Liu, H., et al.: VR-SGD: a simple stochastic variance reduction method for machine learning. IEEE Trans. Knowl. Data Eng. 32, 188–202 (2018)
Wang, X., Ma, S., Goldfarb, D., Liu, W.: Stochastic quasi-Newton methods for nonconvex stochastic optimization. SIAM J. Optim. 27, 927–956 (2017)
Wei, Z., Yu, G., Yuan, G., et al.: The superlinear convergence of a modified BFGS-type method for unconstrained optimization. Comput. Optim. Appl. 29, 315–332 (2004)
Wei, Z., Yu, G., Yuan, G., Lian, Z.: The superlinear convergence of a modified BFGS-type method for unconstrained optimization. Comput. Optim. Appl. 29, 315–332 (2004)
Yang, Z., Wang, C., Zang, Y., et al.: Mini-batch algorithms with Barzilai–Borwein update step. Neurocomputing 314, 177–185 (2018)
Yuan, G., Wei, Z.: Convergence analysis of a modified BFGS method on convex minimizations. Comput. Optim. Appl. 47, 237–255 (2010)
Yuan, G., Zhou, Y., Wang, L., et al.: Stochastic bigger subspace algorithms for nonconvex stochastic optimization. IEEE Access 9, 119818–119829 (2021)
Zhang, J.Z., Deng, N.Y., Chen, L.H.: New quasi-Newton equation and related methods for unconstrained optimization. J. Optim. Theory Appl. 102, 147–167 (1999)
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare to have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supported by the Special Funds for Local Science and Technology Development Guided by the Central Government (No. ZY20198003), the High Level Innovation Teams and Excellent Scholars Program in Guangxi institutions of higher education (Grant No. [2019]52), the Guangxi Natural Science Key Fund (No. 2017GXNSFDA198046), and the National Natural Science Foundation of China (Grant No. 11661009)
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Chen, X., Feng, H. A modified stochastic quasi-Newton algorithm for summing functions problem in machine learning. J. Appl. Math. Comput. 69, 1491–1506 (2023). https://doi.org/10.1007/s12190-022-01800-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12190-022-01800-4