Abstract
It is well known that the norm of the gradient may be unreliable as a stopping test in unconstrained optimization, and that it often exhibits oscillations in the course of the optimization. In this paper we present results descibing the properties of the gradient norm for the steepest descent method applied to quadratic objective functions. We also make some general observations that apply to nonlinear problems, relating the gradient norm, the objective function value, and the path generated by the iterates.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
H. Akaike, “On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method,” Ann. Inst. Stat. Math. Tokyo, vol. 11, pp. 1–16, 1959.
J. Barzilai and J.M. Borwein, “Two-point step size gradient methods,” IMA Journal of Numerical Analysis, vol. 8, pp. 141–148, 1988.
M.S. Bazaraa, H.D. Sherali, and C.M. Shetty, Nonlinear Programming, 2nd edn., John Wiley & Sons: New York, 1993.
I. Bongartz, A.R. Conn, N.I.M. Gould, and Ph.L. Toint, “CUTE: Constrained and unconstrained testing environment,” ACM Transactions on Mathematical Software, vol. 21, no. 1, pp. 123–160, 1995.
R.H. Byrd, P. Lu, J. Nocedal, and C. Zhu, “A limited memory algorithm for bound constrained optimization,” SIAM Journal on Scientific Computing, vol. 16, no. 5, pp. 1190–1208, 1995.
R. Byrd, J. Nocedal, and C. Zhu, “Towards a discrete Newton method with memory for large-scale optimization,” in Nonlinear Optimization and Applications. G. Di Pillo and F. Giannessi (Eds.), Plenum: New York, 1996.
A.R. Conn, N.I.M. Gould, and Ph.L. Toint, “LANCELOT: A FORTRAN package for large-scale nonlinear optimization (Release A),” Number 17 in Springer Series in Computational Mathematics, Springer-Verlag: New York, 1992.
G.E. Forsythe, “On the asymptotic directions of the s-dimensional optimum gradient method,” Numerische Mathematik, vol. 11, pp. 57–76, 1968.
J.Ch. Gilbert, Private communication, 1994.
P.E. Gill, W. Murray, and M.H. Wright, Practical Optimization, Academic Press: London, 1981.
D.G. Luenberger, Linear and Nonlinear Programming, 2nd edn., Addison-Wesley: Reading, MA, 1984.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Nocedal, J., Sartenaer, A. & Zhu, C. On the Behavior of the Gradient Norm in the Steepest Descent Method. Computational Optimization and Applications 22, 5–35 (2002). https://doi.org/10.1023/A:1014897230089
Issue Date:
DOI: https://doi.org/10.1023/A:1014897230089