Exploiting damped techniques for nonlinear conjugate gradient methods

Abstract

In this paper we propose the use of damped techniques within Nonlinear Conjugate Gradient (NCG) methods. Damped techniques were introduced by Powell and recently reproposed by Al-Baali and till now, only applied in the framework of quasi-Newton methods. We extend their use to NCG methods in large scale unconstrained optimization, aiming at possibly improving the efficiency and the robustness of the latter methods, especially when solving difficult problems. We consider both unpreconditioned and Preconditioned NCG. In the latter case, we embed damped techniques within a class of preconditioners based on quasi-Newton updates. Our purpose is to possibly provide efficient preconditioners which approximate, in some sense, the inverse of the Hessian matrix, while still preserving information provided by the secant equation or some of its modifications. The results of an extensive numerical experience highlights that the proposed approach is quite promising.

Appendix

In this appendix, we report a technical result used in the proofs of Proposition 1 (see Grippo and Sciandrone 2011).

Lemma 1

Let $\{\xi _k\}$ be a sequence of nonnegative real numbers. Let $\varOmega >0$ and $q \in (0,1)$ and suppose that there exists $k_1 \ge 1$ such that

$$\begin{aligned} \xi _k \le \varOmega + q \xi _{k-1}, \qquad \hbox {for any}\qquad k \ge k_1. \end{aligned}$$

Then,

$$\begin{aligned} \xi _k \le \frac{\varOmega }{1-q} + \left( \xi _{k_1} - \frac{\varOmega }{1-q}\right) q^{k-k_1}, \qquad \hbox {for any}\qquad k \ge k_1. \end{aligned}$$

Proof

Starting from relation

$$\begin{aligned} \xi _k \le \varOmega + q \xi _{k-1}, \qquad \hbox {for any}\qquad k \ge k_1, \end{aligned}$$

considering $k-k_1$ iterations we get:

$$\begin{aligned} \xi _k \le \varOmega \left( \sum _{i=0}^{k-k_1-1}q^i \right) + q^{k-k_1}\xi _{k_1}, \end{aligned}$$

from which we obtain

$$\begin{aligned} \xi _k \le \varOmega \Big (\frac{1-q^{k-k_1}}{1-q}\Big )+q^{k-k_1} \xi _{k_1} = \frac{\varOmega }{1-q} + \left( \xi _{k_1} - \frac{\varOmega }{1-q} \right) q^{k-k_1}. \end{aligned}$$

$\square$