Asia-Pacific Journal of Operational Research Vol. 28, No. 5 (2011) 585–600 c World Scientific Publishing Co. & Operational Research Society of Singapore
DOI: 10.1142/S0217595911003454 COMBINATION ADAPTIVE TRUST REGION METHOD BY NON-MONOTONE STRATEGY FOR UNCONSTRAINED NONLINEAR PROGRAMMING KEYVAN AMINI∗ and MASOUD AHOOKHOSH† Department of Sciences, Razi University Kermanshah, Iran ∗kamini@razi.ac.ir †ahoo.math@gmail.com In this paper, we present a new trust region method for unconstrained nonlinear programming in which we blend adaptive trust region algorithm by non-monotone strategy to propose a new non-monotone trust region algorithm with automatically adjusted radius. Both non-monotone strategy and adaptive technique can help us introduce a new algorithm that reduces the number of iterations and function evaluations. The new algorithm preserves the global convergence and has local superlinear and quadratic convergence under suitable conditions. Numerical experiments exhibit that the new trust region algorithm is very efficient and robust for unconstrained optimization problems. Keywords: Unconstrained optimization; trust region method; non-monotone technique; global convergence; superlinear convergence; quadratic convergence; second necessary condition. 1. Introduction Consider the following unconstrained optimization problem min f (x) x∈Rn (1.1) where f : Rn → R is a twice continuously differentiable function. There are many methods that were proposed to solve (1.1), but most of these methods are iterative methods. Trust region family are one of the most well-known methods for solving nonlinear programming of which concept has matured over than 30 years. In this family of methods, one considers a model mk of the objective function which is assumed to be adequate in neighborhood around the current iterate xk , namely trust region. This neighborhood is often represented by a ball in some norms of which radius δk is updated from iteration k to k + 1 by considering how well mk predicts the objective function in iterate xk+1 . Trust region methods try to find the area around the current iterate xk in which the quadratic model must have an agreement with objective function. In 585 586 K. Amini & M. Ahookhosh the standard trust region method, this agreement is measured by the following ratio: rk = f (xk ) − f (xk + dk ) mk (0) − mk (dk ) (1.2) the numerator of (1.2) is called the actual reduction and the denominator of (1.2) is called the predicted reduction; and dk is a trial step determined by solving the following sub-problem: 1 min mk (d) = gkT d + dT Bk d, 2 d∈Rn d ≤ δk (1.3) where  ·  is the Euclidean norm, gk = ∇f (xk ), Bk is a symmetric approximation of Hk = ∇2 f (xk ), and δk is trust region radius in kth iteration. If rk is close to 1, it can be concluded that there is a good agreement between the model and the objective function over this step, so this step is successful, otherwise the step is unsuccessful. There are two prominent factors in trust region method, initial trust region radius and ratio that measure the agreement of function and quadratic model. It is well known that the standard trust region method is very sensitive on initial radius, but in standard trust region method, δk is independent of gk and Bk . So we do not know if the radius δk is suitable for the whole iterations of algorithm. This situation possibly increases the number of solving subproblems in the inner iteration and so decreases the efficiency of the algorithm. A favorite goal for avoiding of this drawback is using the information of problem in current iterate to reduce the number of ineffective iterations. To overcome this drawback, Sartenear (1997) proposed a strategy to determine the initial trust region radius. Motivated by a neural network problem, Zhang et al. (2002) proposed another approach to determine the trust region radius. They set δk = ρp gk  · B̂ −1  where ρ ∈ (0, 1), p is a non-negative integer, and B̂ = Bk + iI is a positive definite matrix for some i ∈ N. Zhang’s approach was an efficient approach to solve unconstrained optimization problems. Shi and Guo (2008), motivated by Zhang’s strategy, introduced a new adaptive radius for trust region that is described in the next section. They proved that the new adaptive trust region algorithm has strong properties like global, superlinear and quadratic convergence. Their numerical experiments exhibited that the new trust region algorithm with automatically adjusted radius is very efficient. As we state, another important factor in trust region is the ratio that compares the agreement between function and quadratic model. It is well known that the pioneer in proposal of non-monotone strategies is Chamberlain et al. (1982), who proposed the watchdog technique for constrained optimization. Based on this idea, Grippo et al. (1986, 1989) introduced a non-monotone line search technique for Combination Adaptive Trust Region Method by Non-Monotone Strategy 587 Newton method. They relaxed Armijo rule such that stepsize θk is satisfied in the following condition: f (xk + θk dk ) ≤ fl(k) + βθk ∇f (xk )T dk (1.4) in which, β ∈ (0, 1) and fl(k) = max {fk−j }, 0≤j≤n(k) k = 0, 1, 2, · · · (1.5) with 0 ≤ n(k) ≤ min{n(k − 1) + 1, N }, in which N ≥ 0 is an integer constant. Motivated by this idea, many authors worked on combination trust region methods with non-monotone technique (see Deng et al., 1993; Toint, 1996, 1997 and Zhang et al., 2003). The basic idea of non-monotone trust region techniques is changing of the ratio (1.2). The most famous non-monotone ratio is as follows: r̃k = fl(k) − f (xk + dk ) mk (0) − mk (dk ) (1.6) in which, in comparison with (1.2), actual reduction is changed. In 2005, Fu and Sun proposed a new non-monotone ratio: r̂k = fl(k) − f (xk + dk ) fl(k) − fk − mk (dk ) (1.7) Indeed, in new definition, in comparison with (1.2), both numerator and dominator are changed. The actual reduction and the predicted reduction is defined as follows: aredk = fl(k) − f (xk + dk ) predk = fl(k) − fk − mk (dk ) (1.8) Numerical experiments exhibited that the non-monotone trust region algorithm was more efficient than the general monotone versions, especially in presence of the narrow curved valley. Pay attention to these attempts to reform trust region algorithm, Zhang et al. (2003) decided to combine a non-monotone strategy (1.6) with adaptive trust region method. Zhang’s non-monotone adaptive trust region algorithm is very efficient. Thus, Fu et al. (2005) proposed another non-monotone adaptive trust region algorithm in which the non-monotone technique (1.7) was combined with Zhang’s adaptive trust region radius. On the other hand, from Shi and Guo (2008), we know that the Shi’s adaptive trust region algorithm is more efficient than the Zhang’s adaptive trust region algorithm; so we decided to combine Shi’s adaptive radius with nonmonotone technique (1.7) to propose a new non-monotone trust region algorithm with automatically adjusted radius. The rest of this paper organized as follows. In Sec. 2, we describe our new nonmonotone adaptive trust region algorithm and give some properties about it. In Sec. 3, we prove the global convergence of the new algorithm. In Sec. 4, the local superlinear and quadratic convergence and second necessary condition are proved. Numerical results in Sec. 5 indicate that the new algorithm is very efficient and robustness. Finally, some conclusions and remarks are delivered in Sec. 6. 588 K. Amini & M. Ahookhosh 2. New Algorithm In this section, we will describe a new non-monotone trust region method with adaptive radius and give some properties about this algorithm. In the new algorithm, we combine the non-monotone strategy (1.7) with Shi’s adaptive trust region radius. Firstly, we describe Shi’s adaptive radius. Pay attention to Shi’s algorithm, we set µ ∈ (0, 1), ρ ∈ (0, 1), and suppose that qk satisfies the following relation − gkT qk ≥τ gk ·qk  (2.1) with τ ∈ (0, 1]. Now we define sk = − gkT qk (2.2) qkT B̂k qk in which B̂k = Bk + iI, and i is the smallest non-negative integer such that qkT B̂k qk > 0 (2.3) With these definitions, they proposed a trust region radius as δk = αk qk  in which αk is the largest α in {sk , ρsk , ρ2 sk , · · · } such that the trial step dk satisfies r̂k ≥ µ (2.4) p So there is a positive integer p such that δk = ρ sk qk . In the new algorithm, we use this radius only with this difference that qk is chosen satisfying in the following inequality qkT B̂k qk > λ0 qkT qk (2.5) Remark 2.1. Choosing qk satisfy in (2.5) is possible because λmin (B̂k ) ≤ qkT B̂k qk ≤ λmax (B̂k ) qkT qk So, it is enough that i is chosen such that λmin (B̂k ) > 0. Now, we can introduce new non-monotone trust region algorithm as follows: Algorithm 2.1: New non-monotone adaptive trust region algorithm. (1) Choose 0 < ρ < 1, 0 < µ < 1, ǫ > 0, x0 ∈ Rn , k = 0, a symmetric matrix B0 ∈ Rn×n , and N ≥ 0. (2) Compute gk . If gk  ≤ ǫ, stop. (3) Choose qk to satisfy (2.1) and set α = sk that sk is defined in (2.2). (4) Solve (1.3) to determine dk and set x̄k+1 = xk + dk . (5) Calculate n(k) and fl(k) , and compute r̂k by (1.7). If r̂k < µ, then let α = ρα and go to Step 4. (6) Let xk+1 = x̄k+1 , p = 0, update Bk+1 , k = k + 1 and go to Step 2. Combination Adaptive Trust Region Method by Non-Monotone Strategy 589 Note that similar to Chamberlain et al. (1982), we solve the quadratic subproblem (1.3) inaccurately by Newton method, such that the following relation holds:
gk  fl(k) − fk − mk (dk ) ≥ βgk  min αk , Bk   (2.6) and it is well-known if sequence {xk } is generated by the trust region algorithm, then |f (xk ) − f (xk + dk ) + mk (dk )| ≤ O(dk 2 ) Throughout this paper, we consider the following assumptions in order to analyze the new algorithm: Assumption 2.1. (1) The objective function f (x) has a lower bound on Rn and g(x) = ∇f (x) is uniformly continuous on a open convex set Ω that contains the level set L(x0 ) = {x ∈ Rn | f (x) ≤ f (x0 )}, where x0 ∈ Rn is given. (2) Bk is uniformly bounded, i.e., there exists a constant M such that Bk  ≤ M , for all k. Remark 2.2. If in definition of algorithm set N = 0, the new algorithm reduces to Shi’s adaptive trust region algorithm. Also in definition of fl(k) , we set n(k) = min{N, k}. Remark 2.3. If f (x) is a twice continuously differentiable function and the level set L(x0 ) is bounded, then Assumption 2.1(1) implies that ∇2 f (x) is uniformly continuous and bounded on the open bounded convex set Ω that contains L(x0 ). Hence, there exists a constant M1 > 0 such that ∇2 f (x) ≤ M1 and by using Mean Value Theorem, we have g(x) − g(y) ≤ M1 x − y, ∀ x, y ∈ Ω Now we can give some properties about the new algorithm: Lemma 2.1. If dk is a optimal solution of subproblem (1.3) with respect to αk ≤ sk , then for all k, we have 1 fl(k) − fk − mk (dk ) ≥ −mk (αk qk ) ≥ − αk gkT qk 2 where dk is an optimal solution of subproblem with respect to αk ≤ sk . Proof. It is clear that d = αk qk is a feasible solution to (1.3), and fl(k) − fk ≥ 0. So, we have fl(k) − fk − mk (dk ) ≥ −mk (dk ) ≥ −mk (αk qk )
 1 = − αk gkT qk + α2k qkT Bk qk 2 590 K. Amini & M. Ahookhosh
 1 2 T T ≥ − αk gk qk + αk qk B̂k qk 2
 1 ≥ − αk gkT qk + αk sk qkT B̂k qk 2 1 = − αk gkT qk 2 So the proof is complete. Lemma 2.2. Suppose that Assumption 2.1 holds. Then Steps 5 and 6 of the new algorithm are well-defined, i.e., the new algorithm doesn’t cycle infinitely in the inner cycle. Proof. First, we prove that when p is sufficiently large, (2.4) holds. Let dik be the solution of subproblem (1.3) corresponding to p = i at xk . It follows from (2.1), Remark 2.1 and Lemma 2.1 that       fk − f (xk + dik ) + mk (dik )   fl(k) − f (xk + dik )      i − 1 =  f fl(k) − fk − mk (dik ) l(k) − fk − mk (dk ) ≤ O(dik 2 ) ≤ − 21 αk(i) gkT qk = O(δk(i) ) →0 1 2 τ gk  2 O(δk(i) ) 1 2 τ δk(i) gk  where δk(i) → 0 as i → ∞. This implies that (2.4) holds, so Steps 5 and 6 of the new algorithm are well defined, and inner cycle does not infinity. Lemma 2.3. Suppose that Assumption 2.1 holds, then sequence {fl(k) } is not increasing monotonically, so sequence {fl(k) } is convergent. Proof. If iteration k + 1 be a successful iteration, then we have fl(k) − f (xk + dk ) ≥ µ(fl(k) − fk − mk (dˆk )) ≥ 0, ∀k Therefore, we will have fl(k) ≥ fk+1 , ∀k (2.7) The rest of this proof is similar to Lemma 3.5 in Zhang et al. (2003). 3. Convergence Analysis Trust region algorithms are one of the most prominent methods for solving optimization problems that have strong convergence properties (see Conn et al., 2000; Combination Adaptive Trust Region Method by Non-Monotone Strategy 591 Fletcher, 1987; Moré, 1983; Nocedal and Wright, 2006; Powell, 1975, 1984; and Schultz et al., 1985). In this section, we prove the global convergence of the new trust region algorithm. Lemma 3.1. Suppose that sequence {xk } is generated by the new algorithm, then there exists a positive scalar c̄ such that dk  ≤ c̄gk  (3.1) Proof. From the definition δk , we have dk  ≤ ρp sk qk . By (2.5) we have 0 < λ0 qk 2 ≤ qkT B̂k qk (3.2) By (3.2) and the Cauchy inequality, we have dk  ≤ ρp sk qk  = −ρp where c̄ = ρp λ0 , gkT qk qk  T qk B̂k qk ≤ −ρp ρp gkT qk ≤ gk  = c̄gk  λ0 qk  λ0 so the proof is completed. Lemma 3.2. Suppose that sequence {xk } is generated by the new algorithm, then we have lim dk  = 0 (3.3) k→∞ Proof. Pay attention to Assumption 2.1(2) and Lemma 3.1, all conditions of Lemma 4.6 in Fu and Sun (2005) is confirmed, so like this theorem we can prove the lemma. Theorem 3.3. Suppose that Assumption 2.1 holds and ǫ = 0, then the new algorithm either stops at stationary point of (1.1) or generates an infinite sequence {xk } such that lim − k→∞ gkT qk =0 qk  (3.4) Proof. Suppose that the new algorithm generates sequence {xk } and lim − k→∞ gkT qk =0 qk  Therefore, there exist a ǫ0 > 0 and an infinite subset K ⊆ {0, 1, 2, · · · }, such that − gkT qk ≥ ǫ0 , qk  ∀k ∈ K (3.5) From assumption 2.1(2), we know ∃ M0 > 0 s.t. B̂k  ≤ M0 , ∀k (3.6) 592 K. Amini & M. Ahookhosh thus, qkT B̂k qk ≤ M0 qk 2 , ∀k (3.7) Now let K1 = {k ∈ K | αk = sk } and K2 = {k ∈ K | αk < sk }. By this definition, we have that K = K1 ∪ K2 is an infinite subset of {0, 1, 2, · · · }. In this part of proof, we prove that neither K1 nor K2 can be an infinite set. Thus we obtain a contradiction to (3.5). We assume that K1 be an infinite subset of K, by using Lemma 2.1 and (3.7) we have 1 fl(k) − f (xk+1 ) ≥ µ(fl(k) − fk − mk (dˆk )) ≥ − µαk gkT qk 2  T 2 1 1 g qk ≥ − µsk gkT qk = µ Tk 2 2 qk B̂qk µ ≥ 2M0  gkT qk qk  2 ≥ µ 2 ǫ , 2M0 0 k ∈ K1 So we have fl(k) − f (xk+1 ) ≥ µ 2 ǫ , 2M0 0 k ∈ K1 (3.8) Now we set Rk = k + N + 2, so by definition of fl(k) and recurrence relation we can prove l(Rk )−(k+1) xk+1 = xl(Rk ) −  dl(Rk )−j j=1 Because l(Rk ) − (k + 1) ≤ N + 1 and f (x) is a uniform continuous function, by Lemma 3.2, we have   l(Rk )−(k+1)  (3.9) dl(Rk )−j  = lim fl(Rk ) lim f (xk+1 ) = lim f xl(Rk ) − k→∞ k→∞ j=1 k→∞ This inequality together with Lemma 2.3 and (3.9) on left hand side of (3.8), as k → ∞, implies µ 2 0≥ ǫ (3.10) 2M0 0 This contradiction shows that K1 can not be an infinite subset of K. Now, let k ∈ K2 , Lemma 2.1 implies that 1 fl(k) − f (xk+1 ) ≥ µ(fl(k) − fk − mk (dˆk )) ≥ − µαk gkT qk 2 1 g T qk 1 ≥ µαk qk ǫ0 = − µαk qk  k 2 qk  2 Combination Adaptive Trust Region Method by Non-Monotone Strategy 593 Using of the same argument for deriving (3.10) on preceding inequality, we deduce lim δk = lim αk qk  = 0, k→∞ k ∈ K2 . k→∞ (3.11) Now, we suppose that d˜k is an optimal solution of the following subproblem 1 minn gkT d + dT Bk d, d∈R 2 d ≤ δ̃k , δ̃k = δk /ρ then with definition of algorithm the following inequality holds: fl(k) − f (xk + d˜k ) < µ, fl(k) − fk − mk (d˜k ) k ∈ K2 (3.12) Hence, (3.11) implies that lim δ̃k = 0, k→∞ k ∈ K2 (3.13) So by Lemma 2.1, (3.5) and (3.13) we have       f − f (x + d˜ ) + m (d˜ )   f  l(k) − f (xk + d˜k )   k k k k k  − 1 =        fl(k) − fk − mk (d˜k ) fl(k) − fk − mk (d˜k ) ≤ = O(d˜k 2 ) fl(k) − fk − mk (d˜k ) ≤ O(δ̃k2 ) − 12 α˜k gkT qk O(δ̃k ) O(δ̃k2 ) ≤ 1 → 0 as k → ∞ 1 ˜ T − 2 δk gk qk /qk  2 ǫ0 where k ∈ K2 . Therefore, fl(k) − f (xk + d˜k ) ≥µ fl(k) − fk − mk (d˜k ) (3.14) But for sufficiently large k ∈ K2 , (3.14) contradicts (3.12). Therefore, there exists no infinite subset of K such that (3.5) holds, so the proof is complete. Theorem 3.4. Suppose that conditions of Theorem 3.3 holds and qk satisfies (2.1), then the new algorithm either stops finitely or generates an infinite sequence {xk } such that lim gk  = 0 k→∞ Proof. If the algorithm stops finitely, then theorem is proved. Otherwise, Theorem 3.3 says that the algorithm generates an infinite sequence {xk } such that satisfies (3.4) and since qk satisfies (2.1) we have 0 ≤ τ gk  ≤ − gkT qk g T qk gk  = − k → 0, gk  · qk  qk  k→∞ Therefore we have limk→∞ gk  = 0, so the proof is competed. (3.15) 594 K. Amini & M. Ahookhosh 4. Convergence Rate Analysis In this section, we first prove that the new algorithm have both superlinear and quadratic convergence rate at suitable conditions, and in continuation a second necessary condition are investigated. In this section, we need qk = −Bk−1 gk satisfy in (2.1). To take this purpose, we need to make additional assumption as follows: Assumption 4.1. Bk is a uniformly bounded condition number matrix. Remark 4.1. Nocedal and Wright (2006) showed that if Bk be a positive definite matrix with uniformly bounded condition number then qk = −Bk−1 gk satisfies in (2.1). Theorem 4.1. Suppose that Assumptions 2.1 and 4.1 hold, qk satisfies (2.1) and the new algorithm generates an infinite sequence {xk } such that xk → x∗ , as k → ∞, H(x) = ∇2 f (x) is continuous in a neighborhood N (x∗ , ǫ) of x∗ , and H(x) and Bk are uniformly positive definite matrices such that [Bk − H(x∗ )]qk  =0 k→∞ qk  lim (4.1) where qk = −Bk−1 gk . Then sequence {xk } converges to x∗ superlinearly. Proof. For sufficiently large k, we have that B̂k = Bk , and it is obvious that sk = 1 and δk = ρp qk , so dˆk = qk is a feasible solution of the subproblem for p = 0. By (4.1), we have that [Bk − H(x∗ )]dˆk  gk + H(x∗ )dˆk  = lim =0 k→∞ k→∞ dˆk  dˆk  lim Hence, we have that [Bk − H(x∗ )]dˆk = o(dˆk ) and [Hk − H(x∗ )]dˆk = o(dˆk ), so we can write [Bk − Hk ]dˆk = o(dˆk ) and dˆk = −H(x∗ )−1 gk + o(dˆk ) Thus, dˆk  ≤ H(x∗ )−1  · gk  + o(dˆk ) Therefore, we have gk  1 o(dˆk ) + ≥ ∗ −1 H(x )  dˆk  dˆk  (4.2) Combination Adaptive Trust Region Method by Non-Monotone Strategy 595 Theorem 3.4 implies gk → 0, when k → ∞. Where right hand side of preceding inequality is strictly positive, so dˆk → 0, when k → ∞. By Lemma 2.1 and −gkT qk = qkT Bk qk , we have q T Bk qk 1 (fl(k) − fk − mk (dˆk )) ≥ − αk gkT qk ≥ k 2 2 (4.3) using (4.1), (4.2), and Taylor expansion we have |fk − f (xk + dˆk ) + mk (dˆk )| ≤ o(dˆk 2 ) So from (4.3) and the fact that qk = dˆk = −B −1 gk , we obtain       f − f (x + dˆ ) + m (dˆ )   f   k  l(k) − f (xk + dˆk ) k k k k  − 1 =        fl(k) − fk − mk (dˆk ) fl(k) − fk − mk (dˆk ) ≤ ≤ o(dˆk 2 ) fl(k) − fk − mk (dˆk ) ≤ o(dˆk 2 )) TB q qk k k 2 o(dˆk 2 )) o(dˆk 2 )) → 0 as k → ∞ = qk 2 dˆk 2 which implies that (2.4) holds, so xk+1 = xk + dˆk , for sufficiently large k, and the new trust region algorithm reduces to standard quasi-Newton algorithm. We know that in presence of (4.1) quasi-Newton methods converges superlinearly, so the new method is convergent superlinearly. Theorem 4.2. Suppose that Assumptions 2.1 and 4.1 hold, qk satisfies (2.1), the new algorithm generates an infinite sequence {xk } such that xk → x∗ as k → ∞, H(x) is Lipschitz continuous and uniformly positive definite on a neighborhood N (x∗ , ǫ) of x∗ , Bk = H(xk ) and qk = −Bk−1 gk . Then sequence {xk } converges to x∗ quadratically. Proof. Bk = H(xk ) implies that all condition of Theorem 4.1 holds, so similar to Theorem 4.1 we can prove that qk = dˆk → 0 as k → ∞. From Lemma 2.1 we have  1 q T Hk qk fl(k) − fk − mk (dˆk ) ≥ − αk gkT qk ≥ k 2 2 Now, similar to Theorem 4.1 we can prove that (2.4) holds. So the new algorithm reduce to standard Newton algorithm for sufficiently large k. Hence the new method converges quadratically. Theorem 4.3. Suppose that Assumption 2.1 holds, Bk = Hk , and infinite sequence {xk } is generated by the new algorithm. If sequence {xk } converges to x∗ , then H(x∗ ) is a semi-positive definite matrix, i.e., x∗ satisfies the second order necessary condition. 596 K. Amini & M. Ahookhosh Proof. Assume that λ1k and λ∗ are the smallest eigenvalues of Hk and H(x∗ ), respectively. Let zk be a normalized eigenvector of Hk corresponding to eigenvalue λ1k and zkT gk ≤ 0, then Hk zk = λ1k zk . Suppose that H(x∗ ) is not a semi-positive definite matrix, then λ∗ < 0 and thus λ1k < 0 for sufficiently large k. Because αk qk  · zk  = δk , it follows that αqk zk is a feasible solution to (1.3). Therefore 1 1 −mk (αk qk zk ) ≥ − α2k qk 2 zkT Bk zk = − α2k qk 2 zkT Hk zk 2 2 1 = − α2k qk 2 λ1k 2 By (4.4) and r̂k ≥ µ, we have (4.4) fl(k) − fk+1 ≥ −µmk (αk qk zk ) ≥ 1 2 µα qk 2 λ1k 2 k Now by using same argument for deriving (3.11) and by λ1k → λ∗ , when k → ∞, we have lim δk = lim αk qk  = 0 k→∞ k→∞ (4.5) Similar to procedure for deriving (3.12) and (3.14), we can give a contradiction. So λ∗ ≥ 0 and thus H(x∗ ) is a semi-positive definite matrix. 5. Numerical Experiments In this section, we present computational results to illustrate the performance of the new trust region method in comparison with other versions of trust region methods. All test problems are selected from Moré, Grabow and Hilstrom (1981). List of test problems are presented in Table 1. Programs are performed in double precision MATLAB 7.4 on a 3.0 GHz Intel Pentium IV WinXP PC with 1G RAM. In entire algorithms, we update Bk by the BFGS formula, and the stoping criterion is gk  ≤ 10−8 . We choose µ = 0.1, N = 2n, where n is dimension of test problems. The quadratic subproblems are solved inaccurately by Nearly exact solution algorithm.(algorithm 7.3.5 in Toint(1997)) Table 1. List of test functions. Problem name Problem name Helical valley function Biggs EXP6 function Gaussian function Powell badly scaled function Box three-dimensional function Variably dimensioned function Watson function Penalty function I Penalty function II Brown badly scaled function Brown and Dennis function Gulf research and development function Trigonometric function Extended Rosenbrock function Extended Powell singular function Beale function Wood function Chebyquad function Combination Adaptive Trust Region Method by Non-Monotone Strategy 597 For proper comparison in this section, we decided to compare the new algorithm by Zhang’s non-monotone adaptive trust region algorithm (2003), NMATR-Z, and Fu et al.’s non-monotone adaptive trust region algorithm (2005), NMATR-S. We decided to use a performance profile that proposed by Dolan and Moré (2002). we let S is a set of all algorithms. Now, if fp,s is the performance index, any of established fourth measure, of algorithm s on problem p, then the performance ratio is defined by rp,s = fp,s min{fp,s : s ∈ S} (5.1) If algorithm s is converged for problem p, and rp,s = rfail otherwise, in which rfail must be strictly larger than any performance ratio (5.1). For any factor τ , the overall performance of algorithm s is given by ρs (τ ) = 1 φs (τ ) np (5.2) where np is the number of test problems, and φs (τ ) is the number of problems that for which rp,s ≤ τ . In fact ρs (τ ) is the probability of algorithm s ∈ S that a performance ratio rp,s is within a factor τ ∈ Rn of the best possible ratio. The function ρs (τ ) is the distribution function for performance ratio. In particular, ρs (1) gives the probability that algorithm s wins over all other algorithms, and limτ →rfail ρs (τ ) gives the probability of that algorithm s solve a problem. So this performance profile can be a measure of robustness of algorithms. In performance profiles we can use one index, but we have three measures ni , nf and ng . Pay attention to this subject we can use performance profile for all of this indexes or combine these measures. On the other hand we know that calculation of gradient evaluation is more expensive than function evaluation, so we decided to use the following index: nc = nf + 5 ng In Fig. 1, we can see all test problems solved by any solver. For nearly 50% of problems new algorithm has better result; and for nearly τ ≥ 1.5 all problem solved by the new algorithm. Hence, the new algorithm is more robust because its performance profile grows faster than the other profiles. It means that in the cases that the new method is not better than the other algorithms, the performance index is close to minimum of performance index. One of the most important factor in benchmarking of the iterative methods is cpu time of algorithms. In Table 1, we present the summation of cpu time for any algorithms, and from this idea we can see that new algorithm has the best result, but for a better comparison we use the performance profile and illustrate the results in Fig. 2. In Fig. 2, we observe that in nearly 60% of cases the new method is faster than the other considered non-monotone adaptive trust region algorithm. In the cases 598 K. Amini & M. Ahookhosh 1 0.95 0.9 0.85 0.8 0.75 0.7 0.65 0.6 0.55 0.5 1 1.5 2 2.5 3 3.5 4 Fig. 1. Performance profile for nf + 5 ng . 1 0.9 0.8 0.7 0.6 0.5 NMATR–S NMATR–Z NMATR–B 0.4 1 1.5 2 2.5 3 3.5 4 4.5 Fig. 2. Performance profile for cputime. 5 5.5 Combination Adaptive Trust Region Method by Non-Monotone Strategy 599 that new algorithm has not the best time, it is very near to the minimum time, of these cases. We can conclude that all of considered algorithms are efficient and robustness. However, between these methods, the new non-monotone adaptive trust region algorithm is the best in terms of efficiency and robustness. 6. Conclusion In this paper, we combine the non-monotone strategy with Shi et al.’s (2008) adaptive trust region method to propose a new non-monotone trust region method with adaptive radius. In this method, the trust region radius can be adjusted automatically according to the current iterative information and computed by simple formula. By choosing different qk , we have a different trust region methods. Natural choice is qk = −gk , and another choice is qk = −Bk−1 gk . Theoretical analysis exhibits that the new method has global convergence and if qk = −Bk−1 gk , we can show the superlinear and quadratic convergence rate of this method. Finally, we have provided the numerical results that indicate the new method is robust and efficient for solving unconstrained optimization problems. References Chamberlain, RM, MJD Powell, C Lemarechal and HC Pedersen (1982). The watchdog technique for forcing convergence in algorithm for constrained optimization. Mathematical Programming Study, 16, 1–17. Conn, AR, NIM Gould and PhL Toint (2000). Trust Region Methods. Society for Industrial and Applied Mathematics, SIAM, Philadelphia. Deng, NY, Y Xiao and FJ Zhou (1993). Non-monotonic trust region algorithm. Journal of Optimization Theory and Applications, 76, 259–285. Dolan, E and JJ Moré (2002). Benchmarking optimization software with performance profiles. Mathematical Programming, 91, 201–213. Fletcher, R (1987). Practical Method of Optimization. New York: John Wiley and Sons. Fu, JH and WY Sun (2005). Nonmonotone adaptive trust-region method for unconstrained optimization problems, Appl. Math. Comput. 163, 489–504. Grippo, L, F Lamparillo and S Lucidi (1986). A non-monotone line search technique for Newton’s method. SIAM Journal of Numerical Analysis, 23, 707–716. Grippo, L, F Lamparillo and S Lucidi (1989). A truncated Newton method with nonmonotone line search for unconstrained optimization. Journal of Optimization Theory and Applications, 60, 401–419. Moré, JJ (1983). Recent developments in algorithms and software for trust region methods. In Bachem, A, M Grotchel and B Korte (eds.), Mathematical Programming: The State of the Art, pp. 258–287. Berlin: Springer. Moré, JJ, BS Grabow and KE Hillstrom (1981). Testing unconstrained optimization software. ACM Transactions on Mathematical Software, 7, 17–41. Nocedal, J and SJ Wright (2006). Numerical Optimization. New York: Springer. Powell, MJD (1975). Convergence properties of a class minimization algorithms. In Mangasarian, OL, RR Meyer and SM Robinson (Eds.), Nonlinear Programming, Vol. 2, pp. 1–27. New York: Academic Press. 600 K. Amini & M. Ahookhosh Powell, MJD (1984). On the global convergence of trust region algorithms for unconstrained optimization. Mathematical Programming, 29, 297–303. Sartenaer, A (1997). Automatic determination of an initial trust region in nonlinear programming. SIAM Journal of Scientific Computing, 18(6), 1788–1803. Schultz, GA, RB Schnabel and RH Byrd (1985). A family of trust-region-based algorithms for unconstrained minimization with strong global convergence. SIAM Journal of Numerical Analysis, 22, 47–67. Shi, ZJ and JH Guo (2008). A new trust region methods for unconstrained optimization. Journal of Computational and Applied Mathematics, 213, 509–520. Toint, PhL (1996). An assessment of nonmonotone line search technique for unconstrained optimization. SIAM Journal of Scientific Computing, 17, 725–739. Toint, PhL (1997). Non-monotone trust-region algorithm for nonlinear optimization subject to convex constraints. Mathematical Programming, 77, 69–94. Zhang, XS, JL Zhang and LZ Liao (2002). An adaptive trust region method and its convergence. Science China, 45, 620–631. Zhang, XS, JL Zhang and LZ Liao (2003). An Nonmonotone adaptive trust region method and its convergence. An International Computer and Mathematics with Applications, 45, 1469–1477. Keyvan Amini is now an Associate Professor at the Department of Mathematics, Razi University (Kermanshah, Iran). He received BS degree in Razi University in 1996, and MS and PhD degrees in Sharif University of Technology (Tehran, Iran). His research is focused on Nonlinear Programming techniques, especially Trust Region Methods. He is also working on Interior point methods for linear programming. He has published papers in Computers and Mathematics with Applications, ANZIAIM JOURNAL, Bulletin of Australian Mathematical Society, Journal of Computational and Applied Mathematics, Applied Mathematics and Computation, Acta Mathematica Sinica, Southeast Asian Bulletin of Mathematics. Masoud AhooKhosh received his BS, MS degrees in Razi University, Kermanshah, Iran. His research focused on Nonlinear Programming techniques, especially Trust Region Methods. He has published paper’s in Computer Mathematics with Applications, Numerical Algorithms and Applied Mathematical Modeling.