On the R-superlinear convergence of the KKT residuals generated by the augmented Lagrangian method for convex composite conic programming

Abstract

Due to the possible lack of primal-dual-type error bounds, it was not clear whether the Karush–Kuhn–Tucker (KKT) residuals of the sequence generated by the augmented Lagrangian method (ALM) for solving convex composite conic programming (CCCP) problems converge superlinearly. In this paper, we resolve this issue by establishing the R-superlinear convergence of the KKT residuals generated by the ALM under only a mild quadratic growth condition on the dual of CCCP, with easy-to-implement stopping criteria for the augmented Lagrangian subproblems. This discovery may help to explain the good numerical performance of several recently developed semismooth Newton-CG based ALM solvers for linear and convex quadratic semidefinite programming.

Appendix.

1.1 1: Proof of Lemma 2

Proof

Obviously if the quadratic growth condition for (D2) holds at $({\bar{w}}, {\bar{y}}, -\mathcal {A}^*{\bar{w}} - \mathcal {B}^*{\bar{y}} - c)\in \mathrm{SOL}_{\mathrm{D2}}$, then the quadratic growth condition for (D) holds at ${\bar{y}}\in \mathrm{SOL}_{\mathrm{D}}$. Now we prove the reverse implication. We first show that there exist positive constants $\varepsilon$ and $\mu$ such that

$$\begin{aligned} -g^0(w,{y},{s}) \geqslant -\sup \,(\mathrm{D}) + \mu \Vert w- {\bar{w}}\Vert ^2, \quad \forall \; (w,y,s)\in \mathrm{F}_{\mathrm{D2}}\cap \mathbb {B}_{\varepsilon }({\bar{w}}, {\bar{y}}, {\bar{s}}). \end{aligned}$$

(31)

It follows from ${\bar{w}}\in \mathrm{dom}\,h^*$ and the local strong convexity of $h^*$ that there exist positive constants $\varepsilon$ and $\mu$ such that

$$\begin{aligned} h^*(w)\geqslant h^*({\bar{w}}) + \langle \nabla h^*({\bar{w}}), w - {\bar{w}}\rangle + \mu \Vert w - {\bar{w}}\Vert ^2, \quad \forall \; w\in \mathbb {B}_{\varepsilon }({\bar{w}})\cap \mathrm{dom}\,h^*. \end{aligned}$$

(32)

Let ${\bar{x}}\in \mathrm{SOL}_P$ be given such that $( {\bar{w}}, {\bar{y}},{\bar{s}}, {\bar{x}})$ satisfies the KKT optimality condition (10). By the convexity of $p^*$, we also get

$$\begin{aligned}&p^*(s)\geqslant p^*({\bar{s}}) + \langle \, {\bar{x}}\,,s-{\bar{s}}\,\rangle = p^*({\bar{s}}) +\langle \, {\bar{x}}\,, \mathcal {A}^*(-w+{\bar{w}}) + \mathcal {B}^*(-y + {\bar{y}})\,\rangle ,\nonumber \\&\quad \forall \; (w,y,s)\in \mathrm{F}_{\mathrm{D2}}. \end{aligned}$$

(33)

Note that $\mathcal {B}{\bar{x}}- b\in \mathcal {N}_{\mathcal {Q}^\circ }({\bar{y}})$ implies $\langle \mathcal {B}{\bar{x}}-b, y\rangle \leqslant 0$ for any $y\in \mathcal {Q}^\circ$ and $\langle \mathcal {B}{\bar{x}}-b, {\bar{y}}\rangle = 0$. One can thus derive the inequality (31) by adding the two inequalities (32) and (33). Now by shrinking $\varepsilon$ if necessary, we have, for any $(w,y,s)\in \mathrm{F}_{\mathrm{D2}}\cap \mathbb {B}_{\varepsilon }({\bar{w}}, {\bar{y}}, {\bar{s}})$,

$$\begin{aligned} \begin{array}{rl} -g^0(w,y,s) \geqslant &{} -g^0(y)/2 - g^0(w,{y},s)/2\\ \geqslant &{} (-\sup \,(\mathrm{D}) + \kappa _2\mathrm{dist}^2\,(y, \mathrm{SOL}_{\mathrm{D}}))/2+ (-\sup \,(\mathrm{D}) + \mu \Vert w -{\bar{w}}\Vert ^2)/2\\ \geqslant &{} -\sup \,(\mathrm{D}) + \min \{\kappa _2/2, \mu /2\}(\mathrm{dist}^2\,(y, \mathrm{SOL}_{\mathrm{D}})+\Vert w -{\bar{w}}\Vert ^2), \end{array} \end{aligned}$$

where in the second inequality the first term is due to the assumed quadratic growth condition for (D) at ${\bar{y}}$ with modulus $\kappa _2$, and the second term comes from (31). Finally, it follows that for any ${\hat{y}}\in \mathrm{SOL}_{\mathrm{D}}$ and any $(w,y,s)\in \mathrm{F}_{\mathrm{D2}}\cap \mathbb {B}_{\varepsilon }({\bar{w}}, {\bar{y}}, {\bar{s}})$,

$$\begin{aligned} \begin{array}{rl} \mathrm{dist}^2((w,y,s), \mathrm{SOL}_{\mathrm{D2}})\leqslant &{} \Vert w - {\bar{w}}\Vert ^2 + \Vert y - {\hat{y}}\Vert ^2+ \Vert s - {\bar{s}}\Vert ^2 \\ =&{} \Vert w - {\bar{w}}\Vert ^2 + \Vert y - {\hat{y}}\Vert ^2+ \Vert \mathcal {A}^*(w - {\bar{w}}) + \mathcal {B}^*(y-{\hat{y}})\Vert ^2\\ \leqslant &{} (1+2\Vert \mathcal {A}^*\Vert ^2)\Vert w - {\bar{w}}\Vert ^2 + (1+2\Vert \mathcal {B}^*\Vert ^2)\Vert y - {\hat{y}}\Vert ^2, \end{array} \end{aligned}$$

which, with ${\hat{y}}: = \varPi _{\mathrm{SOL}_{\mathrm{D}}}(y)$, implies

$$\begin{aligned} \mathrm{dist}^2((w,y,s), \mathrm{SOL}_{\mathrm{D2}})\leqslant (1+2\Vert \mathcal {A}^*\Vert ^2)\Vert w - {\bar{w}}\Vert ^2 + (1+2\Vert \mathcal {B}^*\Vert ^2)\mathrm{dist}^2\,(y, \mathrm{SOL}_{\mathrm{D}}). \end{aligned}$$

Thus, the quadratic growth condition (11) holds at $({\bar{w}}, {\bar{y}}, {\bar{s}})\in \mathrm{SOL}_{\mathrm{D2}}$ for (D2). $\square$

1.2 2: Proof of Proposition 1(c)

Proof

The convergence of $\{z^k\}$ under criterion (A) has been proven in Proposition 1. To establish the desired convergence rate, we first recall that the calmness of the mapping $T^{-1}$ at the origin for $z^{\infty }$ with modulus $\kappa$ asks for the existence of positive constants $\varepsilon$ and $\delta$ such that

$$\begin{aligned} \text {dist}(z, T^{-1}(0)) \leqslant \kappa \Vert u\Vert , \quad \forall \; z\in T^{-1}(u)\cap \mathbb {B}_{\delta }(z^\infty ), \;\, \forall \; u\in \mathbb {B}_{\varepsilon }(0). \end{aligned}$$

From parts (a) and (b) in Proposition 1 and the convergence of $\{z^k\}$, we obtain that

$$\begin{aligned} P_k(z^k)\in T^{-1}((z^k - P_k(z^k))/\sigma _k),\ \forall \; k\geqslant 0 \quad \mathrm{and} \quad P_k(z^k) \rightarrow z^\infty \ \mathrm{as } \ k\rightarrow \infty , \end{aligned}$$

which, imply the existence of a nonnegative integer ${\bar{k}}$ such that

$$\begin{aligned} \text {dist}(P_k(z^k), T^{-1}(0)) \leqslant (\kappa /\sigma _k) \Vert z^k - P_k(z^k)\Vert , \quad \forall \; k\geqslant {\bar{k}}. \end{aligned}$$

Now taking ${\bar{z}} = \varPi _{T^{-1}(0)}(z^k)$ in Proposition 1(b), we deduce that for any $k\geqslant 0$,

$$\begin{aligned} \begin{array}{ll} \Vert z^k - P_k(z^k)\Vert ^2&{} \leqslant \Vert z^k -\varPi _{T^{-1}(0)}(z^k)\Vert ^2 - \Vert P_k(z^k) -\varPi _{T^{-1}(0)}(z^k)\Vert ^2\\ &{} \leqslant \text {dist}^2(z^k,T^{-1}(0)) - \text {dist}^2(P_k(z^k),T^{-1}(0)). \end{array} \end{aligned}$$

Thus, it holds that

$$\begin{aligned} \text {dist}(P_k(z^k), T^{-1}(0))\leqslant \kappa /\sqrt{\kappa ^2 +\sigma _k^2}\,\text {dist}(z^k, T^{-1}(0)) , \quad \forall \, k\geqslant {\bar{k}}. \end{aligned}$$

Hence, if criterion (B) is also executed, we have that for any $k\geqslant {\bar{k}}$,

$$\begin{aligned} \begin{array}{ll} &{} \Vert z^{k+1} - \varPi _{T^{-1}(0)}(P_k(z^k))\Vert \\ &{}\quad \leqslant \Vert z^{k+1} - P_k(z^k)\Vert + \Vert P_k(z^k)-\varPi _{T^{-1}(0)}(P_k(z^k))\Vert \\ &{}\quad \leqslant \eta _k\Vert z^{k+1} - z^k\Vert + \Vert P_k(z^k)-\varPi _{T^{-1}(0)}(P_k(z^k))\Vert \\ &{}\quad \leqslant \eta _k(\Vert z^{k+1} - \varPi _{T^{-1}(0)}(P_k(z^k))\Vert + \Vert z^k - P_k(z^k)\Vert )\\ &{}\qquad + (\eta _k+1)\Vert P_k(z^k)-\varPi _{T^{-1}(0)}(P_k(z^k))\Vert \\ &{}\quad \leqslant \eta _k\Vert z^{k+1} - \varPi _{T^{-1}(0)}(P_k(z^k))\Vert + \left[ \eta _k + (\eta _k+1)\kappa /\sqrt{\kappa ^2 +\sigma _k^2}\right] \text {dist}(z^k,T^{-1}(0)). \end{array} \end{aligned}$$

Then the inequality in part (c) of Proposition 1 readily follows from the fact that $\text {dist}(z^{k+1}, T^{-1}(0))\leqslant \Vert z^{k+1} - \varPi _{T^{-1}(0)}(P_k(z^k))\Vert$ for any $k\geqslant 0$. $\square$

1.3 3: Proof of Proposition 3(b)

Proof

By Lemma 1(b) and the convergence of $\{y^k\}$, we have $\mathrm{dist}\,(0, T_l(x^{k+1},y^{k+1}))\rightarrow 0$ under criterion $({\widetilde{B}})$. Therefore, by the upper Lipschitz continuity of $T_l^{-1}$ at the origin, we can derive, for k sufficiently large,

$$\begin{aligned} \begin{array}{rl} \mathrm{dist}\,((x^{k+1},y^{k+1}), T_l^{-1}(0))\leqslant &{} \kappa _l\,\mathrm{dist}\,(0, T_l(x^{k+1},y^{k+1}))\\ \leqslant &{} \kappa _l\,(\mathrm{dist}^2\,(0,\partial f_k(x^{k+1}) + (1/\sigma _k^2)\Vert y^{k+1}-y^k\Vert ^2)^{1/2}\\ \leqslant &{} (\kappa _l/\sigma _k)(1+\eta _k'^2)\Vert y^{k+1} - y^k\Vert . \end{array} \end{aligned}$$

This completes the proof of this part. $\square$