Inexact Successive quadratic approximation for regularized optimization

Abstract

Successive quadratic approximations, or second-order proximal methods, are useful for minimizing functions that are a sum of a smooth part and a convex, possibly nonsmooth part that promotes regularization. Most analyses of iteration complexity focus on the special case of proximal gradient method, or accelerated variants thereof. There have been only a few studies of methods that use a second-order approximation to the smooth part, due in part to the difficulty of obtaining closed-form solutions to the subproblems at each iteration. In fact, iterative algorithms may need to be used to find inexact solutions to these subproblems. In this work, we present global analysis of the iteration complexity of inexact successive quadratic approximation methods, showing that an inexact solution of the subproblem that is within a fixed multiplicative precision of optimality suffices to guarantee the same order of convergence rate as the exact version, with complexity related in an intuitive way to the measure of inexactness. Our result allows flexible choices of the second-order term, including Newton and quasi-Newton choices, and does not necessarily require increasing precision of the subproblem solution on later iterations. For problems exhibiting a property related to strong convexity, the algorithms converge at global linear rates. For general convex problems, the convergence rate is linear in early stages, while the overall rate is O(1 / k). For nonconvex problems, a first-order optimality criterion converges to zero at a rate of $O(1/\sqrt{k})$.

Appendices

Proof of Lemma 5

Proof

We have

$$\begin{aligned} Q^*&= \min _{d} \, \nabla f\left( x\right) ^T d + \frac{1}{2}d^T H d + \psi \left( x+ d\right) - \psi \left( x\right) \nonumber \\&\le \min _d \, f\left( x+d\right) + \psi \left( x+d\right) + \frac{1}{2}d^T H d - F\left( x\right) \end{aligned}$$

(56a)

$$\begin{aligned}&\le F\left( x + \lambda \left( P_{{\varOmega }}\left( x\right) - x \right) \right) + \frac{\lambda ^2}{2}\left( P_{\varOmega }\left( x\right) - x\right) ^T H \left( P_{\varOmega }\left( x\right) - x \right) - F\left( x\right) \;\; \nonumber \\&\qquad \forall \lambda \in [0,1] \end{aligned}$$

(56b)

$$\begin{aligned}&\le \left( 1 - \lambda \right) F\left( x\right) + \lambda F^* - \frac{\mu \lambda \left( 1 -\lambda \right) }{2} \left\| x - P_{\varOmega }\left( x\right) \right\| ^2 \end{aligned}$$

(56c)

$$\begin{aligned}&\qquad + \frac{\lambda ^2 }{2} \left( x - P_{\varOmega }\left( x\right) \right) ^T H\left( x - P_{\varOmega }\left( x\right) \right) - F\left( x\right) \;\; \forall \lambda \in [0,1] \nonumber \\&\le \lambda (F^* - F\left( x\right) ) - \frac{\mu \lambda \left( 1 -\lambda \right) }{2} \left\| x - P_{\varOmega }\left( x\right) \right\| ^2 \nonumber \\&\quad + \frac{\lambda ^2 }{2} \Vert H\Vert \Vert x - P_{\varOmega }\left( x\right) \Vert ^2 \;\; \forall \lambda \in [0,1], \end{aligned}$$

(56d)

where in (56a) we used the convexity of f, in (56b) we set $d=\lambda (P_{{\varOmega }}(x)-x)$, and in (56c) we used the optimal set strong convexity (10) of F. Thus we obtain (25). $\square$

Proof of Lemma 6

Proof

Consider

$$\begin{aligned} \lambda _k = \arg \min _{\lambda \in [0,1]} \, -\lambda \delta _k + \frac{\lambda ^2}{2}A_k, \end{aligned}$$

(57)

then by setting the derivative to zero in (57), we have

$$\begin{aligned} \lambda _k = \min \left\{ 1, \frac{\delta _k}{A_k}\right\} . \end{aligned}$$

(58)

When $\delta _k \ge A_k$, we have from (58) that $\lambda _k = 1$. Therefore, from (30) we get

$$\begin{aligned} \delta _{k+1} \le \delta _{k} + c_k\left( - \delta _k + \frac{A_k}{2}\right) \le \delta _{k} + c_k\left( -\delta _k + \frac{\delta _k}{2}\right) = \left( 1 - \frac{c_k}{2}\right) \delta _{k}, \end{aligned}$$

proving (31).

On the other hand, since $A \ge A_k > 0, c_k \ge 0$ for all k, (30) can be further upper-bounded by

$$\begin{aligned} \delta _{k+1} \le \delta _k + c_k\left( -\lambda _k \delta _k + \frac{A_k}{2}\lambda _k^2\right) \le \delta _k + c_k \left( -\lambda _k \delta _k + \frac{A}{2} \lambda _k^2 \right) ,\quad \forall \lambda _k \in [0,1]. \end{aligned}$$

Now take

$$\begin{aligned} \lambda _k = \min \left\{ 1, \frac{\delta _k}{A}\right\} . \end{aligned}$$

(59)

For $\delta _k \ge A \ge A_k$, (31) still applies. If $A > \delta _k$, we have from (59) that $\lambda _k = \delta _k / A$, hence

$$\begin{aligned} \delta _{k+1} \le \delta _k - \frac{c_k}{2A}\delta _k^2. \end{aligned}$$

(60)

This together with (31) imply that $\{\delta _k\}$ is a monotonically decreasing sequence. Dividing both sides of (60) by $\delta _{k+1}\delta _k$, and from the fact that $\delta _k$ is decreasing and nonnegative, we conclude

$$\begin{aligned} \delta _k^{-1} \le \delta _{k+1}^{-1} - \frac{c_k \delta _k}{2\delta _{k+1}A} \le \delta _{k+1}^{-1} - \frac{c_k}{2A} \end{aligned}$$

Summing this inequality from $k_0$, and using $\delta _{k_0} < A$, we obtain

$$\begin{aligned} \delta _k^{-1} \ge \delta _{k_0}^{-1} + \frac{\sum _{t=k_0}^{k-1}c_t}{2 A} \ge \frac{\sum _{t=k_0}^{k-1}c_t + 2}{2 A} \Rightarrow \delta _k \le \frac{2 A}{\sum _{t=k_0}^{k-1} c_t + 2}, \end{aligned}$$

proving (32). $\square$