On sampling Kaczmarz–Motzkin methods for solving large-scale nonlinear systems

Abstract

In this paper, for solving large-scale nonlinear equations, we propose a nonlinear sampling Kaczmarz–Motzkin (NSKM) method. Based on the local tangential cone condition and the Jensen’s inequality, we prove convergence of our method with two different assumptions. Then, for solving nonlinear equations with the convex constraints, we present two variants of the NSKM method: the projected sampling Kaczmarz–Motzkin (PSKM) method and the accelerated projected sampling Kaczmarz–Motzkin (APSKM) method. With the use of the nonexpansive property of the projection and the convergence of the NSKM method, the convergence analysis is obtained. Numerical results show that the NSKM method with the sample of the suitable size outperforms the nonlinear randomized Kaczmarz method in terms of calculation times. The APSKM and PSKM methods are practical and promising for the constrained nonlinear problem.

Appendices

Appendix

A Proof of Lemma 2

Proof

$$\begin{aligned}&\Vert x_{k+1}-x^*\Vert ^2-\Vert x_{k}-x^*\Vert ^2\\&\quad =\Vert x_{k+1}-x_{k}\Vert ^2+2\left<x_{k+1}-x_{k},x_{k}-x^*\right>\\&\quad =\Vert -\frac{f_{i_{k+1}}(x_{k})}{\Vert \nabla f_{i_{k+1}}(x_{k})\Vert ^2}\nabla f_{i_{k+1}}(x_{k})^T\Vert ^2+2\left\langle -\frac{f_{i_{k+1}}(x_{k})}{\Vert \nabla f_{i_{k+1}}(x_{k})\Vert ^2}\nabla f_{i_{k+1}}(x_{k})^T, x_{k}-x^*\right\rangle \\&\quad =\frac{f_{i_{k+1}}^2(x_{k})}{\Vert \nabla f_{i_{k+1}}(x_{k})\Vert ^2}-2\frac{f_{i_{k+1}}(x_{k})}{\Vert \nabla f_{i_{k+1}}(x_{k})\Vert ^2}\nabla f_{i_{k+1}}(x_{k})(x_{k}-x^*)\\&\quad =\frac{f_{i_{k+1}}^2(x_{k})}{\Vert \nabla f_{i_{k+1}}(x_{k})\Vert ^2}-2\frac{f_{i_{k+1}}(x_{k})}{\Vert \nabla f_{i_{k+1}}(x_{k})\Vert ^2}f_{i_{k+1}}(x_{k})\\&\qquad +2\frac{f_{i_{k+1}}(x_{k})}{\Vert \nabla f_{i_{k+1}}(x_{k})\Vert ^2}(f_{i_{k+1}}(x_{k})-f_{i_{k+1}}(x^*)-\nabla f_{i_{k+1}}(x_{k})(x_{k}-x^*)). \end{aligned}$$

When $k=0$, $x_0\in \mathscr {B}_\rho (x_0)$ and $|f_i(x_0)-f_i(x^*)-\nabla f_i(x_0)(x_0-x^*)|\le \eta _i|f_i(x_0)-f_i(x^*)|$ $(i=1,2,\ldots ,m)$, then we have

$$\begin{aligned}&\Vert x_{1}-x^*\Vert ^2-\Vert x_{0}-x^*\Vert ^2\nonumber \\&\quad =\frac{f_{i_{1}}^2(x_{0})}{\Vert \nabla f_{i_{1}}(x_{0})\Vert ^2}+2\frac{f_{i_{1}}(x_{0})}{\Vert \nabla f_{i_{1}}(x_{0})\Vert ^2}(f_{i_{1}}(x_{0})-f_{i_{1}}(x^*)-\nabla f_{i_{1}}(x_{0})(x_{0}-x^*))\nonumber \\&\qquad -2\frac{f_{i_{1}}(x_{0})}{\Vert \nabla f_{i_{1}}(x_{0})\Vert ^2}f_{i_{1}}(x_{0})\nonumber \\&\quad \le \frac{f_{i_{1}}^2(x_{0})}{\Vert \nabla f_{i_{1}}(x_{0})\Vert ^2}+2\eta _{i_{1}}\frac{\mid f_{i_{1}}(x_{0})\mid }{\Vert \nabla f_{i_{1}}(x_{0})\Vert ^2}\mid f_{i_{1}}(x_{0})\mid -2\frac{f_{i_{1}}^2(x_{0})}{\Vert \nabla f_{i_{1}}(x_{0})\Vert ^2}\nonumber \\&\quad =-(1-2\eta _{i_{1}})\frac{f_{i_{1}}^2(x_{0})}{\Vert \nabla f_{i_{1}}(x_{0})\Vert ^2}. \end{aligned}$$

(11)

Since $x^*\in \mathscr {B}_{\rho /2}(x_0)$ and (11), we have

$$\begin{aligned} \Vert x_1-x_0\Vert =\Vert x_1-x^*+x^*-x_0\Vert \le \Vert x_1-x^*\Vert +\Vert x^*-x_0\Vert \le \rho . \end{aligned}$$

Thus, $x_1\in \mathscr {B}_\rho (x_0)$.

We assume that when $k\le n$ $(n\in \mathbb {N})$, $x_k\in \mathscr {B}_\rho (x_0)$ and (4) holds, then, for $k=n+1$, similar to the derivation of $k=0$, we have $x_{n+1}\in \mathscr {B}_\rho (x_0)$ and (4) holds. $\square$

B Proof of Lemma 4

Proof

Since $f: \mathscr {D}(f)\rightarrow \mathbb {R}$ is a convex function, we have that

$$\begin{aligned} f((1-\alpha )x+\alpha y)\le (1-\alpha )f(x)+\alpha f(y), \forall \alpha \in [0,1], \forall x,y \in \mathscr {D}(f). \end{aligned}$$

(12)

By Taylor formula, it holds

$$\begin{aligned} f((1-\alpha )x+\alpha y)=f(x)+\alpha f'(x)(y-x)+o(\Vert \alpha (y-x)\Vert ). \end{aligned}$$

(13)

Combining (12) and (13), we obtain that

$$\begin{aligned} f(y)-f(x)\ge f'(x)(y-x)+\frac{o(\Vert \alpha (y-x)\Vert )}{\alpha }. \end{aligned}$$

Let $\alpha \rightarrow 0$

$$\begin{aligned} f(y)\ge f(x)+ f'(x)(y-x). \end{aligned}$$

This completes the proof. $\square$

C Proof of Lemma 8

Proof

There are two cases to consider the following:

Case 1. $x_{k+\frac{2}{4}}= x_{k+\frac{3}{4}}$. In this case, we have

$$\begin{aligned} x_{k+\frac{1}{4}}\ne x_{k+\frac{2}{4}}= x_{k+\frac{3}{4}}. \end{aligned}$$

Case 2. $x_{k+\frac{2}{4}}\ne x_{k+\frac{3}{4}}$. By Lemma 7, we obtain that

$$\begin{aligned} \Vert x_{k+\frac{2}{4}}-x^*\Vert _2^2\le \Vert x_{k+\frac{1}{4}}-x^*\Vert _2^2-\Vert x_{k+\frac{1}{4}}-x_{k+\frac{2}{4}}\Vert _2^2. \end{aligned}$$

Because $x_{k+\frac{1}{4}}\ne x_{k+\frac{2}{4}}$, we get that

$$\begin{aligned} \Vert x_{k+\frac{2}{4}}-x^*\Vert _2^2 <\Vert x_{k+\frac{1}{4}}-x^*\Vert _2^2. \end{aligned}$$

Besides, from $x_{k+\frac{2}{4}}\ne x_{k+\frac{3}{4}}$, it can also be obtained that

$$\begin{aligned} \Vert x_{k+\frac{3}{4}}-x^*\Vert _2^2\le \Vert x_{k+\frac{2}{4}}-x^*\Vert _2^2-\Vert x_{k+\frac{2}{4}}-x_{k+\frac{3}{4}}\Vert _2^2 <\Vert x_{k+\frac{2}{4}}-x^*\Vert _2^2. \end{aligned}$$

Therefore, $\Vert x_{k+\frac{3}{4}}-x^*\Vert _2^2<\Vert x_{k+\frac{2}{4}}-x^*\Vert _2^2 <\Vert x_{k+\frac{1}{4}}-x^*\Vert _2^2,$ which implies that $x_{k+\frac{1}{4}}\ne x_{k+\frac{3}{4}}$.

This completes the proof. $\square$

D Proof of Lemma 9

Proof

We first show that $\lambda _k\ge 1$. Observe that

$$\begin{aligned}&2\left\langle x_{k-\frac{3}{5}}-x_{k-\frac{1}{5}},x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}}\right\rangle \\&\quad =\left\| x_{k-\frac{3}{5}}-x_{k-\frac{1}{5}}\right\| ^2+\left\| x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}}\right\| ^2-\left\| x_{k-\frac{1}{5}}-x_{k-\frac{2}{5}}\right\| ^2\\&\quad \le 2\left( \left\| x_{k-\frac{2}{5}}-x_{k-\frac{3}{5}}\right\| ^2-\left\| x_{k-\frac{1}{5}}-x_{k-\frac{2}{5}}\right\| ^2\right) , \end{aligned}$$

where the last inequality follows from Lemma 7.

Hence $\left\langle x_{k-\frac{3}{5}}-x_{k-\frac{1}{5}},x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}}\right\rangle \le \Vert x_{k-\frac{2}{5}}-x_{k-\frac{3}{5}}\Vert ^2$, which implies that

$$\begin{aligned} \lambda _k=\frac{\Vert x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}}\Vert ^2}{\left<x_{k-\frac{3}{5}}-x_{k-\frac{1}{5}}, x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}}\right>}\ge 1. \end{aligned}$$

(14)

Next, we will prove that $x_{k}-x_{k-\frac{2}{5}}$ and $x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}}$ are orthogonal

$$\begin{aligned} \left\langle x_{k}-x_{k-\frac{2}{5}}, x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}} \right\rangle&=\left\langle (x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}})+\lambda _k (x_{k-\frac{1}{5}}-x_{k-\frac{3}{5}}), x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}} \right\rangle \nonumber \\&=\Vert x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}}\Vert ^2+\lambda _k \left\langle x_{k-\frac{1}{5}}-x_{k-\frac{3}{5}}, x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}} \right\rangle \nonumber \\&=\Vert x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}}\Vert ^2\left( 1+\frac{\left\langle x_{k-\frac{1}{5}}-x_{k-\frac{3}{5}}, x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}} \right\rangle }{\left<x_{k-\frac{3}{5}}-x_{k-\frac{1}{5}}, x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}}\right>}\right) \nonumber \\&=0. \end{aligned}$$

(15)

Finally, we utilize (14) and (15) to prove (10).

For every $x\in C_{\alpha _1^k} \cap C_{\alpha _2^k}$, we have

$$\begin{aligned} \Vert x_k-x\Vert ^2=\Vert x_k-x_{k-\frac{1}{5}}\Vert ^2+\Vert x_{k-\frac{1}{5}}-x\Vert ^2+2\langle x_k-x_{k-\frac{1}{5}}, x_{k-\frac{1}{5}}-x\rangle . \end{aligned}$$

By writing $\langle x_k-x_{k-\frac{1}{5}}, x_{k-\frac{1}{5}}-x \rangle =\langle x_k-x_{k-\frac{1}{5}}, x_{k-\frac{1}{5}}-x_k \rangle +\langle x_k-x_{k-\frac{1}{5}}, x_{k}-x \rangle$, we find that

$$\begin{aligned} \Vert x_k-x\Vert ^2=\Vert x_{k-\frac{1}{5}}-x\Vert ^2-\Vert x_{k}-x_{k-\frac{1}{5}}\Vert ^2+2\langle x_k-x_{k-\frac{1}{5}}, x_{k}-x\rangle . \end{aligned}$$

By the definition of $x_{k}$, we obtain that

$$\begin{aligned} \langle x_k-x_{k-\frac{1}{5}}, x_{k}-x\rangle&=(1-\lambda _k)\langle x_{k-\frac{3}{5}}-x_{k-\frac{1}{5}}, x_{k}-x\rangle \\&=(1-\lambda _k)(\langle x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}}, x_{k}-x\rangle +\langle x_{k-\frac{2}{5}}-x_{k-\frac{1}{5}}, x_{k}-x\rangle ). \end{aligned}$$

For the first inner product of the above formula, we have

$$\begin{aligned} \langle x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}}, x_{k}-x\rangle =\langle x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}}, x_{k}-x_{k-\frac{2}{5}}\rangle +\langle x_{k-\frac{3}{5}}-x_{k-\frac{2}{5}}, x_{k-\frac{2}{5}}-x\rangle \ge 0, \end{aligned}$$

where the inequality comes from Lemma 7 and (15).

For the second inner product, we can obtain

$$\begin{aligned} \langle x_{k-\frac{2}{5}}-x_{k-\frac{1}{5}}, x_{k}-x\rangle&=\langle x_{k-\frac{2}{5}}-x_{k-\frac{1}{5}}, x_{k}-x_{k-\frac{1}{5}}\rangle +\langle x_{k-\frac{2}{5}}-x_{k-\frac{1}{5}}, x_{k-\frac{1}{5}}-x\rangle \\&\ge \langle x_{k-\frac{2}{5}}-x_{k-\frac{1}{5}}, x_{k}-x_{k-\frac{1}{5}}\rangle \\&=(1-\lambda _k)\langle x_{k-\frac{2}{5}}-x_{k-\frac{1}{5}}, x_{k-\frac{3}{5}}-x_{k-\frac{1}{5}}\rangle \\&\ge 0, \end{aligned}$$

where the first inequality follows Lemma 7, the second equality comes from the definition of $x_k$, and the second inequality is from Lemma 7 and (14).

Thus

$$\begin{aligned} \Vert x_k-x\Vert ^2\le \Vert x_{k-\frac{1}{5}}-x\Vert ^2-\Vert x_k-x_{k-\frac{1}{5}}\Vert ^2 \le \Vert x_{k-\frac{1}{5}}-x\Vert ^2. \end{aligned}$$

(16)

Since the iteration points $x_i$ $(i=k-\frac{3}{5},k-\frac{2}{5},k-\frac{1}{5})$ are obtained by projecting on the closed convex sets, by Lemma 7, it results in

$$\begin{aligned} \Vert x_i-x\Vert ^2\le \Vert x_{i-\frac{1}{5}}-x\Vert ^2-\Vert x_i-x_{i-\frac{1}{5}}\Vert ^2. \end{aligned}$$

Thus

$$\begin{aligned} \Vert x_{k-\frac{1}{5}}-x\Vert ^2\le \Vert x_{k-\frac{2}{5}}-x\Vert ^2\le \Vert x_{k-\frac{3}{5}}-x\Vert ^2\le \Vert x_{k-\frac{4}{5}}-x\Vert ^2. \end{aligned}$$

(17)

From (16) and (17), we get that

$$\begin{aligned} \Vert x_k-x\Vert ^2\le \Vert x_{k-\frac{4}{5}}-x\Vert ^2. \end{aligned}$$

$\square$