Multi-block Bregman proximal alternating linearized minimization and its application to orthogonal nonnegative matrix factorization

Abstract

We introduce and analyze BPALM and A-BPALM, two multi-block proximal alternating linearized minimization algorithms using Bregman distances for solving structured nonconvex problems. The objective function is the sum of a multi-block relatively smooth function (i.e., relatively smooth by fixing all the blocks except one) and block separable (nonsmooth) nonconvex functions. The sequences generated by our algorithms are subsequentially convergent to critical points of the objective function, while they are globally convergent under the KL inequality assumption. Moreover, the rate of convergence is further analyzed for functions satisfying the Łojasiewicz’s gradient inequality. We apply this framework to orthogonal nonnegative matrix factorization (ONMF) that satisfies all of our assumptions and the related subproblems are solved in closed forms, where some preliminary numerical results are reported.

Appendix

Lemma 8.1

Let all assumptions of Theorem 4.4 be valid. Then, the following assertions hold:

• (i) $\mathbf{lim}_{k\rightarrow \infty }\mathbf{dist}\left( \varvec{x}^k,\omega (\varvec{x}^0)\right) =0$;

• (ii) $\omega (\varvec{x}^0)$ is a nonempty, compact, and connected set;

• (iii) the objective function $\varphi$ is finite and constant on $\omega (\varvec{x}^0)$.

Proof

Lemma 8.1(i) is a direct consequence of Theorem 4.4, and Lemma 8.1(ii) and Lemma 8.1(iii) can be proved in the same way as [23, Lemma 5(iii)-(iv)].

Lemma 8.2

Let all assumptions of Theorem 4.5 is satisfied. If $\varphi (\varvec{x}^k)>\varphi ^\star$, there exists $\varepsilon ,\eta >0$ and the desingularizing function $\psi$ such that

$$\begin{aligned} \psi '(\varphi (\varvec{x}^k)-\varphi ^\star )\mathbf{dist}(0,\partial \varphi (\varvec{x}^k))\ge 1 \quad \text { for } k\ge k_0. \end{aligned}$$

(8.1)

Proof

From Lemma 8.1(ii), the set of limit points $\omega (\varvec{x}^0)$ of ${(\varvec{x}^k)_{{k\in \mathbb {N}}}}$ is nonempty and compact and $\varphi$ is finite and constant on $\omega (\varvec{x}^0)$ due to Lemma 8.1(iii). Moreover, $\varphi (\varvec{x}^k)>\varphi ^\star$ and the sequence ${(\varphi (\varvec{x}^k))_{{k\in \mathbb {N}}}}$ is decreasing (Proposition 4.1(i)), i.e., there exist $\eta >0$ and $k_1\in \mathbb {N}$ such that $\varphi ^\star<\varphi (\varvec{x}^k)<\varphi ^\star +\eta$ for all $k\ge k_1$. For $\varepsilon >0$, Proposition 4.1(i) implies that there exists $k_2\in \mathbb {N}$ such that $\mathbf{dist}(\varvec{x}^k,\omega (\varvec{x}^0))<\varepsilon$ for $k\ge k_2$. Setting $k_0:=\mathbf{max}\{k_1,k_2\}$ and according to Fact 2.2, there exist $\varepsilon , \eta >0$ and a desingularization function $\psi$ such that for any element in

$$\begin{aligned} \{\varvec{x}^k~\mid ~\mathbf{dist}(\varvec{x}^k,\omega (\varvec{x}^0))<\varepsilon \}\cap [\varphi ^\star<\varphi (\varvec{x}^k)<\varphi ^\star +\eta ] \quad \text { for } k\ge k_0, \end{aligned}$$

the inequality (8.1) is valid.

We next present the proof of Theorem 4.7.

Proof of Theorem 4.7. The proof has two key parts.

In the first part, we show that there exist $c>0$ and $\overline{k}\in \mathbb {N}$ such that for all $k\ge \overline{k}$ the following inequalities hold for $i=1,\ldots ,N$:

$$\begin{aligned} \Vert x_i^k-x_i^\star \Vert \le \left\{ \begin{array}{ll} c \mathbf{max}\{1,\tfrac{\kappa }{1-\theta }\} \sqrt{{\mathcal {S}}_{k-1}} &{}~~~ \mathrm {if}\ \theta \in (0,1/2], \\ c \tfrac{\kappa }{1-\theta } {\mathcal {S}}_{k-1}^{1-\theta } &{}~~~ \mathrm {if}\ \theta \in (1/2,1). \end{array} \right. \end{aligned}$$

(8.2)

Let $\varepsilon >0$ be as described in (4.16) and $x^k\in \mathbf{B}(x^\star ; \varepsilon )$ for all $k\ge \tilde{k}$ and $\tilde{k}\in \mathbb {N}$. By the definitions of $a_k$ and $b_k$ in (4.15) and using (4.14), we get $a_{k+1}\le \tfrac{1}{2} a_k+b_k$ for all $k\ge \tilde{k}$. Since ${(\varphi (\varvec{x}^k))_{{k\in \mathbb {N}}}}$ is nonincreasing,

$$\begin{aligned} \sum _{j=k}^\infty a_{j+1}\le \tfrac{1}{2} \sum _{j=k}^\infty (a_j-a_{j+1}+a_{j+1})+ \tfrac{\widehat{c} N}{2}\sum _{j=k}^\infty \left( \Delta _j-\Delta _{j+1}\right) = \tfrac{1}{2}\sum _{j=k}^\infty a_{j+1}+\tfrac{1}{2} a_k+\tfrac{\widehat{c} N}{2} \Delta _k. \end{aligned}$$

Together with the arithmetic and quadratic mean inequalities, $\psi ({\mathcal {S}}_{k})\le \psi ({\mathcal {S}}_{k-1})$, and Proposition 4.1(i), this lead to

$$\begin{aligned} {\begin{matrix} \sum _{j=k}^\infty a_{j+1}&{}\le a_k+\widehat{c} N\Delta _k = \sum _{i=1}^N \Vert x_i^{k}-x_i^{k-1}\Vert +\widehat{c} N\psi ({\mathcal {S}}_k) \le \sqrt{N} \sqrt{\sum _{i=1}^N \Vert x_i^{k}-x_i^{k-1}\Vert ^2}+\widehat{c} N\psi ({\mathcal {S}}_k)\\ &{}\le \sqrt{2N}\mathbf{max}\{\tfrac{1}{\sqrt{\sigma _1}},\ldots ,\tfrac{1}{\sqrt{\sigma _N}}\} \sqrt{\sum _{i=1}^N {\mathbf {D}}_{h}(\varvec{x}^{k-1,i},\varvec{x}^{k-1,i-1})} +\widehat{c} N\psi ({\mathcal {S}}_k)\\ &{}\le \sqrt{\tfrac{2N}{\rho }} \mathbf{max}\{\tfrac{1}{\sqrt{\sigma _1}},\ldots ,\tfrac{1}{\sqrt{\sigma _N}}\} \sqrt{{\mathcal {S}}_{k-1}-{\mathcal {S}}_{k}}+\widehat{c} N\psi ({\mathcal {S}}_{k-1}). \end{matrix}} \end{aligned}$$

(8.3)

On the other hand, for $i=1,\ldots ,N$, we have

$$\begin{aligned} \Vert x_i^k-x_i^\star \Vert \le \Vert x_i^{k+1}-x_i^k\Vert +\Vert x_i^{k+1}-x_i^\star \Vert \le \ldots \le \sum _{j=k}^\infty \Vert x_i^{j+1}-x_i^j\Vert . \end{aligned}$$

This inequality, together with (8.3), yields

$$\begin{aligned} \sum _{i=1}^N\Vert x_i^k-x_i^\star \Vert \le \sqrt{\tfrac{2N}{\rho }} \mathbf{max}\{\tfrac{1}{\sqrt{\sigma _1}},\ldots ,\tfrac{1}{\sqrt{\sigma _N}}\} \sqrt{{\mathcal {S}}_{k-1}-{\mathcal {S}}_{k}}+\widehat{c} N \psi ({\mathcal {S}}_{k-1}), \end{aligned}$$

leading to

$$\begin{aligned} \Vert x_i^k-x_i^\star \Vert \le c \mathbf{max}\{\sqrt{{\mathcal {S}}_{k-1}},\psi ({\mathcal {S}}_{k-1})\}\quad i=1,\ldots ,N, \end{aligned}$$

(8.4)

where $c:=\sqrt{\tfrac{2N}{\rho }} \mathbf{max}\left\{\tfrac {1}{\sqrt{\sigma _1}},\ldots ,\tfrac {1}{\sqrt{\sigma _N}}\right\}+\widehat{c} N$ and $\psi (s):=\frac{\kappa }{1-\theta } s^{1-\theta }$. Let us consider the nonlinear equation

$$\begin{aligned} \sqrt{{\mathcal {S}}_{k-1}}-\frac{\kappa }{1-\theta } {\mathcal {S}}_{k-1}^{1-\theta }=0, \end{aligned}$$

which has a solution at ${\mathcal {S}}_{k-1}=\left( \tfrac {(1-\theta )}{\kappa }\right) ^{\tfrac{2}{1-2\theta }}$. Form the monotonicity of ${\mathcal {S}}_k$, there exists $\hat{k}\in \mathbb {N}$ such that for $k\ge \hat{k}$ (8.4) holds and

$$\begin{aligned} {\mathcal {S}}_{k-1}\le \left( \frac{1-\theta }{\kappa }\right) ^{\tfrac{2}{1-2\theta }}. \end{aligned}$$

We now consider two cases: (a) $\theta \in (0,1/2]$; (b) $\theta \in (1/2,1)$. In Case (a), if $\theta \in (0,1/2)$, then $\psi ({\mathcal {S}}_{k-1})\le \sqrt{{\mathcal {S}}_{k-1}}$. If $\theta =1/2$, then $\psi ({\mathcal {S}}_{k-1})=\tfrac{\kappa }{1-\theta }\sqrt{{\mathcal {S}}_{k-1}}$, i.e.,

$$\begin{aligned} \mathbf{max}\{\sqrt{{\mathcal {S}}_{k-1}},\psi ({\mathcal {S}}_{k-1})\}=\mathbf{max}\{1,\tfrac{\kappa }{1-\theta }\} \sqrt{{\mathcal {S}}_{k-1}}. \end{aligned}$$

Therefore, it holds that $\mathbf{max}\{\sqrt{{\mathcal {S}}_{k-1}},\psi ({\mathcal {S}}_{k-1})\}\le \mathbf{max}\{1,\tfrac{\kappa }{1-\theta }\} \sqrt{{\mathcal {S}}_{k-1}}$. In Case (b), we have that

$$\begin{aligned} \psi ({\mathcal {S}}_{k-1})\ge \sqrt{{\mathcal {S}}_{k-1}}, \end{aligned}$$

i.e., $\mathbf{max}\{\sqrt{{\mathcal {S}}_{k-1}},\psi ({\mathcal {S}}_{k-1})\}= \tfrac{\kappa }{1-\theta } {\mathcal {S}}_{k-1}^{1-\theta }$. Then, it follows from (8.4) that (8.2) holds for all $k\ge \overline{k}:=\mathbf{max}\{\tilde{k}, \hat{k}\}$.

In the second part of the proof, we will show the assertions in the statement of the theorem. For $({\mathcal {G}}_i^{k},\ldots ,{\mathcal {G}}_N^{k})\in \partial \varphi (\varvec{x}^k)$ as defined in Proposition 4.3, by Proposition 4.1(i), we infer

$$\begin{aligned} {\mathcal {S}}_{k-1}&-{\mathcal {S}}_{k}=\varphi (x^{k-1})-\varphi (x^{k}) \ge \rho \sum _{i=1}^N {\mathbf {D}}_{h}(x^{k-1,i},x^{k-1,i-1})\ge \frac{\rho }{2} \sum _{i=1}^N \sigma _i \Vert x_i^{k}-x_i^{k-1}\Vert ^2\\&\ge \frac{\rho }{2N}\min \{\sigma _1,\ldots ,\sigma _N\} \left( \sum _{i=1}^N \Vert x_i^{k}-x_i^{k-1}\Vert \right) ^2 \ge \frac{\rho }{2N\overline{c}^2}\min \{\sigma _1,\ldots ,\sigma _N\} \Vert ({\mathcal {G}}_i^{k},\ldots ,{\mathcal {G}}_N^{k})\Vert ^2\\&\ge \frac{\rho }{2N\overline{c}^2}\min \{\sigma _1,\ldots ,\sigma _N\}\mathbf{dist}(0,\partial \varphi (x^k))^2 \ge \frac{\rho }{2N\overline{c}^2\kappa ^2}\min \{\sigma _1,\ldots ,\sigma _N\} {\mathcal {S}}_{k-1}^{2\theta }=\widetilde{c}~ {\mathcal {S}}_{k-1}^{2\theta }, \end{aligned}$$

with $\widetilde{c}:=\frac{\rho }{2N\overline{c}^2\kappa ^2}\min \{\sigma _1,\ldots ,\sigma _N\}$ and for all $k\ge \overline{k}$. Hence, all assumptions of Fact 4.6 hold with $\alpha =2\theta$. Therefore, our results follows from this fact and (8.2). $\square$