Complexity of unconstrained $L_2-L_p$ minimization

Abstract

We consider the unconstrained $L_q$-$L_p$ minimization: find a minimizer of $\Vert Ax-b\Vert ^q_q+\lambda \Vert x\Vert ^p_p$ for given $A \in R^{m\times n}$, $b\in R^m$ and parameters $\lambda >0$, $p\in [0, 1)$ and $q\ge 1$. This problem has been studied extensively in many areas. Especially, for the case when $q=2$, this problem is known as the $L_2-L_p$ minimization problem and has found its applications in variable selection problems and sparse least squares fitting for high dimensional data. Theoretical results show that the minimizers of the $L_q$-$L_p$ problem have various attractive features due to the concavity and non-Lipschitzian property of the regularization function $\Vert \cdot \Vert ^p_p$. In this paper, we show that the $L_q$-$L_p$ minimization problem is strongly NP-hard for any $p\in [0,1)$ and $q\ge 1$, including its smoothed version. On the other hand, we show that, by choosing parameters $(p,\lambda )$ carefully, a minimizer, global or local, will have certain desired sparsity. We believe that these results provide new theoretical insights to the studies and applications of the concave regularized optimization problems.

Appendix

1.1 Proof of Lemma 4

First it is easy to see that when $p=0$, $g(z)$ has a unique minimizer at $z = 1$, and the optimal value is $\frac{1}{2}$. Now we consider the case when $p\ne 0$. Note that $g(z)>g(0)=1$ for all $z<0$, and $g(z)>g(1)=\frac{1}{2}$ for all $z>1$. Therefore the minimum point must lie within $[0,1]$.

To optimize $g(z)$ on $[0,1]$, we check its first derivative

$$\begin{aligned} g^{\prime }(z)=-q(1-z)^{q-1}+\frac{pz^{p-1}}{2}. \end{aligned}$$

(20)

We have $g^{\prime }(0^+)=+\infty$ and $g^{\prime }(1)=\frac{p}{2}>0$. Therefore, if function $g(z)$ has at most two stationary points in (0,1), the first one must be a local maximum and the second one must be the unique global minimum and the minimum value $c(p,q)$ must be less than $\frac{1}{2}$.

Now we check the possible stationary points of $g(z)$. Consider solving $g^{\prime }(z)=-q(1-z)^{q-1}+\frac{p z^{p-1}}{2}=0$. We get $z^{1-p}(1-z)^{q-1}=\frac{p}{2q}$.

Define $h(z)=z^{1-p}(1-z)^{q-1}$. We have

$$\begin{aligned} h^{\prime }(z)=h(z)\left(\frac{1-p}{z}-\frac{q-1}{1-z}\right). \end{aligned}$$

Note that $\frac{1-p}{z}-\frac{q-1}{1-z}$ is decreasing in $z$ and must have a root on $(0,1)$. Therefore, there exists a point $\bar{z}\in (0,1)$ such that $h^{\prime }(z)>0$ for $z<\bar{z}$ and $h^{\prime }(z)<0$ for $z>\bar{z}$. This implies that $h(z)=\frac{p}{2q}$ can have at most two solutions in $(0,1)$, i.e., $g(z)$ can have at most two stationary points. By the previous discussions, the lemma holds. $\square$

1.2 Proof of Theorem 6

We again consider the same $3$-partition problem, we claim that it can be reduced to a minimization problem in form (6). Again, it suffices to only consider the case when $\lambda = \frac{1}{2}$ (Here we consider the hardness result for any given $\epsilon > 0$. Note that after the scaling, $\epsilon$ may have changed). Consider:

$$\begin{aligned}&\text{ Minimize}_x \quad {\displaystyle P_{\epsilon }(x)=\sum _{j=1}^m\left|\sum _{i=1}^n a_{i}x_{ij}-B\right|^q + \sum _{i=1}^n \left|\sum _{j=1}^m x_{ij}-1\right|^q }\nonumber \\&\qquad \qquad \qquad \quad \quad + \frac{1}{2} \sum _{i=1}^n\sum _{j=1}^m (|x_{ij}|+\epsilon )^p. \end{aligned}$$

(21)

We have

$$\begin{aligned} \text{ Minimize}_x P_{\epsilon }(x)&\ge \text{ Minimize}_x\ \sum _{i=1}^n \left|\sum _{j=1}^m x_{ij}-1\right|^q + \frac{1}{2} \sum _{i=1}^n\sum _{j=1}^m (|x_{ij}|+\epsilon )^p\\&= \sum _{i=1}^n \text{ Minimize}_x \ \left|\sum _{j=1}^m x_{ij}-1\right|^q + \frac{1}{2} \sum _{j=1}^m (|x_{ij}|+\epsilon )^p\\&= n\cdot \text{ Minimize}_{z}\ |1-z|^q+\frac{1}{2} (|z|+\epsilon )^p+ \frac{(m-1)}{2} {\epsilon }^p. \end{aligned}$$

The last equality comes from the submodularity of the function $(x+\epsilon )^p$ and the fact that one can always choose only one of $x_{ij}$ to be nonzero in each set such that the equality holds. Consider function $g_{\epsilon }(z)=|1-z|^q+\frac{1}{2} (|z|+\epsilon )^p$. Similar to Lemma 4, one can prove that $g_{\epsilon }(z)$ has a unique minimizer in $[0,1]$. Denote this minimum value by $c(p, q, \epsilon )$, we know that $P_{\epsilon }(x)\ge n c(p, q, \epsilon )$. Then we can argue that the 3-partition problem has a solution if and only if $P_{\epsilon }(x)= n c(p, q, \epsilon )$. Therefore Theorem 6 holds.$\square$