Removing Mixture of Gaussian and Impulse Noise by Patch-Based Weighted Means

Abstract

We first establish a law of large numbers and a convergence theorem in distribution to show the rate of convergence of the non-local means filter for removing Gaussian noise. Based on the convergence theorems, we propose a patch-based weighted means filter for removing an impulse noise and its mixture with a Gaussian noise by combining the essential idea of the trilateral filter and that of the non-local means filter. Experiments show that our filter is competitive compared to recently proposed methods. We also introduce the notion of degree of similarity to measure the impact of the similarity among patches on the non-local means filter for removing a Gaussian noise, as well as on our new filter for removing an impulse noise or a mixed noise. Using again the convergence theorem in distribution, together with the notion of degree of similarity, we obtain an estimation for the PSNR value of the denoised image by the non-local means filter or by the new proposed filter, which is close to the real PSNR value.

Appendix

In this appendix we will prove Theorems 1 and 2.

1.1 Convergence Theorems for Random Weighted Means

We first show a Marcinkiewicz law of large numbers (Theorem 3) and a convergence theorem in distribution for random weighted means (Theorem 4) for l-dependent random variables, which we will use to prove Theorems 1 and 2.

Theorem 3

Let $\{(a_k, v_k)\}$ be a sequence of l-dependent identically distributed random variables, with $\mathbb {E}|a_1|^p<\infty$ and $\mathbb {E}|a_1v_1|^p<\infty$ for some $p\in [1,2),$ and $\mathbb {E} a_1\ne 0$. Then

$$\begin{aligned} \frac{\sum _{k=1}^na_kv_k}{\sum _{k=1}^na_k}-\frac{\mathbb {E}a_1v_1}{\mathbb {E}a_1}=o(n^{-(1-1/p)}) \quad \text{ almost } \text{ surely }. \end{aligned}$$

We need the following lemma to prove it.

Lemma 2

[24] If $\{X_n\}$ are l-dependent and identically distributed random variables with $\mathbb {E}X_1=0$ and $\mathbb {E}|X_1|^p<\infty$ for some $p\in [1,2)$, then

$$\begin{aligned} \lim _{n\rightarrow \infty }\frac{X_1+ \cdots +X_n}{n^{1/p} }=0 \quad \text{ almost } \text{ surely }. \end{aligned}$$

This lemma is a direct consequence of Marcinkiewicz law of large numbers for independent random variables (see e.g. [9], p. 118), since for all $k\in \{1,\ldots ,l+1\}, \{X_{i(l+1)+k}:i\ge 0\}$ is a sequence of i.i.d. random variables, and for each positive integer n, we have

$$\begin{aligned} X_1+ \cdots +X_n=\sum _{k=1}^{l+1}\sum _{i=0}^{m-1}X_{i(l+1)+k}+\sum _{1\le k\le k_0}X_{m(l+1)+k}, \end{aligned}$$

where $m,k_0$ are positive integers determined by $n=m(l+1)+k_0, 0\le k_0 \le l$.

   Proof of Theorem 3

Notice that

$$\begin{aligned} n \bigg (\frac{\sum _{k=1}^na_kv_k}{\sum _{k=1}^na_k}-\frac{\mathbb {E}a_1v_1}{\mathbb {E}a_1}\bigg )= \frac{n}{(\sum _{k=1}^na_k)} \frac{1}{\mathbb {E}a_1} \sum _{k=1}^n a_k z_k, \end{aligned}$$

where

$$\begin{aligned} z_k=v_k \mathbb {E}a_1-\mathbb {E}a_1 v_1. \end{aligned}$$

Since $\mathbb {E}|a_1|\le (\mathbb {E}|a_1|^p)^{1/p}<\infty$, by Lemma 2 with $p=1$, we have

$$\begin{aligned} \lim _{n\rightarrow \infty }\frac{\sum _{k=1}^na_k}{n}=\mathbb {E}a_1 \quad \text{ almost } \text{ surely }. \end{aligned}$$

Since $\mathbb {E}a_1z_1=0$, and $\mathbb {E}|a_1z_1|^p<\infty$, again by Lemma 2, we get

$$\begin{aligned} \lim _{n\rightarrow \infty }\frac{\sum _{k=1}^na_kz_k}{n^{1/p}}=0 \quad \text{ almost } \text{ surely }. \end{aligned}$$

Thus the conclusion follows. $\square$

Theorem 4

Let $\{(a_k,v_k)\}$ be a stationary sequence of l-dependent and identically distributed random variables with $\mathbb {E}a_1\ne 0, \mathbb {E}a_1^2<\infty ,$ and $\mathbb {E}(a_1v_1)^2<\infty$. Then

$$\begin{aligned} {\sqrt{n}} \bigg (\frac{\sum _{k=1}^na_kv_k}{\sum _{k=1}^na_k}-\frac{\mathbb {E}a_1v_1}{\mathbb {E}a_1}\bigg )\mathop {\rightarrow }\limits ^{d} N(0,c^2), \end{aligned}$$

that is,

$$\begin{aligned} \lim _{n\rightarrow \infty }\mathbb {P}\bigg \{\frac{\sqrt{n}}{c} \bigg (\frac{\sum _{k=1}^na_kv_k}{\sum _{k=1}^na_k}-\frac{\mathbb {E}a_1v_1}{\mathbb {E}a_1}\bigg )\le z\bigg \}=\varPhi (z), \quad z\in \mathbb {R}, \end{aligned}$$

where $\varPhi (z)$ is the cumulative distribution function of the standard normal distribution,

$$\begin{aligned} \varPhi (z)=\frac{1}{\sqrt{2\pi }}\int _{-\infty }^z e^{-\frac{t^2}{2}}dt, \end{aligned}$$

and

$$\begin{aligned} c= \frac{1}{(\mathbb {E}a_1)^2}\sqrt{\mathbb {E}(a_1z_1)^2+2\lambda } \end{aligned}$$

(35)

with $\lambda =\sum _{k=1}^l\mathbb {E}a_1z_1a_{1+k}z_{1+k}, \; z_k=v_k \mathbb {E}a_1-\mathbb {E}a_1 v_1$.

We need the following lemma to prove the theorem.

Lemma 3

[36] Let $\{X_n\}$ be a stationary sequence of l-dependent and identically distributed random variables with $\mathbb {E}X_1=0$ and $\mathbb {E}X_1^2<\infty$. Set $S_n=X_1+ \cdots +X_n (n\ge 1)$,

$$\begin{aligned} c_1= \mathbb {E}X_1^2 +2\sum _{k=1}^l\mathbb {E}X_1X_{1+k}, \quad \text{ and } \quad c_2=2\sum _{k=1}^lk\mathbb {E}X_1X_{1+k}. \end{aligned}$$

Then var$(S_n)=c_1n-c_2$ for $n > l$, and as $n\rightarrow \infty$,

$$\begin{aligned} \frac{S_n}{\sqrt{c_1 n}}\mathop {\rightarrow }\limits ^{d}N(0,1). \end{aligned}$$

   Proof of Theorem 4

As in the proof of Theorem 3, we have

$$\begin{aligned} {\sqrt{n}} \bigg (\frac{\sum _{k=1}^na_kv_k}{\sum _{k=1}^na_k}-\frac{\mathbb {E}a_1v_1}{\mathbb {E}a_1}\bigg )= \frac{n}{(\sum _{k=1}^na_k)} \frac{1}{\mathbb {E}a_1} \frac{\sum _{k=1}^n a_k z_k}{\sqrt{n}}, \end{aligned}$$

where

$$\begin{aligned} z_k=v_k \mathbb {E}a_1-\mathbb {E}a_1 v_1. \end{aligned}$$

Notice that the l-dependence of $\{(a_k,v_k)\}$ and the stationarity imply those of $\{(a_k,z_k)\}$. Therefore by Lemma 3, we get

$$\begin{aligned} \frac{\sum _{k=1}^n a_k z_k}{\sqrt{n}c_0}\rightarrow N(0,1), \end{aligned}$$

where

$$\begin{aligned} c_0=\sqrt{\mathbb {E}(a_1z_1)^2+2\lambda }, \quad \text{ with }\, \lambda =\sum _{k=1}^l\mathbb {E}a_1z_1a_{1+k}z_{1+k}. \end{aligned}$$

Since $\mathbb {E}|a_1|\le (\mathbb {E}|a_1|^p)^{1/p}$, by Lemma 2 with $p=1$, we obtain

$$\begin{aligned} \lim _{n\rightarrow \infty }\frac{\sum _{k=1}^na_k}{n}=\mathbb {E}a_1 \quad \text{ almost } \text{ surely }. \end{aligned}$$

Thus the conclusion follows with $c=c_0 /(\mathbb {E}a_1)^2$. $\square$

1.2 Proofs of Theorems 1 and 2

We now come to the proofs of Theorems 1 and 2, using Theorems 3 and 4.

For Theorem 1, we need to prove that for any $\epsilon \in (0,\frac{1}{2}]$, as $n\rightarrow \infty$,

$$\begin{aligned} \frac{\sum _{k=1}^n{w^{0}}(i,j_k)v(j_k)}{\sum _{k=1}^n{w^{0}}(i,j_k)}-u(i)=o(n^{-(\frac{1}{2}-\epsilon )}) \quad \text{ almost } \text{ surely }, \end{aligned}$$

where

$$\begin{aligned} w^{0}(i,j_k)=e^{-\Vert v(\mathcal {N}^{0}_{i})-v(\mathcal {N}^{0}_{j_k})\Vert ^2/(2\sigma _r^2)}. \end{aligned}$$

We will apply Theorem 3 to prove this. Note that the sequence $\{{w^{0}}(i,j_k),v(j_k)\}$ ${(k=1,2,\ldots ,n)}$ is usually not l-dependent, since the central random variable $v(\mathcal {N}^{0}_{i})$ is contained in all the terms. To make use of Theorem 3, we first take a fixed vector to replace the central random variable.

   Proof of Theorem 1

Fix $x\in \mathbb {R}^{|\mathcal {N}_i^0|}$. Let

$$\begin{aligned} a_{k}=w^0(x,j_k)=e^{-\Vert x-v(\mathcal {N}^{0}_{j_k})\Vert ^2/(2\sigma _r^2)}. \end{aligned}$$

Then $a_{k}$ and $v(j_k)$ are independent since $j_k\not \in \mathcal {N}^{0}_{j_k}$, so that

$$\begin{aligned} \frac{\mathbb {E}a_{k}v(j_k)}{\mathbb {E}a_{k}}=\mathbb {E}v(j_k)=u(j_k)=u(i). \end{aligned}$$

By Lemma 1, the sequence $\{v(\mathcal {N}_{j_k})\}$ is l-dependent for $l=(2d-1)^2 -1$; thus the sequence $\{\big (a_{k},v(j_k)\big )\}$ is also l-dependent. Since $v=u+\eta$, with the range of u being bounded and $\eta$ being Gaussian, we have $\mathbb {E}|v(j_k)|^p<\infty$ for $p\in [1,2)$. (In fact, it holds for all $p\ge 1$.) Hence $\mathbb {E}|a_{k}v(j_k)|^p<\infty$, as $a_k\le 1$.

Applying Theorem 3, we have, for any fixed $x=v(\mathcal {N}^0_i)\in \mathbb {R}^{|\mathcal {N}_i^0|}$ and any positive integer $k_0$,

$$\begin{aligned} \frac{\sum _{k=k_0}^n{w^{0}}(x,j_k)v(j_k)}{\sum _{k=k_0}^n{w^{0}}(x,j_k)}-u(i)=o(n^{-(1-1/p})\quad \text{ almost } \text{ surely }. \end{aligned}$$

(36)

Let $k_0 > l$, so that $v(\mathcal {N}_i^0)$ is independent of $v(\mathcal {N}^{0}_{j_k})$ for all $k\ge k_0$. By Fubini’s theorem, we can replace $w^{0}(x,j_k)$ in (36) by

$$\begin{aligned} w^{0}(i,j_k)=e^{-\Vert v(\mathcal {N}^{0}_{i})-v(\mathcal {N}^{0}_{j_k})\Vert ^2/(2\sigma _r^2)}. \end{aligned}$$

That is,

$$\begin{aligned} \frac{\sum _{k=k_0}^n{w^{0}}(i,j_k)v(j_k)}{\sum _{k=k_0}^n{w^{0}}(i,j_k)}-u(i)=o(n^{-(1-1/p)})\quad \text{ almost } \text{ surely }. \end{aligned}$$

(37)

To prove the theorem, we need to estimate the difference between the left-hand sides of (8) and (37). Let

$$\begin{aligned} A_0= & {} \sum _{k=1}^{k_0-1}{w^{0}}(i,j_k)v(j_k), \quad A_n=\sum _{k=k_0}^n{w^{0}}(i,j_k)v(j_k),\\ B_0= & {} \sum _{k=1}^{k_0-1}{w^{0}}(i,j_k), \quad B_n=\sum _{k=k_0}^n{w^{0}}(i,j_k). \end{aligned}$$

Then as before, fixing $x\in \mathbb {R}^{|\mathcal {N}_i^0|}$, applying Theorem 3 with $p=1$ and Fubini’s theorem, and replacing x by $v(\mathcal {N}_i^0)$, we obtain

$$\begin{aligned} \lim _{n\rightarrow \infty } \frac{A_n}{n}=\mathbb {E}{w^{0}}(i,j_k)v(j_k), \quad \text{ and } \lim _{n \rightarrow \infty } \frac{B_n}{n}=\mathbb {E}{w^{0}}(i,j_k). \end{aligned}$$

Using this and the fact that

$$\begin{aligned} \frac{A_0+A_n}{B_0+B_n}-\frac{A_n}{B_n}=\frac{A_0B_n-A_nB_0}{B_n(B_0+B_n)}, \end{aligned}$$

we see that

$$\begin{aligned} \bigg |\frac{\sum _{k=k_0}^n{w^{0}}(i,j_k)v(j_k)}{\sum _{k=k_0}^n{w^{0}}(i,j_k)} -\frac{\sum _{k=1}^n{w^{0}}(i,j_k)v(j_k)}{\sum _{k=1}^n{w^{0}}(i,j_k)}\bigg |=O\left( \frac{1}{n}\right) \quad \text{ almost } \text{ surely }. \end{aligned}$$

(38)

Therefore, (37) implies that

$$\begin{aligned} \frac{\sum _{k=1}^n{w^{0}}(i,j_k)v(j_k)}{\sum _{k=1}^n{w^{0}}(i,j_k)}-u(i)=o(n^{-(1-1/p)})\quad \text{ almost } \text{ surely }. \end{aligned}$$

(39)

As (39) holds for any $p \in [1,2)$, we see that (8) holds for all $\epsilon \in (0,\frac{1}{2}]$. $\square$

We will prove Theorem 2, which demonstrates that as $n\rightarrow \infty$,

$$\begin{aligned} \sqrt{n} \left( \frac{\sum _{k=1}^n{w^{0}}(i,j_k)v(j_k)}{\sum _{k=1}^n{w^{0}}(i,j_k)}-u(i)\right) \mathop {\rightarrow }\limits ^{d} \mathcal {L}, \end{aligned}$$

where

$$\begin{aligned} w^{0}(i,j_k)=e^{-\Vert v(\mathcal {N}^{0}_{i})-v(\mathcal {N}^{0}_{j_k})\Vert ^2/(2\sigma _r^2)}, \end{aligned}$$

and $\mathcal {L}$ is a mixture of centered Gaussian laws in the sense that it has a density of the form (15).

   Proof of Theorem 2

The procedure of the proof is similar to that of the proof of Theorem 1. Fix $x\in \mathbb {R}^{|\mathcal {N}_i^0|}$, and set

$$\begin{aligned} a_{k}=w^0(x,j_k)=e^{-\Vert x-v(\mathcal {N}^{0}_{j_k})\Vert ^2/(2\sigma _r^2)}, \quad k=1,2,\ldots ,n. \end{aligned}$$

Then $a_{k}$ and $v(j_k)$ are independent, ${\mathbb {E}a_{k}v(j_k)}/{\mathbb {E}a_{k}}=\mathbb {E}v(j_k)=u(i)$, and $\{(a_{k},v(j_k))\}$ is a sequence of l-dependent and identically distributed random vectors with $l= (2d-1)^2 -1$, and $\mathbb {E}|a_{k}v(j_k)|^2\le \mathbb {E}|v(j_k)|^2<\infty$. Hence applying Theorem 4, we get, for any fixed x and any positive integer $k_0$,

$$\begin{aligned} Z_n(x):={\sqrt{n}} \bigg (\frac{\sum _{k=k_0}^n{w^{0}}(x,j_k)v(j_k)}{\sum _{k=k_0}^n{w^{0}}(x,j_k)}-u(i)\bigg )\mathop {\rightarrow }\limits ^{d} N(0,c_x^2), \end{aligned}$$

where $c_x>0$ will be calculated by the end of the proof. This means that for any $t\in \mathbb {R}$,

$$\begin{aligned} \lim _{n\rightarrow \infty }\mathbb {P}(Z_n(x) \le t)=\int _{-\infty }^t\frac{1}{\sqrt{2\pi }c_x}e^{-\frac{z^2}{2c_x^2}}dz. \end{aligned}$$

Let $k_0>l$ be the positive integer such that $v(\mathcal {N}_i^0)$ is independent of $v(\mathcal {N}^{0}_{j_k})$ for all $k\ge k_0$. Then by Fubini’s theorem and Lebesgue’s dominated convergence theorem, we have

$$\begin{aligned} \lim _{n\rightarrow \infty }\mathbb {P}\Big (Z_n\big (v(\mathcal {N}^0_i)\big ) \le t\Big )=\int _{-\infty }^t f(z) dz, \end{aligned}$$

where

$$\begin{aligned} f(z)=\int _{\mathbb {R}^{|\mathcal {N}_i^0|}}\frac{1}{\sqrt{2\pi }c_x}e^{-\frac{z^2}{2c_x^2}}\mu (dx), \end{aligned}$$

with $\mu$ being the law of $v(\mathcal {N}_i^0)$. In other words,

$$\begin{aligned} {\sqrt{n}} \bigg (\frac{\sum _{k=k_0}^n{w^{0}}(i,j_k)v(j_k)}{\sum _{k=k_0}^n{w^{0}}(i,j_k)}-u(i)\bigg )\mathop {\rightarrow }\limits ^{d} \mathcal {L}, \end{aligned}$$

where $\mathcal {L}$ is the law with density f. This together with (38) prove the equation (15) of Theorem 2.

We now turn to the calculation of $c_x$. Let $v_k=v(j_k)$ and $z_k=v_k \mathbb {E}a_1-\mathbb {E}(a_1 v_1)$. Because of the independence of $a_1$ and $v_1$, we get $z_k=(v_k-\mathbb {E} v_1)\mathbb {E}a_1$. Then, it follows that

$$\begin{aligned} \mathbb {E} (a_1z_1)^2=\mathbb {E} (a_1^2 (v_1-\mathbb {E} v_1)^2)\mathbb {E} ^2a_1=\mathbb {E} (a_1^2) \mathbb {E} ^2( a_1) \mathbb {E} (v_1-\mathbb {E} v_1)^2, \end{aligned}$$

and

$$\begin{aligned} \mathbb {E} (a_1z_1a_kz_k)= & {} \mathbb {E}( a_1a_k (v_1- \mathbb {E}v_1)(v_k-\mathbb {E} v_1)\mathbb {E}^2a_1)\\= & {} \mathbb {E} ^2(a_1) \mathbb {E} (a_1a_k(v_1-\mathbb {E} v_1)(v_k-\mathbb {E} v_k)). \end{aligned}$$

Note that if $(a_1,a_k)$ is independent of $(v_1,v_k)$, it holds that

$$\begin{aligned} \mathbb {E} (a_1z_1a_kz_k)=\mathbb {E} ^2(a_1) \mathbb {E}(a_1a_k) \mathbb {E} (v_1-\mathbb {E} v_1) \mathbb {E}(v_k-\mathbb {E} v_k)=0, \end{aligned}$$

(40)

by the independence of $v_1$ and $v_k$. If $v_1$ is not contained in $v(\mathcal {N}^{0}_{j_k})$, then $(a_1,a_k)$ is independent of $(v_1,v_k)$. Notice that according to the order of $I_i$ defined in Sect. 2.2, when $k>d^2$, $v_1$ is not contained in $v(\mathcal {N}^{0}_{j_k})$, so that (40) holds; therefore by Theorem 4,

$$\begin{aligned} c_x= & {} \sqrt{\mathbb {E}(a_1z_1)^2+2\lambda }/(\mathbb {E}a_1)^2 \\= & {} \frac{1}{\mathbb {E}a_1}\sqrt{\mathbb {E} (a_1^2) \mathbb {E} (v_1-\mathbb {E} v_1)^2 +2\sum \nolimits _{k=2}^{d^2} \mathbb {E} (a_1a_k(v_1-\mathbb {E} v_1)(v_k-\mathbb {E} v_k))}. \end{aligned}$$

We finally give an approximation of $c_x$. Recall that $a_{k}=e^{-\Vert x-v(\mathcal {N}^{0}_{j_k})\Vert ^2/(2\sigma _r^2)}$, and $v_k=v(j_k)$. Let $\mathcal {T}(j)=j-j_1+j_k$ be the translation mapping $j_1$ to $j_k$ (thus mapping $\mathcal {N}^{0}_{j_1}$ onto $\mathcal {N}^{0}_{j_k}$).

If $v( j_1)$ is not contained in $v(\mathcal {N}^{0}_{j_k})$, we have already seen that $\mathbb {E} (a_1z_1a_kz_k)= 0$. If $v(j_1)$ is contained in $v(\mathcal {N}^{0}_{j_k})$, to make $(a_1,a_k)$ independent of $(v_1,v_k)$, we can remove $v(j_1)$ from $v(\mathcal {N}^{0}_{j_k})$ and the corresponding term $v(\mathcal {T}^{-1}(j_1))$ from $v(\mathcal {N}^{0}_{j_1})$; remove $v(j_k)$ from $v(\mathcal {N}^{0}_{j_1})$ and the corresponding term $v(\mathcal {T}(j_k))$from $v(\mathcal {N}^{0}_{j_k})$. The obtained values of $a_1,a_k$ are very close to the initial values of $a_1,a_k$ respectively. Hence, we can consider that $\mathbb {E} (a_1z_1a_kz_k)\approx 0$. Therefore

$$\begin{aligned} c_x \approx \frac{\sqrt{\mathbb {E} (a_1z_1)^2}}{\mathbb {E} ^2a_1}=\frac{\sqrt{\mathbb {E} a_1^2}}{\mathbb {E} a_1}\sigma =\sigma \frac{\sqrt{\int _{R^m}e^{-\Vert x-v\Vert ^2/\sigma _r^2}\mu (dv)}}{\int _{R^m}e^{-\Vert x-v\Vert ^2/2\sigma _r^2}\mu (dv)}, \end{aligned}$$

where $v=v(\mathcal {N}^{0}_{j_k})$ and $m=|\mathcal {N}_{j_k}^0|$. Recall that $\mu (dv)$ is the law of $v(\mathcal {N}^{0}_{i})$, so it is also the law of $v(\mathcal {N}^{0}_{j_k})$. Let

$$\begin{aligned} \nu =\mathbb {E}(v(\mathcal {N}^{0}_{i})), \end{aligned}$$

then $v\sim N(\nu , \sigma ^2Id_m)$, where $Id_m$ denotes the identity matrix of size $m\times m$. Since

$$\begin{aligned} \int _{R^m}e^{-\Vert x-v\Vert ^2/a}\mu (dv) =\frac{1}{(\sqrt{2/a+1/\sigma ^2}\sigma )^m}\exp (-\Vert x-\nu \Vert ^2/(a+2\sigma ^2)), \end{aligned}$$

we get

$$\begin{aligned} c_x=\sigma ^{m/2+1}\left( \frac{(\sigma ^2+\sigma _r^2)^2}{\sigma ^2\sigma _r^2(2\sigma ^2+\sigma _r^2)}\right) ^{m/4} \exp \left( \frac{\sigma ^2}{2(\sigma _r^2+\sigma ^2)(\sigma _r^2+2\sigma ^2)}\Vert x-\nu \Vert ^2\right) . \end{aligned}$$

This ends the proof of Theorem 2. $\square$