Weak Versus Strong Dominance of Shrinkage Estimators

Abstract

We consider the estimation of the mean of a multivariate normal distribution with known variance. Most studies consider the risk of competing estimators, that is the trace of the mean squared error matrix. In contrast we consider the whole mean squared error matrix, in particular its eigenvalues. We prove that there are only two distinct eigenvalues and apply our findings to the James–Stein and the Thompson class of estimators. It turns out that the famous Stein paradox is no longer a paradox when we consider the whole mean squared error matrix rather than only its trace.

Appendices

Appendices

A Some Results Involving Idempotent Matrices

Our first result is not new.

Lemma 1

Let A be a symmetric idempotent $p\times p$ matrix of rank $r(A)=r$. Then, $r(I_p-A)=p-r$ and $A(I_p-A)=0$. Next, let

$$\begin{aligned} V=\nu _1 A + \nu _2(I_p-A). \end{aligned}$$

Then the eigenvalues of V are $\nu _1$ (multiplicity r) and $\nu _2$ (multiplicity $p-r$), its determinant is $|V|=\nu _1^{r}\nu _2^{p-r}$, and its inverse is $V^{-1}=(1/\nu _1) A + (1/\nu _2)(I_p-A)$ when $\nu _1\ne 0$ and $\nu _2\ne 0$.

Proof

This is a simple version of a much more general result, see Abadir and Magnus (2005, Exercises 8.72 and 8.73). $\square$

Now let’s consider an extension where the $p\times p$ matrix V is partitioned into blocks of dimensions $p_1$ and $p_2$ as follows:

$$\begin{aligned} V =\begin{pmatrix} \nu _1J_1 + \nu _2(I_{p_1}-J_1) &{} \gamma \imath _1^{}\imath _2' \\ \gamma \imath _2^{}\imath _1' &{} \xi _1J_2 + \xi _2(I_{p_2}-J_2) \end{pmatrix}, \end{aligned}$$

(32)

where $\imath _1^{}=(1,1,\ldots ,1)'$ and $\imath _2^{}=(1,1,\ldots ,1)'$ have dimensions $p_1$ and $p_2$ respectively, and $J_1=\imath _1^{}\imath _1'/p_1$ and $J_2=\imath _2^{}\imath _2'/p_2$.

In the special case $p_1=p_2=p$ we can write $V= V_1 \otimes J + V_2\otimes (I_p-J)$, where

$$\begin{aligned} V_1=\begin{pmatrix} \nu _1 &{} \gamma p \\ \gamma p &{} \xi _1 \end{pmatrix}, \qquad V_2= \begin{pmatrix} \nu _2 &{} 0 \\ 0 &{} \xi _2 \end{pmatrix}, \qquad J=(1/p)\,\imath \imath ', \end{aligned}$$

which implies that the eigenvalues of V are given by $\nu _2$ ($p-1$ times), $\xi _2$ ($p-1$ times), and the two eigenvalues of $V_1$ (Magnus 1982, Lemma 2.1).

When $p_1\ne p_2$ we cannot write V in terms of Kronecker matrices, but the result is still essentially the same.

Lemma 2

The eigenvalues of V are $\nu _2$ ($p_1-1$ times), $\xi _2$ ($p_2-1$ times) and two additional eigenvalues $\nu _1^*$ and $\xi _1^*$ given by

$$\begin{aligned} \frac{\nu _1+\xi _1}{2} \pm \frac{1}{2}\sqrt{(\nu _1-\xi _1)^2 + 4\gamma ^2 p_1p_2}. \end{aligned}$$

The sum of the eigenvalues is

$$\begin{aligned} {\text {tr}}V= \nu _1 + (p_1-1)\nu _2 + \xi _1 + (p_2-1)\xi _2, \end{aligned}$$

and V is positive semidefinite if and only if $\nu _1$, $\nu _2$, $\xi _1$, and $\xi _2$ are all $\ge 0$ and in addition $\nu _1\xi _1\ge \gamma ^2p_1p_2$.

Proof

We write

$$\begin{aligned} V-\lambda I_p = \begin{pmatrix} \bar{\nu }_1 J_1 + \bar{\nu }_2 (I_{p_1}-J_1) &{} \gamma \imath _1^{}\imath _2' \\ \gamma \imath _2^{}\imath _1' &{} \bar{\xi }_1 J_2 + \bar{\xi }_2 (I_{p_2}-J_2) \end{pmatrix}, \end{aligned}$$

where

$$\begin{aligned} \bar{\nu }_1=\nu _1-\lambda , \qquad \bar{\nu }_2=\nu _2-\lambda , \qquad \bar{\xi }_1=\xi _1-\lambda , \qquad \bar{\xi }_2=\xi _2-\lambda . \end{aligned}$$

The determinant is

$$\begin{aligned} |V-\lambda I_p|&= |\bar{\nu }_1J_1+\bar{\nu }_2(I_{p_1}-J_1)| \\&\quad \times |\bar{\xi }_1J_2+\bar{\xi }_2(I_{p_2}-J_2) - \gamma ^2\imath _2^{}\imath _1'[\bar{\nu }_1J_1+\bar{\nu }_2(I_{p_1}-J_1)]^{-1}\imath _1^{}\imath _2'|\\&= \bar{\nu }_1 {\bar{\nu }_2}^{p_1-1} \left| \bar{\xi }_1J_2+\bar{\xi }_2(I_{p_2}-J_2) - \gamma ^2\imath _2^{}\imath _1' \left( \frac{1}{\bar{\nu }_1}J_1+\frac{1}{\bar{\nu }_2}(I_{p_1}-J_1)\right) \imath _1^{}\imath _2'\right| \\&=\bar{\nu }_1{\bar{\nu }_2}^{p_1-1} |\bar{\xi }_1 J_2+\bar{\xi }_2 (I_{p_2}-J_2) - (\gamma ^2/\bar{\nu }_1)p_1p_2J_2| \\&=\bar{\nu }_1{\bar{\nu }_2}^{p_1-1} \left| \frac{\bar{\nu }_1\bar{\xi }_1 - \gamma ^2p_1p_2}{\bar{\nu }_1}J_2 + \bar{\xi }_2 (I_{p_2}-J_2)\right| \\&={\bar{\nu }_2}^{p_1-1}{\bar{\xi }_2}^{p_2-1}\left( \bar{\nu }_1\bar{\xi }_1 - \gamma ^2p_1p_2\right) \\&=(\nu _2-\lambda )^{p_1-1}(\xi _2-\lambda )^{p_2-1}\left( (\nu _1-\lambda )(\xi _1-\lambda ) - \gamma ^2p_1p_2\right) . \end{aligned}$$

Hence the eigenvalues of V are $\nu _2$ ($p_1-1$ times), $\xi _2$ ($p_2-1$ times), and the two solutions $\nu _1^*$ and $\xi _1^*$ of the quadratic equation $(\nu _1-\lambda )(\xi _1-\lambda ) - \gamma ^2p_1p_2=0$. Note that $\nu _1^*+\xi _1^*=\nu _1+\xi _1$. In the special case $\gamma =0$ we have $\nu _1^*=\nu _1$ and $\xi _1^*=\xi _1$.

The sum of the eigenvalues is

$$\begin{aligned} (p_1-1)\nu _2 + (p_2-1)\xi _2 + \nu _1 + \xi _1, \end{aligned}$$

and it easy to verify that this equals the trace of V, as of course it should. $\square$

B Stein’s Lemma

Stein’s lemma (Stein 1981) is a rather surprising and strong result. We first consider the univariate, then the multivariate case. The generalization is not straightforward.

Lemma 3

Let $x\sim {\text {N}}(\theta ,1)$ and let $h:\mathfrak {R}\rightarrow \mathfrak {R}$ be an absolutely continuous function with derivative $h'$. Assume that ${\text {E}}|h'(x)|<\infty$. Then,

$$\begin{aligned} {\text {cov}}(h(x),x)={\text {E}}[h(x)(x-\theta )]={\text {E}}[h'(x)]. \end{aligned}$$

Proof

We write

$$\begin{aligned}{}[h(x)\phi (x-\theta )]'&= h'(x)\phi (x-\theta ) + h(x)\phi '(x-\theta )\\&= h'(x)\phi (x-\theta ) - h(x)(x-\theta )\phi (x-\theta ), \end{aligned}$$

where $\phi$ denotes the standard normal density. Integrating gives

$$\begin{aligned} 0=\Bigl .h(x)\phi (x-\theta )\Bigr |_{-\infty }^{\infty } = {\text {E}}[h'(x)]-{\text {E}}[h(x)(x-\theta )]. \end{aligned}$$

$\square$

Note the requirement of absolute continuity, which imposes a smoothness property on h that is stronger than (uniform) continuity, but weaker than continuous differentiability. It guarantees that h is differentiable almost everywhere. In the applications it is important to place minimal restrictions on the function h, for example that it may be kinked.

The univariate version of Stein’s lemma is a powerful result with many nontrivial applications. As a simple example, let $h(x)=x^m$. Then we immediately obtain all moments of the normal distribution through the recursion ${\text {E}}[x^{m+1}]=\theta {\text {E}}[x^{m}] + m{\text {E}}[x^{m-1}]$.

In the multivariate case we have $x\sim {\text {N}}(\theta ,I_p)$ with $p\ge 2$ and we need the concept of ‘almost differentiability’ (in Stein’s terminology). We write $x = (x_i,x_{-i})$ to decompose a point $x\in \mathfrak {R}^p$ in terms of its ith component $x_i$ and all other components $x_{-i}$. Thus, $h(\cdot ,x_{-i})$ refers to h as a function of its ith argument with all other arguments fixed at $x_{-i}$. Then h is ‘almost differentiable’ if for each $i=1,\ldots ,p$ and almost every $x_{-i}\in \mathfrak {R}^{p-1}$ the function $h(\cdot ,x_{-i}):\mathfrak {R}\rightarrow \mathfrak {R}$ is absolutely continuous. An almost differentiable function h has partial derivatives almost everywhere.

Given this multivariate extension of the concept of absolute continuity, Stein’s lemma reads as follows.

Lemma 4

(Stein) Let $x\sim {\text {N}}(\theta ,I_p)$ with $p\ge 2$ and let $h:\mathfrak {R}^p\rightarrow \mathfrak {R}$ be almost differentiable with ${\text {E}}\Vert \nabla h(x)\Vert <\infty$, where $\nabla h(x)$ denotes the gradient of h(x). Then,

$$\begin{aligned} {\text {E}}\left[ h(x)(x-\theta )\right] ={\text {E}}[\nabla h(x)]. \end{aligned}$$

Proof

See Stein (1981, Lemma 2). $\square$

Stein’s result can be generalized straightforwardly to the case where h is a vector function.

Lemma 5

Let $x\sim {\text {N}}(\theta ,I_p)$ with $p\ge 2$ and let $h:\mathfrak {R}^p\rightarrow \mathfrak {R}^q$. If $h_j:\mathfrak {R}^p\rightarrow \mathfrak {R}$ is almost differentiable with ${\text {E}}\Vert \nabla h_j(x)\Vert <\infty$ for all $j=1,\ldots ,q$, then

$$\begin{aligned} {\text {E}}\left[ h(x)(x-\theta )'\right] ={\text {E}}\left[ \frac{\partial h(x)}{\partial x'}\right] . \end{aligned}$$

Proof

This follows from Lemma 4 by considering each row of $h(x)(x-\theta )'$ separately. Then,

$$\begin{aligned} {\text {E}}\left[ h_j(x)(x-\theta )'\right] ={\text {E}}\left[ \frac{\partial h_j(x)}{\partial x'}\right] \end{aligned}$$

for each j and the result follows. $\square$