A power-controlled reliability assessment for multi-class probabilistic classifiers

Abstract

In multi-class classification, the output of a probabilistic classifier is a probability distribution of the classes. In this work, we focus on a statistical assessment of the reliability of probabilistic classifiers for multi-class problems. Our approach generates a Pearson $\chi ^2$ statistic based on the k-nearest-neighbors in the prediction space. Further, we develop a Bayesian approach for estimating the expected power of the reliability test that can be used for an appropriate sample size k. We propose a sampling algorithm and demonstrate that this algorithm obtains a valid prior distribution. The effectiveness of the proposed reliability test and expected power is evaluated through a simulation study. We also provide illustrative examples of the proposed methods with practical applications.

Appendix A Proof of Theorem 1

We show that the total area under $f_{\textbf{r}}(r_1,\ldots ,r_c)$ equals to 1. Because there are $\left( {\begin{array}{c}c\\ h\end{array}}\right)$ cases that h number of the $r_i$ $(i=1,\ldots ,c)$ values are negative, the total area can be expressed as

$$\begin{aligned} \idotsint f_{\textbf{r}}(r_1,r_2,\ldots ,r_c) \,dr_1 \ldots \,dr_c = \sum _{h=1}^{c-1} \left( {\begin{array}{c}c\\ h\end{array}}\right) \text {A}_h, \end{aligned}$$

where $\text {A}_h$ represents the probability such that

$$\begin{aligned} \sum _{i=1}^{h} r_i = -\frac{\epsilon }{2}, \quad \sum _{i=h+1}^{c} r_i = \frac{\epsilon }{2}, \quad r_1,\ldots ,r_h < 0, \text { and} \quad r_{h+1},\ldots ,r_c > 0 \end{aligned}$$

and thus the support of $(r_1,\ldots ,r_c)$ becomes

$$\begin{aligned}&r_1 \in \left( -\frac{\epsilon }{2},0\right) , \\&r_2 \in \left( -\frac{\epsilon }{2}- r_1,0\right) , \\&... \\&r_{h-1} \in \left( -\frac{\epsilon }{2}- r_1...-r_{h-2},0\right) , \\&r_h = -\frac{\epsilon }{2}- r_1...-r_{h-1}, \\&r_{h+1} \in \left( 0,\frac{\epsilon }{2}\right) , \\&r_{h+2} \in \left( 0,\frac{\epsilon }{2}-r_{h+1}\right) , \\&... \\&r_{c-1} \in \left( 0,\frac{\epsilon }{2}-r_{h+1} ... - r_{c-2}\right) , \text { and}\\&r_c = \frac{\epsilon }{2}-r_{h+1} ... - r_{c-1}. \end{aligned}$$

Using change of variable, we define $w_i = - \epsilon /2 - \sum _{j=0}^{i}r_{h-j}$ ($i=0,\ldots ,h-2)$ and $v_i = \epsilon /2 - \sum _{j=0}^{i}r_{c-j}$ ($i=0,\ldots ,c-h-2)$. Then, we have

$$\begin{aligned} \text {A}_h&= \int _{-\frac{\epsilon }{2}}^{0} \int _{-\frac{\epsilon }{2}}^{w_{h-2}} \ldots \int _{-\frac{\epsilon }{2}}^{w_1} \int _{0}^{\frac{\epsilon }{2}} \int _{v_{c-h-2}}^{\frac{\epsilon }{2}} \ldots \int _{v_1}^{\frac{\epsilon }{2}} \frac{1}{\left( {\begin{array}{c}c\\ h\end{array}}\right) } \frac{(c-2)!}{\epsilon ^{c-2}} \vert J \vert \,dw_0 \ldots \,dv_{c-h-2} \\&=\frac{1}{\left( {\begin{array}{c}c\\ h\end{array}}\right) } \frac{(c-2)!}{\epsilon ^{c-2}} \left[ \int _{0}^{\frac{\epsilon }{2}} \ldots \int _{v_{1}}^{\frac{\epsilon }{2}} \left[ \int _{-\frac{\epsilon }{2}}^{0} \ldots \int _{-\frac{\epsilon }{2}}^{w_1} \,dw_0 \ldots \,dw_{h-2} \right] \,dv_0 \ldots \,dv_{c-h-2} \right] , \end{aligned}$$

where the Jacobian $\vert J \vert = 1$ due to the property of the determinant of a triangular matrix.

We first prove by induction that

$$\begin{aligned} \int _{-\frac{\epsilon }{2}}^{w_n} \ldots \int _{-\frac{\epsilon }{2}}^{w_1} \,dw_0 \ldots \,dw_{n-1} = \sum _{j=0}^{n} \left( \frac{\epsilon }{2}\right) ^j \frac{1}{j!} \frac{1}{(n-j)!} w_n^{n-j} \end{aligned}$$

(A1)

for any positive integer n. When $n=1$, we have

$$\begin{aligned} \int _{-\frac{\epsilon }{2}}^{w_1} \,dw_0 = w_1 + \frac{\epsilon }{2} = \sum _{j=0}^{1} \left( \frac{\epsilon }{2}\right) ^j \frac{1}{j!} \frac{1}{(1-j)!} w_1^{1-j}. \end{aligned}$$

Assuming that

$$\begin{aligned} \int _{-\frac{\epsilon }{2}}^{w_k} \ldots \int _{-\frac{\epsilon }{2}}^{w_1} \,dw_0 \ldots \,dw_{k-1} = \sum _{j=0}^{k} \left( \frac{\epsilon }{2}\right) ^j \frac{1}{j!} \frac{1}{(k-j)!} w_k^{k-j}, \end{aligned}$$

we have

$$\begin{aligned} \int _{-\frac{\epsilon }{2}}^{w_{k+1}} \ldots \int _{-\frac{\epsilon }{2}}^{w_1} \,dw_0 \ldots \,dw_{k}&= \int _{-\frac{\epsilon }{2}}^{w_{k+1}} \left( \sum _{j=0}^{k} \left( \frac{\epsilon }{2}\right) ^j \frac{1}{j!} \frac{1}{(k-j)!} w_k^{k-j} \right) \,dw_k \\&= \sum _{j=0}^{k} \left( \frac{\epsilon }{2}\right) ^j \frac{1}{j!} \frac{1}{(k-j)!} \left( \int _{-\frac{\epsilon }{2}}^{w_{k+1}} w_k^{k-j} \,dw_k \right) \\&= \sum _{j=0}^{k} \left( \frac{\epsilon }{2}\right) ^j \frac{1}{j!} \frac{1}{(k-j+1)!} w_{k+1}^{k-j+1} \\&\quad - \sum _{j=0}^{k} \left( \frac{\epsilon }{2}\right) ^j \left( - \frac{\epsilon }{2} \right) ^{k-j+1} \frac{1}{j!} \frac{1}{(k-j+1)!}. \end{aligned}$$

From the binomial theorem that is given as

$$\begin{aligned} \sum _{j=0}^{k+1} \left( \frac{\epsilon }{2}\right) ^j \left( - \frac{\epsilon }{2} \right) ^{k-j+1} \frac{(k+1)!}{j! (k-j+1)!} = 0, \end{aligned}$$

we have

$$\begin{aligned} \sum _{j=0}^{k} \left( \frac{\epsilon }{2}\right) ^j \left( - \frac{\epsilon }{2} \right) ^{k-j+1} \frac{1}{j!} \frac{1}{(k-j+1)!} = -\frac{1}{(k+1)!} \left( \frac{\epsilon }{2} \right) ^{k+1} \end{aligned}$$

Hence,

$$\begin{aligned} \int _{-\frac{\epsilon }{2}}^{w_{k+1}} \ldots \int _{-\frac{\epsilon }{2}}^{w_1} \,dw_0 \ldots \,dw_{k}&= \sum _{j=0}^{k} \left( \frac{\epsilon }{2}\right) ^j \frac{1}{j!} \frac{1}{(k-j+1)!} w_{k+1}^{k-j+1} + \frac{1}{(k+1)!} \left( \frac{\epsilon }{2} \right) ^{k+1} \\&= \sum _{j=0}^{k+1} \left( \frac{\epsilon }{2}\right) ^j \frac{1}{j!} \frac{1}{(k-j+1)!} w_{k+1}^{k-j+1}. \end{aligned}$$

Then, using the result in Eq. (A1),

$$\begin{aligned} \int _{-\frac{\epsilon }{2}}^{0} \int _{-\frac{\epsilon }{2}}^{w_{h-2}} \ldots \int _{-\frac{\epsilon }{2}}^{w_1} \,dw_0 \ldots \,dw_{h-2}&= \int _{-\frac{\epsilon }{2}}^{0} \left( \sum _{j=0}^{k+1} \left( \frac{\epsilon }{2}\right) ^j \frac{1}{j!} \frac{1}{(k-j+1)!} w_{k+1}^{k-j+1} \right) \,dw_{h-2} \\&= \frac{1}{(h-1)!} \left( \frac{\epsilon }{2}\right) ^{h-1}. \end{aligned}$$

Similarly, we can show that

$$\begin{aligned} \int _{0}^{\frac{\epsilon }{2}} \int _{v_{c-h-2}}^{\frac{\epsilon }{2}} \ldots \int _{v_{1}}^{\frac{\epsilon }{2}} \,dv_0 \ldots \,dv_{c-h-2} = \frac{1}{(c-h-1)!} \left( \frac{\epsilon }{2}\right) ^{c-h-1}. \end{aligned}$$

Therefore,

$$\begin{aligned} \idotsint f_{\textbf{r}}(r_1,\ldots ,r_c) \,dr_1 \ldots \,dr_c&= \sum _{h=1}^{c-1} \left( {\begin{array}{c}c\\ h\end{array}}\right) \text {A}_h \\&= \frac{(c-2)!}{\epsilon ^{c-2}} \sum _{h=1}^{c-1} \left( \frac{(\epsilon /2)^{h-1}}{(h-1)!} \frac{(\epsilon /2)^{c-h-1}}{(c-h-1)!} \right) \\&= \frac{1}{2^{c-2}} \sum _{h=1}^{c-1} \frac{(c-2)!}{(h-1)!(c-h-1)!} \\&= \frac{1}{2^{c-2}} \sum _{b=0}^{c-2} \frac{(c-2)!}{b!(c-b-2)!}= 1 \end{aligned}$$