Model selection and application to high-dimensional count data clustering

Abstract

EDCM, the Exponential-family approximation to the Dirichlet Compound Multinomial (DCM), proposed by Elkan (2006), is an efficient statistical model for high-dimensional and sparse count data. EDCM models take into account the burstiness phenomenon correctly while being many times faster than DCM. This work proposes the use of Minimum Message Length (MML) criterion for determining the number of components that best describes the data with a finite EDCM mixture model. Parameters estimation is based on the previously proposed Deterministic Annealing Expectation-Maximization (DAEM) approach. The validation of the proposed unsupervised algorithm involves different real applications: text document modeling, topic novelty detection and hierarchical image clustering. A comparison with results obtained for other information-theory based selection criteria is provided.

Appendices

Appendix A: Proof of (16)

We have the negative of log-likelihood function as:

$$\begin{array}{@{}rcl@{}} -\mathcal{L}(\mathcal{X}_{j}|\boldsymbol{\varphi}_{j})&=&-\log \left( \prod\limits_{d=l}^{l+\eta_{j}-1} \mathcal{EDCM}(\mathbf{X}_{d}|\boldsymbol{\varphi}_{j}) \right) \\ &=&\eta_{j}(-\log {\Gamma}(s_{j}))+\sum\limits_{d=l}^{l+\eta_{j}-1} \log {\Gamma}(s_{j}+n_{d}) \\ &&-\sum\limits_{w:x_{dw}\geq1} \log \varphi_{jw} + \log x_{dw} \end{array} $$

(30)

Then, the first order derivative of the negative log-likelihood, also called the Fisher score function, is:

$$\begin{array}{@{}rcl@{}} -\frac{\partial \mathcal{L}(\mathcal{X}_{j}|\varphi_{j})}{\partial\varphi_{jw}}&=&\eta_{j}(-{\Psi}(s_{j}))+\sum\limits_{d=l}^{l+\eta_{j}-1} {\Psi}(s_{j}+n_{d}) \\ && -\sum\limits_{d=l}^{l+\eta_{j}-1} I(x_{dw} \geq 1) \frac{1}{\varphi_{jw}} \end{array} $$

(31)

where Ψ is the digamma function. Then,

$$\begin{array}{@{}rcl@{}} -\frac{\partial^{2} \mathcal{L}(\mathcal{X}_{j}|\varphi_{j})}{\partial\varphi^{2}_{jw}}&=&\eta_{j}(-{\Psi}^{\prime}(s_{j}))+\sum\limits_{d=l}^{l+\eta_{j}-1} {\Psi}^{\prime}(s_{j}+n_{d}) \\ &&+\sum\limits_{d=l}^{l+\eta_{j}-1} I(x_{dw} \geq 1) \frac{1}{\varphi_{jw}^{2}} \end{array} $$

(32)

and:

$$\begin{array}{@{}rcl@{}} -\frac{\partial^{2} \mathcal{L}(\mathcal{X}_{j}|\varphi_{j})}{\partial\varphi_{jw1} \partial\varphi_{jw2}}&=& \eta_{j}(-{\Psi}^{\prime}(s_{j}))\\ &&+\sum\limits_{d=l}^{l+\eta_{j}-1} {\Psi}^{\prime}(s_{j}+n_{d}) , w_{1} \neq w_{2} \end{array} $$

(33)

where Ψ^′ is the trigamma function. We remark that F(φ_j) can be written as:

$$ F(\boldsymbol{\varphi}_{j})=D_{j} +\gamma_{j} \mathbf{AA}^{T} $$

(34)

where $D=diag\left [\sum \limits _{d=l}^{l+\eta _{j}-1} I(x_{dw} \geq 1) \left .\frac {1}{\varphi _{j1}^{2}} \right ), {\dots } , \sum \limits _{d=l}^{l+\eta _{j}-1} I\right .$$\left .\vphantom {\sum \limits _{d=l}^{l+\eta _{j}-1}}(x_{dw} \geq 1) \left .\frac {1}{\varphi _{jW}^{2}} \right ) \right ]$, $\gamma =\eta _{j}(-{\Psi }^{\prime }(s_{j}))+\sum \limits _{d=l}^{l+\eta _{j}-1} {\Psi }^{\prime }(s_{j}+n_{d})$, and A^T = 1. Then, according to (Theorem 8.4.3) given by Graybill [33], the determinant of the Fisher information matrix F(φ_j) is:

$$ |F(\boldsymbol{\varphi}_{j})|=\left( 1+\gamma_{j} \sum\limits_{w = 1}^{W} \frac{a_{jw}^{2}}{D_{jw}} \right) \prod\limits_{w = 1}^{W} D_{jw} $$

(35)

By substituting (35) and (15) into (14), we obtain:

$$ |F({\Theta})|\simeq\frac{N}{\prod\limits_{j = 1}^{M} \mu_{j}} \prod\limits_{j = 1}^{M} \left[ \left( 1+\gamma_{j} \sum\limits_{w = 1}^{W} \frac{a_{jw}^{2}}{D_{jw}} \right) \prod\limits_{w = 1}^{W} D_{jw} \right] $$

(36)

Then, taking the log gives (16).

Appendix B: Proof of (29)

The KL divergence between two distributions that belong to the exponential family is defined as [16]:

$$\begin{array}{@{}rcl@{}} KL \left( P(X|\theta_{j1}),P(X|\theta_{j2})\right)&=&{\Phi}(\theta_{j1})-{\Phi}(\theta_{j2})\\ &&+\left( G(\theta_{j1})\right.\\ &&\left.-G(\theta_{j2})\right) E_{\theta_{j1}}[T(X)] \end{array} $$

(37)

where E_𝜃 is the expectation with respect to P(X|𝜃). We have:

$$\begin{array}{@{}rcl@{}} {\Phi}(\theta)&=&\frac{{\Gamma}(s)}{{\Gamma}(s+n)} \end{array} $$

(38)

$$\begin{array}{@{}rcl@{}} G(\theta)&=&\log(\varphi_{jw}) \end{array} $$

(39)

$$\begin{array}{@{}rcl@{}} T(X)&=&I(x_{w} \geq 1) \end{array} $$

(40)

Moreover, we have the following [16]:

$$ E_{\theta}[T(X)]=-{\Phi}^{\prime}(\theta) $$

(41)

Thus, according to (38 and 40), we have:

$$ E_{\theta}\left[I(x_{w} \geq 1)\right]=-\frac{\partial {\Phi}(\theta)}{\partial \varphi_{jw}}= {\Psi}(s+n)-{\Psi}(s) $$

(42)

substituting (42, 38 and 39) in (37) gives (29).