Probabilistic Topic Modeling for Comparative Analysis of Document Collections Probabilistic Topic Modeling for Comparative Analysis of Document Collections

3.3.1 Joint Distribution. When the dataset label $s$ of the document is observed, its labeling prior $\mu$ is d-separated from the rest of the model. When $s$ is invisible, the CDTM model can be used to predict the collection label for documents. Based on Algorithm 1 and the graphical model in Figure 3, the joint distribution of the CDTM model can be represented as follows:

(1)

3.3.2 Hidden Variables. The key to this inference problem is to estimate the posterior distributions of the following hidden variables: (1) the topic assignment indicator for words; (2) the common/distinctive preference indicator for words; and (3) the topic mixture proportion , and preference mixture proportion for documents. As a special case of a Markov chain Monte Carlo, Gibbs sampling iteratively samples one instance at a time, conditional on the values of the remaining given variables. We only present the result here; the detailed derivation process is omitted in order to keep the overall dicussion more clear. According to Bayes’ rule, the conditional probability of can be computed by dividing the joint distribution in Equation (1) by all the variables except . Since is dependent on the value of , the sampling of is discussed separately for two situations: or 1. When , which indicates that the topic is chosen from a common topic, the conditional probability of is as follows:

(2)

(3)

Similar to the inference of $z$, the derivation for the posterior of is discussed for two cases: or . Specifically, when , the inference is calculated as Equation (4), where $n_m^{0}$ is the number of words choosing in document $m$:

(4)

(5)

3.3.3 Multinomial Parameters. Variables , , $\theta _{c}$, and $\lambda$ are multinomial distributions with Dirichlet priors. According to Bayes rule and the definition of Dirichlet priors, these multinomial parameters can be computed from the above posteriors. For example, the common topic word distribution $\phi _{cz}$ for term $v$ and the common topic mixture $\theta _{cm}$ for document $m$ are computed as follows:

(11)

3.3.4 Gibbs Sampling Algorithm. The Gibbs sampling process for then CDTM model is shown in Algorithm 2. The procedure has the following five count variables: $n_{1sz}^v$ and $n_{0z}^v$ are matrices with dimension $K \times V$, $n_{1sm}^z$ has $M$ rows and $K_d$ columns, $n_{m}^x$ has dimension $M \times 2$, and $n_{0}^z$ is $K_c$ dimensional vector.

The Gibbs sampling algorithm has the following three stages: initialization, a burn-in period, and a sampling period. The determination of the optimum burn-in period duration is essential for Markov chain Monte Carlo (MCMC) approaches. In this article, we observe changes in the perplexity to check whether the Markov chain has converged. There are several strategies for using the results from Gibbs samplers. One is to read the results from one iteration (e.g., last iteration), another is to use the average of multiple samples. To obtain independent Markov chain states, here we use “sampling lag” to read results, which will leave an interval of $I$ iterations between subsequent chosen samplers.

4.1.1 Datasets and Experimental Settings. The generation strategy for the synthetic data are as follows. Different ground truth input matrices $D_i$ for $i=1,2,\ldots,S$ can be considered as term-document matrices based on common topic matrices $W_{i,c}=[w_{i,c}^{(1)},\ldots,w_{i,c}^{(K_c)}]$ and distinctive topic matrices $W_{i,d}=[w_{i,d}^{(1)},\ldots,w_{i,d}^{(K_d)}]$. The $t$-th element ${(w_{i,c}^{(k)})_t}$ and ${(w_{i,d}^{(k)})_t}$ in the matrices can be computed as follows, where $inx(i,k)=100K_c+100(i-1)K_d+100(k-1)$:

• Label Error Rate. Label Error Rate (LER) here measures the relationship between the predicted and the ground truth labels. A lower LER value corresponds to a better performance. Given a document $m$ in dataset $\mathcal {D}$, all the models in our experiments will produce a document-level topic mixture, the topic with the largest probability value will be given the final class label $l_m$. Given the ground truth label $k_m$, the LER is computed as follows:
  \begin{equation} LER(\mathcal {D}) = \frac{{\eta (l_m,k_m)}}{ {M}}, \end{equation} (17)
  where $M$ is the total number of documents, $\eta (x,y)$ is equal to zero, if $x = y$, and equals to one, otherwise. In other words, $\eta (x,y)$ is the number of documents incorrectly labeled by the method.
  
• Similarity Score. Similarity score evaluates the similarity among the common topics of different datasets. In this article, the similarity score, which indicates how close the common topics $\Phi _{c,i}$ in dataset $\mathcal {D}_i$ is with their corresponding topics $\Phi _{c,j}$ in the other dataset $\mathcal {D}_j$, is defined as follows:
  \begin{equation} f(\mathcal {D}_i,\mathcal {D}_j)_s=\sum _{z=1}^{K_c}||\phi _{c,i}^z-\phi _{c,j}^z||^2. \end{equation} (18)
  In an ideal scenario, the similarity score of common topics should be zero, since they are shared by all the datasets. Generally, a smaller value of similarity score corresponds to a better model performance.
  
• Reconstruction Score. Reconstruction score measures the closeness of the estimated parameters produced by the models to the ground truth words. More specifically, for each document, the product of topic-term distribution and document-topic mixture will give an estimation of word occurrence probability. Reconstruction score is computed as the difference between such estimated values in document $m$ and its true words $W_m$:
  (19)
  Similar to similarity score and LER, a smaller value in reconstruction score means better model performance.
  

4.1.3 CDTM Vs Batch-CDTM. In this part, we compared CDTM model and its variation batch-CDTM model through illustration of result matrices and LER.

Figure 5(a) and (b) show the examples of the resulting topic-term distributions of batch-CDTM and CDTM, respecitvely. As can be seen in these figures, the performance of batch-CDTM is better than that of the CDTM. Batch-CDTM can successfully recognize all the topics and there are no overlaps between any two topics. However, CDTM makes obvious mistakes that the first common topic (topic #9) shares the same terms (terms #1,000 to #1,100) with the third distinctive topic (topic #3). To justify this observation, we run each method 20 times with random initializations and set identical initializations to both methods at each run.

Fig. 5. Matrices for topic-term distribution results of CDTM and batch-CDTM model. Green blocks are distinctive topics, and yellow blocks are common topics. X-axis corresponds to topics, and Y-axis denotes the terms. The settings here are $K_d=8$ and $K_c=3$.

In Figure 6(a) and (b), CDTM generally achieves lower error rate, compared to batch-CDTM model. As shown in Figure 6(a), both methods gain better performance with the increase of distinctive topic number $K_d$. Batch-CDTM performs poorly at the beginning, with a small value $K_d$ (equal to one). And its performance appears to be linear with respect to the values of $K_d$. CDTM model can yield relatively better performance even at the initial stage, where $K_d$ is set to be one. No obvious linear relationship between $K_d$ and performance is observed for CDTM model. However, the general trend is still similar to that of batch-CDTM: larger distinctive topic number will benefit the performance. Figure 6(b) shows the performance changes as the common topic number $K_c$ increases. Unlike the impact of distinctive topic number $K_d$, when changing common topic number $K_c$, there are no linear correlations that are observed for CDTM, nor for batch-CDTM. Both the two methods will produce lower error rates with larger common topic settings, but the curves are fluctuating.

Fig. 6. Error rates of CDTM and batch-CDTM model, when changing common and distinctive topic number settings.

In general, Figure 5 evaluates the model performance at the dataset-level by examining the learned topics, while Figure 6 shows the results on document-level classification. As can be seen from the above the experiments, although batch-CDTM occasionally provides better topics, it is less robust than CDTM on document-level classification. Batch-CDTM can generally get better topics, as the $\lambda$ is the global parameter, and CDTM has more flexibility at the document level because its $\lambda$ parameter is assigned locally. Based on these observations, in the following document-level evaluation on real-world data, we will compare the performance of CDTM model to the other state-of-the-arts methods.

4.1.4 CDTM Vs discNMF on Multiple Datasets. As mentioned above, CDTM can be used in the analysis of multiple datasets, which is not supported by the baseline methods. In this part, we compare the performance of CDTM model with its closest baselines discNMF using the metrics of reconstruction error and similarity score on multiple synthetic datasets. Since discNMF does not support multiple datasets, we repeatedly apply it to achieve the results comparable to those of the CDTM model. However, it only works on the situation that includes $2_i (i=1,2,\ldots)$ datasets. The results studied for five datasets are presented in Table 2. Here discNMF is applied three times to get the results on five datasets. In the first application of DiscNMF, we obtained the results as combinations of dataset 1 and 2 and dataset 3 and 4. The second application of discNMF separated dataset 1 and 2, and similarly the third run of discNMF distinguished dataset 3 and 4.

As can be seen from Table 2, CDTM model performs better than discNMF in both metrics, similarity score and reconstruction error. Actually, because CDTM designs a global shared data structure to model the common topics, its similarity score is zero for all datasets. DiscNMF gets a score of 0.11 in the results on combination dataset 1 and 2 and dataset 3 and 4. Smaller scores are achieved in the further separations, that discNMF gets similarity score 0.07 in topics from dataset 1 and 2, and 0.09 in the common topic similarity score for dataset 3 and 4.

4.2.1 Datasets and Experimental Settings. To evaluate our method and other baseline comparisons, three real-world document datasets are used in our experiments: 20 News group data (20 clusters, 18,828 documents, and 43,009 keywords), Reuters dataset (65 clusters, 8,293 documents, and 18,933 keywords), and four area dataset (4 groups, 15,110 documents, 6,487 keywords). These datasets have been selected for their public availability and wide usage in topic modeling evaluations [19, 23]. Three sub-datasets are formed with different clusters, as shown in Table 3. To conduct an extensive comparison, various ratios of common and distinctive clusters are assigned to the datasets. In the Reuters data, the number of topics contained in the common cluster is the same as that in each of the exclusive clusters. while in the four area dataset, the number of common topics is larger than the number of distinctive topics.

For the CDTM model, weak symmetric priors are used for all Dirichlet or Beta parameters: $\alpha _0=\alpha _1=0.1, \beta _0=\beta _1=0.001, \gamma =0.5, \mu =0.1$. It has been shown that a better performance can be obtained when the values of topic number are set closer to the real-world cases [19]. The distinctive topic number $K_d$ and common topic number $K_c$ for each dataset are therefore set as follows: (1) $K_d=3$ and $K_c=3$ in the Reuters dataset; (2) $K_d=3$ and $K_c=1$ in the 20news dataset; (3) $K_d=1$ and $K_c=2$ in the four area dataset. The Gibbs sampler is run for 400 iterations, with the first 100 iterations as burnt-in period.

4.2.2 Comparison Methods. Although there are many well studied applications on topic-class modeling [27, 41] or global-local aspect mining [7, 18], few general approaches are proposed for the purpose of discriminative learning. Since the research focus of our article is discriminative learning with topic models, the following five methods are chosen as our baseline methods, which are the most relevant approaches for our problem. LDA and NMF are most widely used state-of-the-art methods, and almost all other topic models are the variations of them. discLDA and discNMF are such variations on discriminative learning, which are the works that are closest to our proposed CDTM model. Recently, deep-learning models such as recurrent neural networks (RNN) are known to be good performers on sequence data such as text [28]. To better evaluate performance of CDTM, we also add one RNN model as the benchmark comparison method on text clustering task [46].

• LDA [2]: This is the standard topic modeling approach widely studied in the literature. We ran the LDA method on different subsets separately. For the best results, we used weak symmetric priors in our experiments: $\alpha =0.1$ and $\beta =0.001$.
  
• NMF [25]: This is the most popular topic modeling method based on matrix decomposition. As with the standard LDA, we applied the NMF method separately to each subset for topic discovery.
  
• discLDA [23]: This is a variant of LDA that is capable of discriminative modeling. There are three tunable hyperparameters $\alpha$, $\beta$, and $\pi$ in this approach, which here is set to $0.1, 0.001$, and 0.1, respectively.
  
• discNMF [19]: This is a discriminative topic modeling method based on NMF. To achieve best the performance, we set parameter $\alpha$ to be 100, and $\beta$ to be 10.
  
• LSTM [46]: Long short-term memory (LSTM) is a RNN architecture, which is widely used in sequence data such as text and speech. To perform text clustering, our implementation consists of one LSTM layer and one softmax layer, with a learning rate of 0.0001.
  

4.2.3 Evauation Metrics. The quality of the topic modeling results are evaluated in terms of different measures described below.

• Perplexity. Perplexity is a standard metric used to evaluate topic modeling approaches [2, 14], and is typically defined as follows:
  \begin{equation} Perplexity(D) = exp\left\lbrace {\frac{{ - \sum_{m = 1}^M {\prod \nolimits _{n = 1}^N l ogP({{{\bf w}}_{mn}})} }}{{\sum_{m = 1}^M {{N_m}} }}} \right\rbrace \end{equation} (20)
  where $M$ is the number of documents, is the word vector for document $m$ and $N_m$ is the number of words in document $m$. A lower perplexity indicates more accurate performance of the model. Here the probability of the words occurring in a document $m$, given its parameters, can be calculated as follows:
  \begin{equation} P(w_{mn}) = \left\lbrace \begin{array}{c}\displaystyle {\phi _{cz_{mn}}^{w_{mn}}}{\theta _{cz_{mn}}^{(m)}},\quad {\rm {if}}{\hspace{1.0pt}} {\hspace{1.0pt}} {x_{mn} = 0} \\[3pt] \displaystyle{\phi _{dz_{mn}}^{w_{mn}}}{\theta _{dz_{mn}}^{(m)}},\quad {\rm {if}}{\hspace{1.0pt}} {\hspace{1.0pt}} {x_{mn} = 1} \end{array} \right. \end{equation} (21)
  where $\phi _{cz_{mn}}^{w_{mn}}$ and ${\phi _{dz_{mn}}^{w_{mn}}}$ can be computed through Equations (6) and (8), while ${\theta _{cz_{mn}}^{(m)}}$ and $\theta _{dz_{mn}}^{(m)}$ can be calculated through Equations (7) and (9).
  
• Accuracy. Clustering accuracy (ACC) quantitatively measures the mapping relationships between resultant clusters and labeled classes [5]. A larger ACC value means better clustering performance. Given a document $m$, result label $r_m$, and ground truth label $s_m$, the cluster accuracy is computed as follows:
  \begin{equation} ACC = \frac{\sum _{m=1}^{M}\delta (s_m,r_m)}{M}, \end{equation} (22)
  where $M$ is the total number of documents, $\delta (x,y)$ is a delta function that is equal to one, if $x = y$, and equals to zero, otherwise. In our evaluation, the ACC metric is used to calculate the quality of clusters, where $M$ is the total number of documents within a cluster in the ground-truth case and $\delta (x,y)$ is the number of documents correctly labeled by the methods.
  
• Normalized Mutual Information. Normalized Mutual Information (NMI) is used to measure the quality of clusters, and is typically defined as follows:
  \begin{equation} NMI = \frac{\sum _{i=1}^{c}\sum _{j=1}^{c}\frac{n_{i,j}}{n}\log \frac{n \cdot n_{i,j}}{n_i\hat{n}_j}}{\sqrt {(\sum _{i=1}^{c}n_i\log \frac{n_i}{n})(\sum _{i=1}^{c}\hat{n}_j\log \frac{\hat{n}_j}{n})}}. \end{equation} (23)
  In this article, NMI is used to evaluate clustering performance, where $c$ is the number of clusters, $n_i$ is the number of documents contained in the ground truth label $\mathcal {C}_i$, $\hat{n}_j$ is the number of documents belonging to result label $\mathcal {L}_i$, and $n_{ij}$ is the number of documents that are in the intersection between the result label $\mathcal {C}_i$ and the ground truth class $\mathcal {L}_j$. Typically, a larger $NMI$ value indicates better clustering performance.
  

4.2.4 Parameter Sensitivity Analysis. The distinctive topic number $K_d$ is an important parameter for the discriminative learning method discLDA, discNMF, and our proposed CDTM. When keeping the total topic number $K=K_c+K_d$ fixed, Figure 7 shows the perplexity of method discLDA, discNMF, and CDTM for varying topic numbers $K_d$. The correct number of common and distinct topics here are both three. Two conclusions can be drawn from this figure.

Fig. 7. Performance comparison in terms of perplexity.

• Minimum perplexity. Both discNMF and our proposed CDTM show minimal perplexity when $K_d$ is set to be 3, which is the correct number of discriminative topic pairs. However, there is no obvious correspondence between minimum perplexity and correct discriminative topic for discLDA.
  
• Sensitivity. Our proposed CDTM consistently obtains low perplexity with changes in values of $K_d$. Interestingly, another graphical model, discLDA is the second best performer in terms of variance, while matrix factorization method discNMF is very sensitive to the setting of parameter $K_d$, which increases dramatically when $K_d$ is set to be 5.
  

In summary, Figure 7 shows that CDTM model consistently provides results that are closer to the ground truth, discLDA is also stable with comparatively small variance, although it is hard to assign the right parameter value, while discNMF is a good performer in most cases except for the extreme values.

4.2.5 Clustering Performance. The clustering performance evaluates the quality of the resulting clusters compared to the ground-truth cluster labels. First, the cluster index for each document is computed as the most strongly associated topic index. For our proposed CDTM model, this step identifies the maximal element $\theta _{dz}$ in vector $\theta _{d}$, which can be computed similar to Equation (7). The results of discLDA are computed by jointly considering the transformation matrices and the topic mixture distribution. For the NMF and discNMF methods, this step is implemented by finding the corresponding column vector for factor matrix $H$. For LSTM, the softmax layer will generate the probability distribution of clusters, and we take the index of the maximum element in the distribution as the predicted cluster. The obtained results are then re-mapped to ground truth labels using the Hungarian algorithm [22]. Two widely adopted cluster quality measures ACC and NMI are used to evaluate the performance; these are listed in Table 4.

As shown in Table 4, we mark the best values with bold, and the second best ones with underlines. Generally, LSTM and our proposed CDTM model are best performers that outperform other existing methods (NMF, LDA, discNMF and discLDA) in all datasets for both measures ACC and NMI.

• Comparisons among different datasets. In general, the values of measure ACC and NMI for all methods will increase as the actual topic number of each cluster (6 topics for the Reuters data, 4 topics for the 20 news dataset, and 3 topics for the four area dataset) decreases. In the Reuters dataset, where the number of common and distinctive topics is the same, the five methods yield very similar results. In the 20 news dataset, as the common topic number decreases, although all methods see an increase in performance, our proposed CDTM model obtains the largest improvement. Also, in the four area dataset, where the distinctive topic number is less than the common topic number, CDTM is still the best performer, with much better ACC and NMI than other methods.
  
• Comparisons between CDTM and LSTM. Generally, LSTM and CDTM achieve comparable performance, that LSTM achieves better values in ACC while CDTM is the winner in terms of NMI. This is not surprising. As a deep neural network, LSTM works as a “black box.” It performs good in prediction task, but lack of explanation and understanding for output. As a result, LSTM is better in the metric of ACC, but is left behind CDTM in NMI, the metric emphasizing the quality of the clusters.
  
• Comparisons between CDTM and other LDA-based approaches. LDA, discLDA, and proposed CDTM model are all LDA-based approaches. CDTM is the best performer in all three datasets. discLDA is less stable, beating LDA in the datasets from Reuters and 20 news, but less well in the four area dataset. The main difference between CDTM and discLDA is that CDTM is entirely inferred through Gibbs sampling, while parameters in discLDA are estimated by a combination of Gibbs sampling and the EM algorithm. Such combination processes may result in performance instability.
  
• Comparisons among NMF-based approaches. Both the NMF and discNMF algorithms model topics through matrix factorization. The NMF method is slightly better than discNMF for the datasets from Reuters and four area, while discNMF performs better for the 20 news dataset. This indicates that discNMF can perform well when there is some imbalance between common and distinctive topics, but degenerates to standard NMF when common and distinctive topics have similar values. It may also suffer from the instability problem due to greater model complexity.
  
• Comparisons between CDTM and discNMF. In the 20 news dataset, NMF performs better than LDA, and discNMF is better than discLDA. In the four area dataset, LDA is also better than NMF, and discLDA is much better than discNMF. This indicates that LDA-based approaches can obtain good performance when there are more common topics, while NMF-based approaches can generate good results when there are more distinctive topics in the dataset.