Semi-supervised Collective Classification in Multi-attribute Network Data

Abstract

Multi-attribute network refers to network data with multiple attribute views and relational view. Although semi-supervised collective classification has been investigated extensively, little attention is received for such kind of network data. In this paper, we aim to study and solve the semi-supervised learning problem for multi-attribute networks. There are two important challenges: (1) how to extract effective information from the rich multi-attribute and relational information; (2) how to make use of unlabeled data in the network. We propose a new generative model with network regularization, called MARL, which addresses the two challenges. In the approach, a generative model based on the probabilistic latent semantic analysis method is developed to leverage attribute information, and a network regularizer is incorporated to smooth label probability with relational information and unlabeled data. Comprehensive experiments on various data sets have been conducted to demonstrate the effectiveness of the proposed MARL, and the results reveal that our approach outperforms existing collective classification methods and multi-view classification methods in terms of accuracy.

Appendix

Definition 1

${\mathcal {U}}(\varTheta , {\varTheta }^{\prime })$ is an auxiliary function for ${\mathcal {O}}(\varTheta )$ if the following conditions are satisfied

$$\begin{aligned} {\mathcal {U}}(\varTheta , {\varTheta }^{\prime }) \le {\mathcal {O}}(\varTheta ),\quad {\mathcal {U}}(\varTheta , {\varTheta }) = {\mathcal {O}}(\varTheta ) \end{aligned}$$

(15)

Lemma 1

If ${\mathcal {U}}$ is an auxiliary function of ${\mathcal {O}}$, then ${\mathcal {O}}$ is non-decreasing under the update

$$\begin{aligned} {\varTheta }^{r+1} = \mathop {{{\mathrm{argmax}}}}_{\varTheta }{{\mathcal {U}}(\varTheta , {\varTheta }^{r})} \end{aligned}$$

(16)

Proof

$$\begin{aligned} {\mathcal {O}}\big ({\varTheta }^{r+1}\big ) \ge {\mathcal {U}}(\varTheta ^{r+1}, {\varTheta }^{r}) \ge {\mathcal {U}}(\varTheta ^{r}, {\varTheta }^{r}) = {\mathcal {O}}({\varTheta }^{r}). \end{aligned}$$

$\square$

Lemma 2

Function

$$\begin{aligned} {\mathcal {U}}(\varTheta , \varTheta ^r)= & {} \sum _{i=1}^N \sum _{t=1}^T \sum _{j=1}^{M_t} n\big (x_i,w_j^t,A_t\big )\sum _{k=1}^K\sum _{q=1}^{Q^t} P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )^r \nonumber \\&\times \log \frac{P\big (w_j^t|z_q^t,A_t\big )P\big (z_q^t|c_k,A_t\big )P\big (c_k |x_i \big )}{P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )^r} - \lambda \sum _{i=1}^N \sum _{s=1}^N D\big (P_i(c), P_s(c)\big ) E_{is}\nonumber \\ \end{aligned}$$

(17)

is an auxiliary function for the objective function ${\mathcal {O}}(\varTheta )$ in Eq. (5), where

$$\begin{aligned} \varTheta = \{P(c_k|x_i), P\big (z_q^t|c_k,A_t\big ), P\big (w_j^t|z_q^t,A_t\big )\} \end{aligned}$$

is the parameters,

$$\begin{aligned}&\varTheta ^{r} = \{P(c_k|x_i)^{r}, P\big (z_q^t|c_k,A_t\big )^{r}, P\big (w_j^t|z_q^t,A_t\big )^{r}\},\nonumber \\&\begin{aligned} P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )^r =\frac{P\big (w_j^t|z_q^t,A_t\big )^r P\big (z_q^t|c_k,A_t\big )^r P\big (c_k |x_i \big )^r}{\sum _{k=1}^K\sum _{q=1}^{Q^t} P\big (w_j^t|z_q^t,A_t\big )^r P\big (z_q^t|c_k,A_t\big )^r P\big (c_k |x_i\big )^r}. \end{aligned}\qquad \end{aligned}$$

(18)

Proof

According to Jensen’s inequality, we have

$$\begin{aligned} \begin{aligned} {\mathcal {U}}(\varTheta , \varTheta ^r)=&\sum _{i=1}^N \sum _{t=1}^T \sum _{j=1}^{M_t} n\big (x_i,w_j^t,A_t\big )\sum _{k=1}^K\sum _{q=1}^{Q^t} P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )^r \\&\times \log \frac{P\big (w_j^t|z_q^t,A_t\big )P\big (z_q^t|c_k,A_t\big )P\big (c_k |x_i \big )}{P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )^r} - \lambda \sum _{i=1}^N \sum _{s=1}^N D\big (P_i(c), P_s(c)\big ) E_{is}\\ \le&\sum _{i=1}^N \sum _{t=1}^T \sum _{j=1}^{M_t} n\big (x_i,w_j^t,A_t\big ) \log \sum _{k=1}^K\sum _{q=1}^{Q^t} P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )^r\\&\frac{P\big (w_j^t|z_q^t,A_t\big )P\big (z_q^t|c_k,A_t\big ) P\big (c_k |x_i \big )}{P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )^r} - \lambda \sum _{i=1}^N \sum _{s=1}^N D\big (P_i(c), P_s(c)\big ) E_{is} \\ =&\sum _{i=1}^N \sum _{t=1}^T \sum _{j=1}^{M_t} n\big (x_i,w_j^t,A_t\big ) \log \sum _{k=1}^K\sum _{q=1}^{Q^t} P\big (w_j^t|z_q^t,A_t\big )P\big (z_q^t|c_k,A_t\big )P\big (c_k |x_i\big ) \\&- \lambda \sum _{i=1}^N \sum _{s=1}^N D\big (P_i(c), P_s(c)\big ) E_{is} \\ =\,&{\mathcal {O}}(\varTheta ), \end{aligned} \end{aligned}$$

It is also easy to verify that

$$\begin{aligned} \begin{aligned} {\mathcal {U}}(\varTheta , \varTheta )=&\sum _{i=1}^N \sum _{t=1}^T \sum _{j=1}^{M_t} n\big (x_i,w_j^t,A_t\big )\sum _{k=1}^K\sum _{q=1}^{Q^t} P\big (z_q^t,c_k|x_i,w_j^t,A_t\big ) \\&\times \log \frac{P\big (w_j^t|z_q^t,A_t\big )P\big (z_q^t|c_k,A_t\big )P\big (c_k |x_i \big )}{P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )} - \lambda \sum _{i=1}^N \sum _{s=1}^N D\big (P_i(c), P_s(c)\big ) E_{is}\\ =&\sum _{i=1}^N \sum _{t=1}^T \sum _{j=1}^{M_t} n\big (x_i,w_j^t,A_t\big )\sum _{k=1}^K\sum _{q=1}^{Q^t} P\big (z_q^t,c_k|x_i,w_j^t,A_t\big ) \\&\times \log \sum _{k=1}^K\sum _{q=1}^{Q^t} P\big (w_j^t|z_q^t,A_t\big )P\big (z_q^t|c_k,A_t\big )P\big (c_k |x_i\big ) \\&- \lambda \sum _{i=1}^N \sum _{s=1}^N D\big (P_i(c), P_s(c)\big ) E_{is} \\ =\,&{\mathcal {O}}(\varTheta ). \end{aligned} \end{aligned}$$

Hence, the result follows. $\square$

Lemma 3

Maximizing

$$\begin{aligned} {\varTheta }^{r+1} = \mathop {{{\mathrm{argmax}}}}_{\varTheta }{\mathcal {Q}}(\varTheta ) \end{aligned}$$

(19)

is equivalent to maximizing the update in Eq. (16), where ${\mathcal {Q}}(\varTheta )$ is the expected complete data log-likelihood in Eq. (7).

Proof

$$\begin{aligned} \begin{aligned} {\mathcal {U}}(\varTheta , \varTheta ^r)=&\sum _{i=1}^N \sum _{t=1}^T \sum _{j=1}^{M_t} n\big (x_i,w_j^t,A_t\big )\sum _{k=1}^K\sum _{q=1}^{Q^t} P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )^r \\&\times \log \frac{P\big (w_j^t|z_q^t,A_t\big )P\big (z_q^t|c_k,A_t\big )P\big (c_k |x_i \big )}{P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )^r} - \lambda \sum _{i=1}^N \sum _{s=1}^N D\big (P_i(c), P_s(c)\big ) E_{is}\\ =&\sum _{i=1}^N \sum _{t=1}^T \sum _{j=1}^{M_t} n\big (x_i,w_j^t,A_t\big )\sum _{k=1}^K\sum _{q=1}^{Q^t} P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )^r \\&\times \log P\big (w_j^t|z_q^t,A_t\big )P\big (z_q^t|c_k,A_t\big )P\big (c_k |x_i\big ) \\&- \sum _{i=1}^N \sum _{t=1}^T \sum _{j=1}^{M_t} n\big (x_i,w_j^t,A_t\big )\sum _{k=1}^K\sum _{q=1}^{Q^t} P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )^r\\&\quad \times \log P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )^r- \lambda \sum _{i=1}^N \sum _{s=1}^N D\big (P_i(c), P_s(c)\big ) E_{is}\\ =&{\mathcal {Q}}(\varTheta )- \sum _{i=1}^N \sum _{t=1}^T \sum _{j=1}^{M_t} n\big (x_i,w_j^t,A_t\big )\\&\sum _{k=1}^K\sum _{q=1}^{Q^t} P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )^r \log P\big (z_q^t,c_k|x_i,w_j^t,A_t\big )^r \\ \end{aligned} \end{aligned}$$

The second term is independent of $\varTheta$, which can be treated as constant. Thus, maximizing the expected complete data log-likelihood function ${\mathcal {Q}}(\varTheta )$ is equivalent to maximizing ${\mathcal {U}}(\varTheta , {\varTheta }^{r})$.

$\square$

Next, we give the proof of Theorem 1.

Proof

According to Lemma 2, we know that ${\mathcal {U}}(\varTheta , \varTheta ^r)$ is an auxiliary function for ${\mathcal {O}}(\varTheta )$. From Lemma 1, we thus know that iteratively maximizing ${\mathcal {U}}(\varTheta , \varTheta ^r)$ leads to a non-decreasing of function ${{\mathcal {O}}}(\varTheta )$. Lemma 3 tells us that maximizing ${\mathcal {U}}(\varTheta , \varTheta ^r)$ is equivalent to maximizing ${{\mathcal {Q}}}(\varTheta )$. As Eqs. (8) and (14) are the exact update rules for maximizing ${\mathcal {Q}}(\varTheta )$, we thus know that ${\mathcal {O}}(\varTheta )$ is also non-decreasing under the two update rules. The result follows. $\square$