Incremental model selection and ensemble prediction under virtual concept drifting environments

Abstract

Model selection for machine learning systems is one of the most important issues to be addressed for obtaining greater generalization capabilities. This paper proposes a strategy to achieve model selection incrementally under virtual concept drifting environments, where the distribution of learning samples varies over time. To carry out incremental model selection, the system generally uses all the learning samples that have been observed until now. Under virtual concept drifting environments, however, the distribution of the observed samples is considerably different from the distribution of cumulative dataset so that model selection is usually unsuccessful. To overcome this problem, the author had earlier proposed the weighted objective function and model-selection criterion based on the predictive input density of the learning samples. Although the previous method described in the author’s previous study shows good performances to some datasets, it occasionally fails to yield appropriate learning results because of the failure in the prediction of the actual input density. To reduce the adverse effect, the method proposed in this paper improves on the previously described method to yield the desired outputs using an ensemble of the constructed radial basis function neural networks (RBFNNs).

Appendices

Appendix: Derivation of Eq. 18

The expected loss of the ensembled network can be rewrite under the condition ∑ _n π _n  = 1 as follows.

$$ \begin{aligned} E&=\int\left\{F(x)-\sum_{n \in E_{x}}\pi_{n} f_{{ {\varvec{\theta}}^{WLS}_{n}}({\user2{x}})}\right\}^{2} W_{1}({\user2{x}}) P({\varvec{x}}) d{\varvec{x}}\\ &=\int \left\{\sum_{n \in E_{x}} \pi_{n} \left(F({\varvec{x}})-f_{ {\varvec{\theta}}^{WLS}_{n}}({\user2{x}})\right)\right\}^2 W_{1}({\user2{x}}) P({\user2{x}})d{\user2{x}} \end{aligned} $$

(21)

Now, if we assume that there are no correlation between the validations of arbitrary two RBFNN’s, the above equation can approximately be rewrite as follows.

$$ \begin{aligned} E \simeq& \int \sum_{n \in E_{x}} \pi_{n}^2 \left \{F({\user2{x}})-f_{ {\varvec{\theta}}^{WLS}_{n}}({\user2{x}})\right\}^2 W_{1}({\user2{x}}) P({\user2{x}}) d {\user2{x}}\\ =&\sum_{n \in E_{x}} \pi_{n}^2 \int \left \{F({\user2{x}})-f_{ {\varvec{\theta}}^{WLS}_{n}}({\user2{x}})\right\}^2 W_{1}({\user2{x}}) P({\user2{x}}) d{\user2{x}} \end{aligned} $$

(22)

According to the definition of information criterions, the value of information criterion represents the averaged expected error of the statistical model. Therefore, Eq. 22 can be approximated by

$$ E\simeq\sum_{n \in E_{x}}\pi_{n}^2 \hat{E}(\lambda_{n}). $$

(23)

From these discussion, the objective function for π _n is

$$ U \equiv \sum_{n \in E_{x}} \pi_{n}^2 \hat{E}(\lambda_{n}) + \lambda \left( 1-\sum_{n \in E_{x}} \pi_{n} \right), $$

(24)

where λ denotes a Lagrangian coefficient. The solution of Eq. 24 is

$$ \frac{{\partial U}}{\partial \pi_{n}}=0 \Longleftrightarrow \ \pi_{n}=\frac{{\lambda}}{2\hat{E}(\lambda_{n})} $$

(25)

$$ \frac{{\partial U}}{\partial \lambda}=0\ \Longleftrightarrow \sum_{n \in E_{x}}\pi_{n}=1. $$

(26)

From Eqs. 25, 26, we obtain

$$ \pi_{n}^{*}=\frac{{1/\hat{E}(\lambda_{n})}}{\sum_{j \in E_{x}}1/ \hat{E}(\lambda_{j})}. $$

(27)

Derivation of Eq. 19

According to the definition of information criteria, IC _w predicts mean log likelihood:

$$ N_{total}\log\hat{E}(\lambda_{n}) \simeq IC_{w}(\lambda_{n}) $$

(28)

Therefore, we obtain:

$$ \hat{E}(\lambda_{n})\simeq \exp \left(\frac{{1}}{N_{total}} IC_{w}(\lambda_{n}) \right). $$

(29)