EnsP^KDE&IncL^KDE: a hybrid time series prediction algorithm integrating dynamic ensemble pruning, incremental learning, and kernel density estimation

Abstract

Ensemble pruning can effectively overcome several shortcomings of the classical ensemble learning paradigm, such as the relatively high time and space complexity. However, each predictor has its own unique ability. One predictor may not perform well on some samples, but it will perform very well on other samples. Blindly underestimating the power of specific predictors is unreasonable. Choosing the best predictor set for each query sample is exactly what dynamic ensemble pruning techniques address. This paper proposes a hybrid Time Series Prediction (TSP) algorithm to implement one-step-ahead prediction task, integrating Dynamic Ensemble Pruning (DEP), Incremental Learning (IL), and Kernel Density Estimation (KDE), abbreviated as the EnsP^KDE&IncL^KDE algorithm. It dynamically selects proper predictor sets based on the kernel density distribution of all base learners’ prediction values. Due to the characteristic of TSP problems that samples arrive in chronological order, the idea of IL is naturally introduced into EnsP^KDE&IncL^KDE, while DEP is a common technology to address the concept drift issue inherent in IL. The algorithm is divided into three subprocesses: 1) Overproduction, which generates the original ensemble learning system; 2) Dynamic Ensemble Pruning (DEP), achieved by one subalgorithm called EnsP^KDE; 3) Incremental Learning (IL), realized by one subalgorithm termed IncL^KDE. Benefited from the advantages of integrating Dynamic Ensemble Pruning scheme, Incremental Learning paradigm and Kernel Density Estimation, in the experimental results, EnsP^KDE&IncL^KDE demonstrates superior prediction performance to several other state-of-the-art algorithms in fulfilling time series forecasting tasks.

APPENDIX Online Extreme Learning Machine with Kernels

1.1 Extreme Learning Machine (ELM)

As shown in Fig. 11, the architecture of Extreme Learning Machine (ELM) [15] is the same with a SLFN. The weights between its input and hidden layers are initialized as random values. While its output weight matrix is to be computed. Supposing that there are N arbitrary distinct samples (x_i, t_i), where x_i ∈ Rⁿ, t_i ∈ R, ELM could be formulated as:

$$ \sum \limits_{i=1}^L{w}_ig\left({\theta}_i^T\cdot {x}_j+{b}_i\right)={o}_j,\kern0.33em j=1,2,\dots N $$

(13)

where θ_i ∈ Rⁿ represents the weight vector of the connection between the input layer and the i-th hidden neuron, b_i ∈ R represents the bias value of the i-th hidden neuron, g(.) is the activation function, w_i is the weight vector of the connection between the i-th hidden neuron and the output neurons, L denotes the hidden neurons number, and o_j denotes the output of the j-th sample. Eq. (13) can be more concisely formulated as:

$$ H\mathrm{w}=\mathrm{o} $$

(14)

where

$$ H={\left[\begin{array}{ccc}g\left({\uptheta}_1\cdot {\mathrm{x}}_1+{b}_1\right)& \cdots & g\left({\uptheta}_L\cdot {\mathrm{x}}_1+{b}_L\right)\\ {}\vdots & \ddots & \vdots \\ {}g\left({\uptheta}_1\cdot {\mathrm{x}}_N+{b}_1\right)& \cdots & g\left({\uptheta}_L\cdot {\mathrm{x}}_N+{b}_L\right)\end{array}\right]}_{N\times L} $$

(15)

Fig. 11

ELM architecture

o = [o₁, o₂, …, o_N]^T represents the output of ELM, and w = [w₁, w₂, …, w_L]^T is weight matrix connecting the hidden and the output layers. The value of w can be calculated by:

$$ \mathrm{w}={H}^{\varPsi}\mathrm{t} $$

(16)

where t = [t₁, t₂, …, t_N]^T denotes the vector of target values, H^Ψ represents the Moore-Penrose generalized inverse of H.

1.2 Extreme Learning Machine with Kernels (ELMK)

ELMK [16] is one promotion of ELM. Define h(x) ∈ R^d(d > > n) as a mapping from low dimensions to high ones. Let H = [h(x₁), h(x₂), …, h(x_N)]^T, the loss function of ELMK is:

$$ {\displaystyle \begin{array}{c}\mathit{\operatorname{Min}}:\kern0.33em {L}_p=\frac{1}{2}{\left\Vert \mathrm{w}\right\Vert}^2+\frac{C}{2}\sum \limits_{i=1}^N{\xi}_i^2\\ {}s.\kern0.33em t.\kern1em \mathrm{h}{\left({\mathrm{x}}_i\right)}^T\cdot \mathrm{w}={t}_i-{\xi}_i,\kern0.33em i=1,2,\dots, N\end{array}} $$

(17)

Using Lagrange multiplier method to solve Eq. (17), then Eq. (18) can be obtained:

$$ f\left(\mathrm{x}\right)=\mathrm{h}{\left(\mathrm{x}\right)}^T\cdot {H}^T{\left(H{H}^T+\frac{I}{C}\right)}^{-1}\mathrm{t} $$

(18)

where x represents the test instance, and f(x) expresses the predicted value. Let K(x_i, x_j) = h(x_i)^T ⋅ h(x_j), K is a kernel function. Eq. (18) can be rewritten as:

$$ f\left(\mathrm{x}\right)=\left[K\left(\mathrm{x},{\mathrm{x}}_1\right),\dots, K\Big(\mathrm{x},{\mathrm{x}}_N\Big)\right]{\left(\left[\begin{array}{ccc}K\left({\mathrm{x}}_1,{\mathrm{x}}_1\right)& \cdots & K\left({\mathrm{x}}_1,{\mathrm{x}}_N\right)\\ {}\vdots & \ddots & \vdots \\ {}K\left({\mathrm{x}}_N,{\mathrm{x}}_1\right)& \cdots & K\left({\mathrm{x}}_N,{\mathrm{x}}_N\right)\end{array}\right]+\frac{I}{C}\right)}^{-1}\mathrm{t} $$

(19)

1.2.1 Online Sequential ELMK (OS-ELMK)

ELM and ELMK are substantially offline learning models. While in TSP problems, samples tend to enter the model one-by-one or batch-by-batch over time, therefore, online learning is especially important for TSP. And thus, online sequential ELMK (OS-ELMK) [17] came into being.

The corresponding Lagrangian dual problem of Eq. (17) could be formulated as:

$$ {L}_D=\frac{1}{2}{\left\Vert \mathrm{w}\right\Vert}^2+\frac{C}{2}\sum \limits_{i=1}^N{\xi}_i^2-\sum \limits_{i=1}^N{\theta}_i\left(\mathrm{h}{\left({\mathrm{x}}_i\right)}^T\mathrm{w}-{t}_i+{\xi}_i\right) $$

(20)

The KKT optimality conditions of Eq. (20) are listed as below:

$$ \frac{\partial {L}_D}{\mathrm{\partial w}}=\mathrm{w}-\sum \limits_{i=1}^N{\theta}_i\mathrm{h}\left({\mathrm{x}}_i\right)=0\Rightarrow \mathrm{w}=\sum \limits_{i=1}^N{\theta}_i\mathrm{h}\left({\mathrm{x}}_i\right) $$

(21)

$$ \frac{\partial {L}_D}{\partial {\xi}_i}=C{\xi}_i-{\theta}_i=0\Rightarrow {\theta}_i=C{\xi}_i,\kern1em i=1,\dots, N $$

(22)

$$ \frac{\partial {L}_D}{\partial {\theta}_i}=\mathrm{h}{\left({\mathrm{x}}_i\right)}^T\mathrm{w}-{t}_i+{\xi}_i=0,\kern1em i=1,\dots, N $$

(23)

If Eqs. (21) and (22) are substituted into Eq. (23), it could be obtained that:

$$ \sum \limits_{j=1}^N{\theta}_j\mathrm{h}{\left({\mathrm{x}}_i\right)}^T\mathrm{h}\left({\mathrm{x}}_j\right)-{t}_i+\frac{\theta_i}{C},\kern1em i=1,\dots, N $$

(24)

When some new samples $\left({\mathrm{x}}_k^{new},{t}_k^{new}\right),k=1,\dots, m$ arrive, Eq. (24) can be updated as:

$$ \sum \limits_{k=1}^m{\theta}_k^{new}\mathrm{h}{\left({\mathrm{x}}_k^{new}\right)}^T\cdot \mathrm{h}\left({\mathrm{x}}_i\right)+\sum \limits_{j=1}^N{\theta}_j^{\ast}\mathrm{h}{\left({\mathrm{x}}_j\right)}^T\cdot \mathrm{h}\left({\mathrm{x}}_i\right)-{t}_i+\frac{\theta_i^{\ast }}{C}=0,\kern0.33em i=1,\dots, N $$

(25)

$$ \sum \limits_{k=1}^m{\theta}_k^{new}\mathrm{h}{\left({\mathrm{x}}_k^{new}\right)}^T\cdot \mathrm{h}\left({\mathrm{x}}_i^{new}\right)+\sum \limits_{j=1}^N{\theta}_j^{\ast}\mathrm{h}{\left({\mathrm{x}}_j\right)}^T\cdot \mathrm{h}\left({\mathrm{x}}_i^{new}\right)-{t}_i^{new}+\frac{\theta_i^{new}}{C}=0,\kern0.33em i=1,\dots, m $$

(26)

where ${\theta}_i^{\ast }$ indicates the updated θ_i, ${\theta}_k^{new}$ indicates the new k-th θ. It is defined that ${\uptheta}^{new}={\left[{\theta}_1^{new},\dots, {\theta}_m^{new}\right]}^T$, and Δθ = [Δθ₁, Δθ₂, …, Δθ_N]^T, where ${\theta}_j^{\ast }={\theta}_j+\varDelta {\theta}_j$, j = 1,2,…,N. Combining Eq. (24) with Eq. (25), then it can be obtained that:

$$ {\displaystyle \begin{array}{c}\sum \limits_{j=1}^N{\theta}_j^{\ast }K\left({\mathrm{x}}_j,{\mathrm{x}}_i\right)+\sum \limits_{k=1}^m{\theta}_k^{new}K\left({\mathrm{x}}_k^{new},{\mathrm{x}}_i\right)-\sum \limits_{j=1}^N{\theta}_jK\left({\mathrm{x}}_j,{\mathrm{x}}_i\right)+\frac{\theta_i^{\ast }-{\theta}_i}{C}=0\\ {}\Rightarrow \sum \limits_{j=1}^N\left({\theta}_j^{\ast }-{\theta}_j\right){K}_{j,i}+\sum \limits_{k=1}^m{\theta}_k^{new}{K}_{k,i}^{\prime }+\frac{\theta_i^{\ast }-{\theta}_i}{C}=0;\kern1em i=1,\dots, N\end{array}} $$

(27)

where K_{j, i} = K(x_j, x_i), ${K}_{k,i}^{\prime }=K\left({\mathrm{x}}_k^{new},{\mathrm{x}}_i\right)$. Equation (27) can be reformulated as its matrix presentation:

$$ \left(\left[\begin{array}{ccc}{K}_{1,1}& \cdots & {K}_{N,1}\\ {}\vdots & \ddots & \vdots \\ {}{K}_{1,N}& \cdots & {K}_{N,N}\end{array}\right]+\frac{I}{C}\right)\varDelta \uptheta =-\left[\begin{array}{ccc}{K}_{1,1}^{\prime }& \cdots & {K}_{m,1}^{\prime}\\ {}\vdots & \ddots & \vdots \\ {}{K}_{1,N}^{\prime }& \cdots & {K}_{m,N}^{\prime}\end{array}\right]{\uptheta}^{new} $$

(28)

For making the expression more concise, the below definitions are made:

$$ R={\left(\left[\begin{array}{ccc}{K}_{1,1}& \cdots & {K}_{N,1}\\ {}\vdots & \ddots & \vdots \\ {}{K}_{1,N}& \cdots & {K}_{N,N}\end{array}\right]+\frac{I}{C}\right)}^{-1} $$

(29)

$$ {K}_m^{\prime }=\left[\begin{array}{ccc}{K}_{1,1}^{\prime }& \cdots & {K}_{m,1}^{\prime}\\ {}\vdots & \ddots & \vdots \\ {}{K}_{1,N}^{\prime }& \cdots & {K}_{m,N}^{\prime}\end{array}\right] $$

(30)

$$ G=-R{K}_m^{\prime } $$

(31)

Equation (28) can be rewritten compactly as:

$$ \varDelta \uptheta =G{\uptheta}^{new} $$

(32)

Substituting ${\theta}_j^{\ast }={\theta}_j+\varDelta {\theta}_j$, j = 1,2,…,N into Eq. (26), we can get:

$$ {\displaystyle \begin{array}{c}\sum \limits_{k=1}^m{\theta}_k^{new}K\left({\mathrm{x}}_k^{new},{\mathrm{x}}_i^{new}\right)+\sum \limits_{j=1}^N\varDelta {\theta}_jK\left({\mathrm{x}}_j,{\mathrm{x}}_i^{new}\right)\\ {}+\sum \limits_{j=1}^N{\theta}_jK\left({\mathrm{x}}_j,{\mathrm{x}}_i^{new}\right)-{t}_i^{new}+\frac{\theta_i^{new}}{C}=0,\kern1em i=1,\dots, m\end{array}} $$

(33)

The matrix form of the above formula is:

$$ \left(\left[\begin{array}{ccc}{K}_{1,1}^{\prime \prime }& \cdots & {K}_{m,1}^{\prime \prime}\\ {}\vdots & \ddots & \vdots \\ {}{K}_{1,m}^{\prime \prime }& \cdots & {K}_{m,m}^{\prime \prime}\end{array}\right]+\frac{I}{C}\right){\uptheta}^{new}+\left[\begin{array}{ccc}{K}_{1,1}^{\prime }& \cdots & {K}_{1,N}^{\prime}\\ {}\vdots & \ddots & \vdots \\ {}{K}_{m,1}^{\prime }& \cdots & {K}_{m,N}^{\prime}\end{array}\right]\varDelta \uptheta =\left[\begin{array}{c}{t}_1^{new}-f\left({\mathrm{x}}_1^{new}\right)\\ {}\vdots \\ {}{t}_m^{new}-f\left({\mathrm{x}}_m^{new}\right)\end{array}\right] $$

(34)

where ${K}_{i,j}^{{\prime\prime} }=K\left({\mathrm{x}}_i^{new},{\mathrm{x}}_j^{new}\right)$. Let:

$$ {K}_m^{\prime \prime }=\left[\begin{array}{ccc}{K}_{1,1}^{\prime \prime }& \cdots & {K}_{m,1}^{\prime \prime}\\ {}\vdots & \ddots & \vdots \\ {}{K}_{1,m}^{\prime \prime }& \cdots & {K}_{m,m}^{\prime \prime}\end{array}\right] $$

(35)

$$ {\mathrm{e}}_m=\left[\begin{array}{c}{t}_1^{new}-f\left({\mathrm{x}}_1^{new}\right)\\ {}\vdots \\ {}{t}_m^{new}-f\left({\mathrm{x}}_m^{new}\right)\end{array}\right] $$

(36)

where f(.) is the regression function without being trained by using the newly added samples.

The compact matrix-vector expression of Eq. (34) is:

$$ \left({K}_m^{\prime \prime }+\frac{I}{C}\right){\uptheta}^{new}+{\left({K}_m^{\prime}\right)}^TG{\uptheta}^{new}={\mathrm{e}}_m $$

(37)

We get:

$$ {\uptheta}^{new}={\left(\left({K}_m^{\prime \prime }+\frac{I}{C}\right)+{\left({K}_m^{\prime}\right)}^TG\right)}^{-1}{\mathrm{e}}_m $$

(38)

After θ^new has been obtained, ∆θ can be computed according to Eq. (32), and finally the updated θ^update = [(θ + Δθ)^T, (θ^new)^T]^T can be acquired. When the new samples are added, R in Eq. (29) also needs to be updated. The kernel matrix in the updated R^* is:

$$ {K}^{\ast }=\left[\begin{array}{cc}K& {K}_m^{\prime}\\ {}{\left({K}_m^{\prime}\right)}^T& {K}_m^{\prime \prime}\end{array}\right] $$

(39)

The updated R^* is:

$$ {R}^{\ast }={\left({K}^{\ast }+\frac{I}{C}\right)}^{-1} $$

(40)

After a series of matrix calculations, can we get the updated R^*:

$$ {\displaystyle \begin{array}{c}{R}^{\ast }={\left[\begin{array}{cc}R& \begin{array}{ccc}0& \cdots & 0\\ {}\vdots & \ddots & \vdots \\ {}0& \cdots & 0\end{array}\\ {}\begin{array}{ccc}0& \cdots & 0\\ {}\vdots & \ddots & \vdots \\ {}0& \cdots & 0\end{array}& \begin{array}{ccc}0& \cdots & 0\\ {}\vdots & \ddots & \vdots \\ {}0& \cdots & 0\end{array}\end{array}\right]}_{N+m,N+m}\\ {}\kern1em +{\left[\begin{array}{c}G\\ {}\begin{array}{ccc}1& \cdots & 0\\ {}\vdots & \ddots & \vdots \\ {}0& \cdots & 1\end{array}\end{array}\right]}_{N+m,m}{\left(\left({K}_m^{\prime \prime }+\frac{I}{C}\right)+{\left({K}_m^{\prime}\right)}^TG\right)}^{-1}{\left[{G}^T\kern0.5em \begin{array}{ccc}1& \cdots & 0\\ {}\vdots & \ddots & \vdots \\ {}0& \cdots & 1\end{array}\right]}_{m,N+m}\end{array}} $$

(41)

In the end, the online sequential learning algorithm of ELMK (OS-ELMK) is presented, in detail, in Algorithm 4.