ν-Support Vector Regression Model Based on Gauss-Laplace Mixture Noise Characteristic for Wind Speed Prediction

Abstract

Most regression techniques assume that the noise characteristics are subject to single noise distribution whereas the wind speed prediction is difficult to model by the single noise distribution because the noise of wind speed is complicated due to its intermittency and random fluctuations. Therefore, we will present the $\nu$-support vector regression model of Gauss-Laplace mixture heteroscedastic noise (GLM-SVR) and Gauss-Laplace mixture homoscedastic noise (GLMH-SVR) for complex noise. The augmented Lagrange multiplier method is introduced to solve models GLM-SVR and GLMH-SVR. The proposed model is applied to short-term wind speed forecasting using historical data to predict future wind speed at a certain time. The experimental results show that the proposed technique outperforms the single noise technique and obtains good performance.

1. Introduction

Wind speed and wind power prediction is becoming increasingly important, and wind speed prediction is crucial for the control, scheduling, maintenance, and resource planning of wind energy conversion systems [1,2]. However, the volatility and uncertainty of wind speed give a fundamental challenge to power system operations. Because the basic characteristics of the wind is its intermittency and random fluctuations [3,4], the integration of wind power into power systems puts forward a series of challenges. The most effective way to resolve the challenges is to improve the prediction accuracy of wind speed and power forecasting [5,6,7].

In general, there are three important types in building a regression algorithm: model structures, objective functions, and optimization strategies. The model structures include linear or nonlinear functions, neural networks [8,9,10], etc. As for objective functions, empirical risk loss has a great effect on the performance of regression models. The selection of empirical risk loss is mostly dependent on the types of noises [11,12]. For example, squared loss is suitable for Gaussian noise [13,14,15], least absolute deviation loss for Laplacian noise [16], and Beta loss for Beta noise [17,18,19]. By the formula of the optimization method, a series of optimization algorithms are developed [20]. This work mainly studies what should be considered in the optimal architecture of the support vector regression (SVR) model in complex or unknown noise.

Recently, SVR has become an increasingly important technology. In 2000, $\nu$-SVR was introduced by Schölkopf, et al. [21] and automatically computes $ϵ$. Suykens et al. [22,23] constructed least squares SVR with Gaussian noise (LS-SVR). Wu [13] and Pontil et al. [24] constructed $\nu$-SVR with Gaussian noise (GN-SVR). In 2002, Bofinger et al. [25] discovered that the output of a wind turbine system is limited between zero and maximum power and that the error statistics do not follow a normal distribution. In 2007, Zhang et al. [26] and Randazzo et al. [27] proposed the estimation of coherent electromagnetic wave impact in the direction of arrival under Laplace noise environment. Bludszuweit et al. [28] explained the advantages of using Beta probability density function (PDF) instead of Gauss PDF to approximate the error distribution of wind power forecasting. According to Bayesian principle, square loss, Beta loss, or Laplacian loss are optimal when the noise is Gaussian, Beta, or Laplacian, respectively [17,18]. However, in some real-world applications, the noise distribution is complex and unknown if the data are collected in muti-source environments. Therefore, a single distribution attended to describes clearly that the real noise is not optimal and almost impossible [29,30]. Generally speaking, mixture distributions have good approximation capability for any continuous distributions. It can adapt well to unknown or complex noises when we have no prior knowledge of real noise. In 2017, the hybrid forecasting model based on multi-objective optimization [29,31], a hybrid method based on singular spectrum analysis, firefly algorithm, and BP neural network forecast the wind speed of complex noise [32]; this shows that the hybrid method has strong prediction ability. The hybrid of least squares support vector machine [33] is applied to predict the wind speed of unknown noise, which improves the forecasting performance of wind speed. Two novel nonlinear regression models [34] where the noise is fitted by mixture of Gaussian were developed, produced good performance compared with current regression algorithms, and provided superior robustness.

To address the above problem, we try to study the $\nu$-SVR model of Gauss-Laplace mixture noise characteristics for complex or unknown noise distribution. In this case, we must design a method to find the optimal solution of the corresponding regression task. Although there has been a large number of SVR algorithm implementations in the past few years, we introduced the augmented Lagrange multiplier method (ALM) method described in Section 4. Sub-gradient descent method can be used if the task is non-differentiable or discontinuous [17], or sequence minimum optimization algorithm (SMO) can be used if the sample size is large [35].

This work offers the following four contributions: (1) the optimal empirical risk loss for general mixture noise characteristic and Gauss-Laplace mixture noise by the use of Bayesian principle is obtained; (2) the $\nu$-SVR model of mixture noise, Gauss-Laplace mixture homoscedastic noise (GLM-SVR), and Gauss-Laplace mixture heteroscedastic noise (GLMH-SVR) for complex or unknown noise is constructed; (3) the augmented Lagrange multiplier method is applied to solve GLM-SVR, which guarantees the stability and validity of the solution; and (4) GLM-SVR is applied to short-term wind speed forecasting using historical data to predict future wind speed at a certain time and to verify the validity of the proposed technique.

A summary of the rest of this article is organized as follows. Section 2 derives the optimal empirical risk loss using Bayesian principle; Section 3 constructs the $\nu$-SVR model of Gauss-Laplace mixture noise characteristics; Section 4 gives the solution and algorithm design of GLM-SVR; numerical experiments are conducted out on short-term wind speed prediction in Section 5; and Section 6 summarizes this article.

2. Bayesian Principle to Empirical Risk Loss of Mixture Noise

In this section, using the theory of Bayesian principle, we obtain the optimal empirical risk loss of mixture noise characteristics.

Given the following dataset

$$D_L=\{(X_1,Y_1),(X_2,Y_2),\cdots ,(X_L,Y_L)\},$$

(1)

where $X_i={(x_{i1},x_{i2},\cdots ,x_{in})}^T\in R^n$ and $Y_i\in R(i=1,2,\cdots ,L)$ are the datasets, R represents real number set, $R^n$ is the n dimensional Euclidean space, L is the number of sample points, and superscript T denotes the matrix transpose, suppose the sample of dataset $D_L$ is generated by the additive noise function $\epsilon$; the following relationship between the measured values $Y_i$ and predicted values $f(X_i)$ is as follows:

$$Y_i=f(X_i)+{\epsilon }_i(i=1,2,\cdots ,L)$$

(2)

where ${\epsilon }_i$ be random and i.i.d. means independent and identical distribution with $P({\epsilon }_i)$ of mean $\mu$ and standard deviation $\sigma$. In engineering technology, the noise density $P(\epsilon )=P(Y-f(X))$ is unknown. We want to predict the unknown decision function $f(X)$ from the training samples $D_f\subseteq D_L$.

Following References [24,36] by the use of Bayesian principle, in maximum likelihood sense, the optimal empirical risk loss is as follows:

$$l(\epsilon )=l(X,Y,f(X))=-logP(Y-f(X)).$$

(3)

i.e., the optimal empirical risk loss $l(\epsilon )$ is the log-likelihood of the noise model.

The probability density function (PDF) of each single distribution model and the parameters estimation formula under Bayesian principle are summarized in Reference [16]. In particular, the noise $\epsilon$ in Equation (2) is Laplacian, with PDF $P(\epsilon )=\frac{1}{2}{e}^{-\left|\epsilon \right|}$. By Equation (3), the optimal empirical risk loss in the sense of maximum likelihood sense should be $l(\epsilon )=\left|\epsilon \right|$. If the noise in Equation (2) is Gaussian, with zero mean and homoscedastic standard deviation $\sigma$, by Equation (3), empirical risk loss about Gaussian noise is $l(\epsilon )=\frac{1}{2{\sigma }^2}·{\epsilon }^2$. Suppose the noise $\epsilon$ in Equation (2) is Gaussian, with zero mean and heteroscedastic standard deviation ${\sigma }_i$($i=1,2,\cdots ,L$). By Equation (3), the loss about Gaussian noise is $l({\epsilon }_i)=\frac{1}{2{\sigma }_i^2}·{\epsilon }_i^2$ ($i=1,\cdots ,L$).

It is assumed that the noise $\epsilon$ in Equation (2) is the mixture distributions of two kinds of noise characteristics with the probability density functions $P_1(\epsilon )$ and $P_2(\epsilon )$, respectively. Suppose that $P(\epsilon )=[P_1{(\epsilon )}^{{\lambda }_1}]·[P_2{(\epsilon )}^{{\lambda }_2}]$, by Equation (3), the optimal empirical risk loss about the mixture noise distributions is as follows:

$$l(\epsilon )={\lambda }_1·l_1(\epsilon )+{\lambda }_2·l_2(\epsilon ).$$

(4)

where $l_1(\epsilon )>0,l_2(\epsilon )>0$ are convex empirical risk losses of the above two kinds of noise characteristics, respectively. Weight factor ${\lambda }_1,{\lambda }_2\ge 0$ and ${\lambda }_1+{\lambda }_2=1$.

The Gauss-Laplace empirical risk loss for different parameter are shown in Figure 1.

3. Model $\mathit{\nu }$-SVR of Gauss-Laplace Mixture Noise

Given dataset $D_L$, we build a linear regressor $f(X)={\varpi }^T·X+b$, where $\varpi$ denotes the weight vector and b is the bias term. To deal with nonlinear problems, the following summaries can be made [37,38]: the input vector $X_i\in R^n$ is mapped by a nonlinear mapping (chosen a priori) $\Phi$: $R^n\to H$ is the high dimensional feature space H (H is Hilbert space), induced by the kernel matrix $K(X_i,X_j)=(\Phi (X_i)·\Phi (X_j))$ $(i,j=1,2,\cdots ,L)$. $(\Phi (X_i)·\Phi (X_j))$ is the inner product in H, and the kernel mapping $\Phi$ may be any positive definite Mercer kernel. Therefore, we will solve the optimization problem in feature space H. The linear $\nu$-SVR is extended to the nonlinear $\nu$-SVR by using the kernel matrix $K(X_i,X_j)$.

We propose the uniform model $\nu$-SVR of mixture noises (M-SVR). The primal problem of model M-SVR is described as follows:

$$\begin{array}{c}{\displaystyle Min\{g_{P_{M-SVR}}=\frac{1}{2}{\varpi }^T·\varpi +\frac{C}{L}·[\nu ·\epsilon +{\lambda }_1·\sum _{i=1}^L(l_1({\xi }_i)+l_1({\xi }_i^{*}))}\hfill \\ +{\lambda }_2·{\sum }_{i=1}^L(l_2({\xi }_i)+l_2({\xi }_i^{*}))]\}\hfill \\ Subjectto:{\varpi }^T·\Phi (X_i)+b-Y_i\le \epsilon +{\xi }_i\hfill \\ Y_i-{\varpi }^T·\Phi (X_i)-b\le \epsilon +{\xi }_i^{*}\hfill \end{array}$$

(5)

where ${\xi }_i$ and ${\xi }_i^{*}$ are random noises and slack variable at time i. $l_1({\xi }_i),l_1({\xi }_i^{*}),l_2({\xi }_i)$, and $l_2({\xi }_i^{*})>0$ ($i=1,2,\cdots ,L$) are convex empirical risk loss function values for general noise characteristic in the sample point $(X_i,Y_i)\in D_L$ ($i,j=1,2,\cdots ,L$). $C>0$ is the penalty parameter, $\epsilon \ge 0$, and $\nu \in (0,1]$. Weight factor ${\lambda }_1,{\lambda }_2\ge 0$ and ${\lambda }_1+{\lambda }_2=1$.

As a function approximation machine, the objection is to estimate an unknown function $f(X)$ from the training samples $D_f\subseteq D_L$. In the field of practical application, most of the distributions do not satisfy the Gauss distribution, and it also does not obey the Laplace distribution. The noise distribution is unknown or complex; a single distribution intended to describe the real noise is almost impossible. Generally, mixture distributions (as Gauss-Laplace mixed distribution) have good approximation capabilities for any continuous distributions, and it can fit the unknown or complex noise. Therefore, we will use the Gauss-Laplace mixed homoscedastic and heteroscedastic noise distribution to fit the unknown or complex noise characteristics in the next section.

3.1. Model $\nu$-SVR of Gauss-Laplace Mixture Homoscedastic Noise

If the noise in Equation (2) is Gaussian, with zero mean and the homoscedastic standard deviation $\sigma$, by Equation (3), the empirical risk loss of homoscedastic Gaussian noise is $l_1({\xi }_i,{\xi }_i^{*})=\frac{1}{2{\sigma }^2}({\xi }_i^2+{({\xi }_i^{*})}^2)$ and the empirical risk loss of Laplace noise is $l_2({\xi }_i,{\xi }_i^{*})=|{\xi }_i|+|{\xi }_i^{*}|$. We adopt the Gauss-Laplace mixture homoscedastic noise distribution to fit the unknown noise characteristics. By Equation (4), the loss function corresponding to Gauss-Laplace mixture homoscedastic noise characteristics is $l({\xi }_i,{\xi }_i^{*})=\frac{{\lambda }_1}{2}·({\xi }_i^2+{({\xi }_i^{*})}^2)+{\lambda }_2·(|{\xi }_i|+|{\xi }_i^{*}|)$. We put forward a technique of $\nu$-SVR model for Gauss-Laplace mixture homoscedastic noise characteristics (GLM-SVR). The primal problem of model GLM-SVR be described as follows:

$$\begin{array}{c}{\displaystyle Min\{g_{P_{GLM-SVR}}=\frac{1}{2}{\varpi }^T·\varpi +\frac{C}{L}·[\nu ·\epsilon +\frac{{\lambda }_1}{2}·\sum _{i=1}^L({\xi }_i^2+{({\xi }_i^{*})}^2)}\hfill \\ +{\lambda }_2·{\sum }_{i=1}^L({\xi }_i+{\xi }_i^{*})]\}\hfill \\ Subjectto:{\varpi }^T·\Phi (X_i)+b-Y_i\le \epsilon +{\xi }_i\hfill \\ Y_i-{\varpi }^T·\Phi (X_i)-b\le \epsilon +{\xi }_i^{*}\hfill \end{array}$$

(6)

where ${\xi }_i,{\xi }_i^{*}\ge 0(i=1,2,\cdots ,L)$ are random noises and slack variables at time i. $C>0$ is the penalty parameter, $\epsilon \ge 0$, and $\nu \in (0,1]$. Weight factor ${\lambda }_1,{\lambda }_2\ge 0$ and ${\lambda }_1+{\lambda }_2=1$.

Proposition 1.

The solution of the primal problem Equation (6) of model GLM-SVR about ϖ exists and is unique.

Theorem 1.

The dual problem of the primal problem of Equation (6) of model GLM-SVR is as follows:

$$\begin{array}{c}{\displaystyle Max\{g_{D_{GLM-SVR}}=-\frac{1}{2}\sum _{i=1}^L\sum _{j=1}^L({\alpha }_i^{*}-{\alpha }_i)·({\alpha }_j^{*}-{\alpha }_j)·K(X_i,X_j)}\hfill \\ +{\sum }_{i=1}^L({\alpha }_i^{*}-{\alpha }_i)·Y_i-\frac{L}{2C·{\lambda }_1}{\sum }_{i=1}^L[{({\alpha }_i-C·{\lambda }_2)}^2\hfill \\ +{({\alpha }_i^{*}-C·{\lambda }_2)}^2]\}\hfill \\ Subjectto{\sum }_{i=1}^L({\alpha }_i^{*}-{\alpha }_i)=0\hfill \\ 0\le {\alpha }_i,{\alpha }_i^{*}\le \frac{C}{L},i=1,2,\cdots ,L\hfill \\ {\sum }_{i=1}^L({\alpha }_i^{*}+{\alpha }_i)\le C·\nu .\hfill \end{array}$$

(7)

where ${\alpha }_i,{\alpha }_i^{*}(i=1,2,\cdots ,L)$ are Lagrange multipliers and $K(X_i,X_j)$ $(i,j=1,2,\cdots ,L)$ is a kernel matrix. $C>0$ is the penalty parameter, $\nu \in (0,1]$. Weight factor ${\lambda }_1,{\lambda }_2\ge 0$ and ${\lambda }_1+{\lambda }_2=1$.

Proof. 

Let us take Lagrange functional $L_{\varpi }(\varpi ,b,\alpha ,{\alpha }^{*},\xi ,{\xi }^{*},\eta ,{\eta }^{*},\epsilon ,\gamma )$ as

$$\begin{array}{c}{L}_{\varpi }=\frac{1}{2}{\varpi }^T·\varpi +\frac{C}{L}·[\nu ·\epsilon +\frac{{\lambda }_1}{2}·{\sum }_{i=1}^L({\xi }_i^2+{({\xi }_i^{*})}^2)+{\lambda }_2·{\sum }_{i=1}^L({\xi }_i+{\xi }_i^{*})]-\gamma \epsilon \hfill \\ -{\sum }_{i=1}^L({\eta }_i{\xi }_i+{\eta }_i^{*}{\xi }_i^{*})-{\sum }_{i=1}^L{\alpha }_i({\xi }_i+Y_i-{\varpi }^T·\Phi (X_i)-b+\epsilon )\hfill \\ -{\sum }_{i=1}^L{\alpha }_i^{*}({\xi }_i^{*}-Y_i+{\varpi }^T·\Phi (X_i)+b+\epsilon ).\hfill \end{array}$$

To minimize $L_{\varpi }$, let us find partial derivatives $\varpi ,b,\xi ,{\xi }^{*},$ and $\epsilon$. On the basis of KKT (Karush–Kuhn–Tucker) conditions, we get

$${\nabla }_{\varpi }(L_{\varpi })=0,{\nabla }_b(L_{\varpi })=0,{\nabla }_{\epsilon }(L_{\varpi })=0,{\nabla }_{\xi }(L_{\varpi })=0,{\nabla }_{{\xi }^{*}}(L_{\varpi })=0.$$

and have

$${\varpi }_i=\sum _{i=1}^L({\alpha }_i^{*}-{\alpha }_i)·\Phi (X_i),$$

$$\sum _{i=1}^L({\alpha }_i^{*}-{\alpha }_i)=0,$$

$$\frac{C}{L}·\nu -\gamma -\sum _{i=1}^L({\alpha }_i^{*}+{\alpha }_i)=0,$$

$$\frac{C}{L}·({\lambda }_1·{\xi }_i+{\lambda }_2)-{\eta }_i-{\alpha }_i=0,$$

$$\frac{C}{L}·({\lambda }_1·{\xi }_i^{*}+{\lambda }_2)-{\eta }_i^{*}-{\alpha }_i^{*}=0,$$

By substituting the above extreme conditions into $L_{\varpi }$ and by seeking maximum $\alpha ,{\alpha }^{*}$, we obtain the dual problem of Equation (7) of the primal problem of Equation (6) of model GLM-SVR. □

Then, we obtain

$${\varpi }_i=\sum _{i=1}^L({\alpha }_i^{*}-{\alpha }_i)·\Phi (X_i),$$

$$b=\frac{1}{L}\sum _{i=1}^L[Y_i-\sum _{i\in RSV}({\alpha }_i^{*}-{\alpha }_i)·K(X_i,X_j)-\frac{(L·{\alpha }_i-C·{\lambda }_2)}{C·{\lambda }_1}].$$

To estimate $\epsilon$, we have

$$\epsilon =\frac{1}{L}\sum _{j=1}^L(\sum _{i\in RSV}({\alpha }_i^{*}-{\alpha }_i)·K(X_i,X_j)+b-Y_j).$$

Thus, the decision function of model GLM-SVR can be written as

$$f(X)={\varpi }^T·\Phi (X)+b=\sum _{i\in RSV}({\alpha }_i^{*}-{\alpha }_i)K(X_i,X)+b,$$

where $RSV$s are samples about ${\alpha }_i^{*}-{\alpha }_i\ne 0$ (called support vectors), $K(X_i,X_j)=(\Phi (X_i)·\Phi (X_j))(i,j=1,2,\cdots ,L)$ is the Kernel function, $\Phi :R^n\to H$ ($R^n$ is the n dimensional Euclidean space, and H is the Hilbert space), and $(\Phi (X_i)·\Phi (X_j))$ is the inner product of H.

3.2. Model $\nu$-SVR of Gauss-Laplace Mixture Heteroscedastic Noise

If the noise in Equation (2) is Gaussian, with zero mean and the heteroscedastic variance ${\sigma }_i^2,{({\sigma }_i^{*})}^2$, that is ${\sigma }_i\ne {\sigma }_j$, ${\sigma }_i^{*}\ne {\sigma }_j^{*}$, where $i\ne j(i,j=1,\cdots ,L)$, by Equation (3), the empirical risk loss of heteroscedastic Gaussian noise characteristic is $l_1({\xi }_i,{\xi }_i^{*})=\frac{1}{2{\sigma }_i^2}{\xi }_i^2+\frac{1}{2{({\sigma }_i^{*})}^2}{({\xi }_i^{*})}^2$, $(i=1,\cdots ,L)$ and the empirical risk loss of Laplace noise characteristic is $l_2({\xi }_i,{\xi }_i^{*})=|{\xi }_i|+|{\xi }_i^{*}|$. We utilize the Gauss-Laplace mixture heteroscedastic noise distribution to predict the unknown noise characteristics. By Equation (4), the empirical risk loss about Gauss-Laplace mixture heteroscedastic noise is $l({\xi }_i,{\xi }_i^{*})=\frac{{\lambda }_1}{2}·(\frac{1}{{\sigma }_i^2}{\xi }_i^2++\frac{1}{{({\sigma }_i^{*})}^2}{({\xi }_i^{*})}^2)+{\lambda }_2·(|{\xi }_i|+|{\xi }_i^{*}|)$ $(i=1,\cdots ,L)$. We propose a novel technique of $\nu$-SVR model for Gauss-Laplace mixture heteroscedastic noise characteristics (GLMH-SVR). The primal problem of model GLMH-SVR can be formulated as follows:

$$\begin{array}{c}{\displaystyle Min\{g_{P_{GLMH-SVR}}=\frac{1}{2}{\varpi }^T·\varpi +\frac{C}{L}·[\nu ·\epsilon +\frac{{\lambda }_1}{2}·\sum _{i=1}^L(\frac{1}{{\sigma }_i^2}·{\xi }_i^2+}\hfill \\ \frac{1}{{({\sigma }_i^{*})}^2}·{({\xi }_i^{*})}^2)+{\lambda }_2·{\sum }_{i=1}^L({\xi }_i+{\xi }_i^{*})]\}\hfill \\ Subjectto:{\varpi }^T·\Phi (X_i)+b-Y_i\le \epsilon +{\xi }_i\hfill \\ Y_i-{\varpi }^T·\Phi (X_i)-b\le \epsilon +{\xi }_i^{*}.\hfill \end{array}$$

(8)

where ${\xi }_i,{\xi }_i^{*}\ge 0,i=1,2,\cdots ,L$ are random noises and slack variables at time i, the variance ${\sigma }_i^2,{({\sigma }_i^{*})}^2(i=1,2,\cdots ,L)$ is heteroscedastic, $C>0$ is the penalty parameter, $\epsilon \ge 0$, and $\nu \in (0,1]$. Weight factor ${\lambda }_1,{\lambda }_2\ge 0$ and ${\lambda }_1+{\lambda }_2=1$.

Proposition 2.

The solution of the primal problem of Equation (8) of GLMH-SVR about ω exists and is unique.

Theorem 2.

The dual problem of GLMH-SVR in the primal problem of Equation (8) is as follows:

$$\begin{array}{c}{\displaystyle Max\{g_{D_{GLMH-SVR}}=-\frac{1}{2}\sum _{i=1}^L\sum _{j=1}^L({\alpha }_i^{*}-{\alpha }_i)·({\alpha }_j^{*}-{\alpha }_j)·K(X_i,X_j)}\hfill \\ +{\sum }_{i=1}^L({\alpha }_i^{*}-{\alpha }_i)·Y_i-\frac{L}{2C·{\lambda }_1}[{\sum }_{i=1}^L{({\sigma }_i^2·{\alpha }_i-C·{\lambda }_2)}^2\hfill \\ +{\sum }_{i=1}^L{({({\sigma }_i^{*})}^2·{\alpha }_i^{*}-C·{\lambda }_2)}^2]\}\hfill \\ Subjectto{\sum }_{i=1}^L({\alpha }_i^{*}-{\alpha }_i)=0\hfill \\ 0\le {\alpha }_i\le \frac{C}{L·{\sigma }_i^2},i=1,2,\cdots ,L\hfill \\ 0\le {\alpha }_i^{*}\le \frac{C}{L·{({\sigma }_i^{*})}^2},i=1,2,\cdots ,L\hfill \\ {\sum }_{i=1}^L({\alpha }_i^{*}+{\alpha }_i)\le C·\nu ,i=1,2,\cdots ,L\hfill \end{array}$$

(9)

where ${\sigma }_i^2,{({\sigma }_i^{*})}^2(i=1,2,\cdots ,L)$ is heteroscedastic, $C>0$ is the penalty parameter, and $\nu \in (0,1]$. Weight factor ${\lambda }_1,{\lambda }_2\ge 0$ and ${\lambda }_1+{\lambda }_2=1$.

Proof. 

An Appendix A to the proof of Theorem 2. □

We get the following:

$${\varpi }_i=\sum _{i=1}^L({\alpha }_i^{*}-{\alpha }_i)·\Phi (X_i),$$

$$b=\frac{1}{L}\sum _{i=1}^L[Y_i-\sum _{i\in RSV}({\alpha }_i^{*}-{\alpha }_i)·K(X_i,X_j)-\frac{(L·{\sigma }_i^2·{\alpha }_i-C·{\lambda }_2)}{C·{\lambda }_1}].$$

To estimate $\epsilon$, we get use the following:

$$\epsilon =\frac{1}{L}\sum _{j=1}^L(\sum _{i\in RSV}({\alpha }_i^{*}-{\alpha }_i)·K(X_i,X_j)+b-Y_j).$$

Thus, the decision function of model GLMH-SVR can be written as follows:

$$f(X)={\varpi }^T·\Phi (X)+b=\sum _{i\in RSV}({\alpha }_i^{*}-{\alpha }_i)K(X_i,X)+b,$$

where $RSV$s are samples about ${\alpha }_i^{*}-{\alpha }_i\ne 0$ (called support vectors), parameter vector $\omega \in R^n$, $\Phi :R^n\to H$, and $K(X_i,X_j)$ is the Kernel function.

If the noise in Equation (2) is Gaussian, with zero mean and homoscedasticity, Theorem 1 can be derived by Theorem 2.

4. Solution Based on the Augmented Lagrange Multiplier Method

The augmented Lagrange multiplier method (ALM) method [39,40,41] is a class of algorithms for solving equality- and inequality-constrained optimization problems. It solves the dual problem of Equation (7) of model GLM-SVR by applying Newton’s method to a series of constrained problems. By eliminating equality and inequality constraints, the optimization problem of Equation (7) can be reduced to an equivalent unconstrained problem. Gradient descent method or Newton method can be used to solve above problems [24,42,43]. If there are large-scale training samples, some fast optimization techniques can also be combined with the proposed objective function, such as stochastic gradient decent [44].

In this section, we apply Newton’s method to the sequence of inequality and equality constraints and use ALM method to solve model GLM-SVR. Theorem 1 and Theorem 2 provide the algorithms for effectively identifying models GLM-SVR and GLMH-SVR, respectively. The solution based on ALM and algorithm design of model GLM-SVR is given. Similarly, model GLMH-SVR can be solved by the use of ALM.

(1) Let training samples $D_L=\{(X_1,Y_1),(X_2,Y_2),\dots ,(X_L,Y_L)\}$, where $X_i\in R^n$, $Y_i\in R$, $i=1,\dots ,L$.

(2) The 10-fold cross-validation strategy is adopted to search the optimal parameters $C,\nu ,{\lambda }_1,$ and ${\lambda }_2$ and to select the appropriate kernel function $K(•,•)$.

(3) Solve the optimization problem of Equation (7), and get the optimal solution $\alpha =({\alpha }_1,\cdots ,{\alpha }_L),{\alpha }^{*}=({\alpha }_1^{*},\cdots ,{\alpha }_L^{*})$.

(4) Construct the decision function as

$$f(X)={\varpi }^T·\Phi (X)+b=\sum _{i\in RSV}({\alpha }_i^{*}-{\alpha }_i)K(X_i,X)+b,$$

where $b=\frac{1}{L}{\sum }_{i=1}^L[Y_i-{\sum }_{i\in RSV}({\alpha }_i^{*}-{\alpha }_i)·K(X_i,X_j)-\frac{(L·{\alpha }_i-C·{\lambda }_2)}{C·{\lambda }_1}]$

and estimate $\epsilon$ as

$$\epsilon =\frac{1}{L}\sum _{j=1}^L(\sum _{i\in RSV}({\alpha }_i^{*}-{\alpha }_i)·K(X_i,X_j)+b-Y_j).$$

Parameter vector $\varpi \in R^n$, $\Phi :R^n\to H$ (H is Hilbert space), where $(\Phi (X_i)·\Phi (X_j))$ is the inner product of H and $K(X_i,X_j)=(\Phi (X_i)·\Phi (X_j))$ is the Kernel function.

5. Case Study

In this section, a case study is implemented to demonstrate the effectiveness of the proposed model GLM-SVR through comparisons with other techniques for training-set $D_f$ from Heilongjiang, China. This case study includes three subsections: Data collection and analysis in Section 5.1; evaluation criteria for forecasting performance in Section 5.2; and short-term wind speed forecasting of a real dataset in Section 5.3.

5.1. Analysis of Wind Speed Mixture Noise Characteristics

In order to analyze the mixture noise characteristics of wind speed forecasting error, we collected wind speed dataset from Heilongjiang, China. The dataset consists of one-year wind speed data, recording the wind speed values every 10 min. We first found the Gauss-Laplace mixture noise in the above data. The researchers have found that turbulence is the major cause of the wind speed’s strong random fluctuation uncertainty. From wind energy perspective, the most striking characteristic of the wind resource is its variability. Now we display the distributions of wind speed. We obtain a value for wind speed after every 10 min and compute the histograms of wind speed in one or two hours. Two typical distributions are given as follows: one was computed when the wind speed was higher and the other was computed when the wind speed was lower, as shown in Figure 2 and Figure 3, respectively.

To analyze one-month time series of wind speed dataset, the persistence method is used to investigate the distribution of wind speed prediction errors [28]. The result indicates that the error $\xi$ does not satisfy the single distribution but approximately obeys the Gauss-Laplace mixed distribution and that the PDF of $\xi$ is $P(\xi )=\frac{1}{2}{e}^{-\left|\xi \right|}·\frac{1}{2{\sigma }^2}{\xi }^2$, as shown in Figure 4. This is a regression learning task of mixture noise.

5.2. Evaluation Criteria for Forecasting Performance

As we all know, no prediction model forecasts perfectly. There are also certain criteria, such as mean absolute error (MAE), the root mean square error (RMSE), mean absolute percentage error (MAPE), and standard error of prediction (SEP), which are used to evaluate the predictive performance of models $\nu$-SVR, GN-SVR, and GLM-SVR. The four criteria are defined as follows:

$$MAE=\frac{1}{L}\sum _{i=1}^L|Y_i^{\prime }-Y_i|,$$

(10)

$$MAPE=\frac{1}{L}\sum _{i=1}^L\frac{|Y_i^{\prime }-Y_i|}{Y_i}\times 100\%,$$

(11)

$$RMSE=\sqrt{\frac{1}{L}\sum _{i=1}^L{(Y_i^{\prime }-Y_i)}^2},$$

(12)

$$SEP=\frac{RMSE}{\overline{Y}}\times 100\%,$$

(13)

Among them, L is the size of the training samples, $Y_i$ is the ith actual measured data, $Y_i^{\prime }$ is the ith forecasted result, and $\overline{Y}$ is the mean value of observations of all selected samples in the training-set $D_L$ [45,46,47]. The MAE reveals how similar the predicted values are to the observed values, whereas the RMSE measures the overall deviation between the predicted and observed values. MAPE is the ratio between errors and observed values, and SEP is the ratio between RMSE and mean values of observations. The indicators MAPE and SEP are unit-free measures of accuracy for predicting wind series and are sensitive to small changes.

5.3. Short-Term Wind Speed Prediction of Real Dataset

In this subsection, we demonstrate the validity of the proposed model by conducting experiments on wind speed dataset from Heilongjiang Province, China. The data records more than one year of wind speeds. The average wind speed in 10 min are stored. As a whole, 62,466 samples with 4 attributes: mean, variance, minimum, and maximum. We first extracted 2160 consecutive data points (from 1 to 2160; the time length is 15 days) as the training set and 720 consecutive data points (from 2161 to 2880, the time length is 5 days) as the testing set. We transform the original sequence into a multivariate regression task using mode $\overrightarrow{X_i}$ = $(X_{i-10},X_{i-9},\cdots ,X_{i-1},X_i)$ as an input vector to predict $X_{i+step}$, in which the vector orders of wind speed is determined by the chaotic operator network method [48], where $X_j$ is the real value of wind speed at time $j(j=i-10,i-9,\cdots ,i)$, $step=1,3,5$, that is to say, using the above mode to predict the wind speed at each point $X_i$ after 10-min, 30-min, and 50-min, respectively.

Models $\nu$-SVR, GN-SVR, and GLM-SVR have been implemented in Matlab 7.8 programming language. The initial parameters are $C\in [1,201]$, $\nu \in (0,1]$, ${\lambda }_1,$ and ${\lambda }_2\in [0,1]$. We use the 10-fold cross validation strategy to find optimal positive parameters $C,\nu ,{\lambda }_1,$ and ${\lambda }_2$, of which the parameters selection technology is studied in detail in References [49,50]. In this article, the parameter assignments are as follows: $C=181,\nu =0.5,{\lambda }_1=0.5,\mathrm{and}{\lambda }_2=0.5$. Many practical applications display that polynomial and Gaussian kernels perform well under general smooth assumptions. In this case study, polynomial and Gaussian kernel functions are utilized in models $\nu$-SVR, the $\nu$-SVR model of Gauss homoscedastic noise (GN-SVR), and GLM-SVR as below [51,52].

$$K(X_i,X_j)={((X_i,X_j)+1)}^d$$

and

$$K(X_i,X_j)=e^{-\frac{\parallel X_i-X_j{\parallel }^2}{{\sigma }^2}},$$

where d is a positive integer and $\sigma$ is positive.

In Figure 5, Figure 6 and Figure 7, the results of wind speed prediction at the $X_i$ point for models $\nu$-SVR, GN-SVR, and GLM-SVR are obtained after 10 min, 30 min, and 50 min, respectively.

In

Table 1. Error statistic of wind speed prediction at $X_i$-point after 10 min.

Model	MAE (m/s)	RMSE (m/s)	MAPE (%)	SEP (%)
$\nu$-SVR	0.3671	0.4854	2.84	3.84
GN-SVR	0.3641	0.4845	2.77	3.83
GLM-SVR	0.3616	0.4813	2.76	3.81

,

Table 2. Error statistic of wind speed prediction at $X_i$-point after 30 min.

Model	MAE (m/s)	RMSE (m/s)	MAPE (%)	SEP (%)
$\nu$-SVR	0.5655	0.7631	4.44	6.08
GN-SVR	0.5577	0.7589	4.36	6.04
GLM-SVR	0.5468	0.7773	4.08	6.19

and

Table 3. Error statistic of wind speed prediction at $X_i$-point after 50 min.

Model	MAE (m/s)	RMSE (m/s)	MAPE (%)	SEP (%)
$\nu$-SVR	0.6281	0.9142	4.70	7.32
GN-SVR	0.6146	0.8711	4.61	6.98
GLM-SVR	0.5920	0.8734	4.52	7.07

and Figure 8, Figure 9, Figure 10 and Figure 11, indicators $MAE,MAPE,RMSE$, and SEP of wind speed prediction at $X_i$-point for models $\nu$-SVR, GN-SVR, and GLM-SVR are obtained after 10 min, 30 min, and 50 min, respectively.

From Table 1, Table 2 and Table 3 and Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11, in most cases, it can be concluded that the error calculation of model GLM-SVR is better than that of models $\nu$-SVR and GN-SVR. As the prediction horizon increases to 30 min and 50 min, the errors obtained by different models rise and the relative difference decreases. However, as can be seen from Table 1, Table 2 and Table 3, the Gauss-Laplace mixture noise model is slightly superior to the classical model in terms of all indicators: MAE, MAPE, RMSE, and SEP.

6. Conclusions

The noise distribution is complex or unknown in the real world; it is almost impossible for a single distribution to describe real noise. This article describes the main results: (1) optimal empirical risk loss for mixture noise model is derived by the Bayesian principle; (2) model $\nu$-SVR of the Gauss-Laplace mixture homoscedastic noise (GLM-SVR) and Gauss-Laplace mixture heteroscedastic noise (GLMH-SVR) for complex or unknown noise is developed; (3) the dual problems of GLM-SVR and GLMH-SVR are derived by introducing Lagrange functional $L_{\varpi }$; (4) the ALM method is applied to solve model GLM-SVR, which guarantees the stability and validity; and (5) model GLM-SVR is applied to short-term wind speed forecasting using historical data to predict future wind speed at a certain time. The experimental results on real-world data of wind speed confirm the effectiveness of the proposed technique.

Analogously, we can study the Gauss-Laplace mixture noise model of classification, which will be successfully used to solve the classification problem for complex or unknown noise characteristics.