Metrics For Evaluating The Accuracy of Solar Power Forecasting
Metrics For Evaluating The Accuracy of Solar Power Forecasting
S. Lu and H. F. Hamann
IBM TJ Watson Research Center
V. Banunarayanan
U.S. Department of Energy
To be presented at 3rd International Workshop on Integration of
Solar Power into Power Systems
London, England
October 21 – 22, 2013
Conference Paper
NREL/CP-5500-60142
October 2013
The submitted manuscript has been offered by an employee of the Alliance for Sustainable Energy, LLC
(Alliance), a contractor of the US Government under Contract No. DE-AC36-08GO28308. Accordingly, the US
Government and Alliance retain a nonexclusive royalty-free license to publish or reproduce the published form of
this contribution, or allow others to do so, for US Government purposes.
This report was prepared as an account of work sponsored by an agency of the United States government.
Neither the United States government nor any agency thereof, nor any of their employees, makes any warranty,
express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of
any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately
owned rights. Reference herein to any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation,
or favoring by the United States government or any agency thereof. The views and opinions of authors
expressed herein do not necessarily state or reflect those of the United States government or any agency thereof.
This report is available at no cost from the National Renewable Energy Laboratory (NREL)
at www.nrel.gov/publications.
Cover Photos: (left to right) photo by Pat Corkery, NREL 16416, photo from SunEdison, NREL 17423, photo by Pat Corkery, NREL
16560, photo by Dennis Schroeder, NREL 17613, photo by Dean Armstrong, NREL 17436, photo by Pat Corkery, NREL 17721.
Printed on paper containing at least 50% wastepaper, including 10% post consumer waste.
Metrics for Evaluating the Accuracy of Solar
Power Forecasting
Jie Zhang, Bri-Mathias Hodge, Anthony Florita Siyuan Lu, Hendrik F. Hamann Venkat Banunarayanan
National Renewable Energy Laboratory IBM TJ Watson Research Center U.S. Department of Energy
Golden, CO, USA Yorktown Heights, NY, USA Washington, D.C., USA
Abstract—Forecasting solar energy generation is a challenging an advanced statistical method for solar power forecasting
task due to the variety of solar power systems and weather based on artificial intelligence techniques. Crispim et al. [7]
regimes encountered. Forecast inaccuracies can result in used total sky imagers (TSI) to extract cloud features using a
substantial economic losses and power system reliability issues. radial basis function neural network model for time horizons
This paper presents a suite of generally applicable and
from 1 to 60 minutes. Chow et al. [8] also used TSI to
value-based metrics for solar forecasting for a comprehensive
set of scenarios (i.e., different time horizons, geographic forecast short-term global horizontal irradiance. The results
locations, applications, etc.). In addition, a comprehensive suggested that TSI was useful for forecasting time horizons
framework is developed to analyze the sensitivity of the up to 15 to 25 minutes. Marquez and Coimbra [9] presented a
proposed metrics to three types of solar forecasting method using TSI images to forecast 1-minute averaged
improvements using a design of experiments methodology, in direct normal irradiance at the ground level for time horizons
conjunction with response surface and sensitivity analysis between 3 and 15 minutes. As discussed above, different
methods. The results show that the developed metrics can solar irradiance forecast methods have been developed for
efficiently evaluate the quality of solar forecasts, and assess the various timescales. Loren et al. [10] showed that cloud
economic and reliability impact of improved solar forecasting.
movement–based forecasts likely provide better results than
Keywords-grid integration; ramps; response surface; solar NWP forecasts for forecast timescales of 3 to 4 hours or less.
forecasting; sensitivity analysis; uncertainty; variability Beyond that, NWP models tend to perform better.
B. Research Motivation and Objectives
I. INTRODUCTION
Significant work has been done to develop solar
The utility solar assessment study reported that solar
forecasting models. However, evaluation of the performance
power could provide 10% of U.S. power needs by 2025 [1].
of different forecasting methodologies is still not
At these high levels of solar energy penetration, solar power
straightforward, because different researchers use different
forecasting will become very important for electricity system
metrics as their own criteria. In addition, solar forecasting
operations. Solar forecasting is a challenging task, and solar
accuracy is dependent on geographic location and timescale
power generation presents different challenges for the
of the data. Conventional measures of solar forecasting
transmission and distribution networks of the grid,
accuracy include root mean square error (RMSE), mean bias
respectively. On the transmission side, solar power takes the
error (MBE), and mean absolute error (MAE) [3, 4].
form of centralized solar plants, a non-dispatchable
Marquez and Coimbra [11] proposed a metric for using the
component of the generation pool. On the distribution side,
ratio of solar uncertainty to solar variability to compare
solar power is generated by a large number of distributed
different solar forecasting models. Espinar et al. [12]
panels installed on building rooftops, which has the effect of
proposed several metrics based on the Kolmogorov–Smirnov
modulating the load. Forecast inaccuracies of solar power
test to quantify differences between the cumulative
generation can result in substantial economic losses and
distribution functions of actual and forecast solar irradiation
power system reliability issues because electric grid
data. However, many of the forecast metrics developed do not
operators must continuously balance supply and demand to
take into account the types of errors that have the most impact
maintain the reliability of the grid [2].
on power system operations. Extreme forecasting errors can
A. Overview of Solar Forecasting have disproportionate economic and reliability impacts on
operations, and therefore must be emphasized to some degree
Solar irradiance variations are caused primarily by cloud by any metric that wishes to capture the true impact of the
movement, cloud formation, and cloud dissipation. In the forecasts. Establishing a standard set of metrics for assessing
literature, researchers have developed a variety of methods solar forecasting accuracy is (i) critical to evaluating the
for solar power forecasting, such as the use of numerical success of a solar forecasting effort, and (ii) useful for
weather prediction (NWP) models [3-5], tracking cloud decision making of power system operators under the
movements from satellite images [6], and tracking cloud scenario of a high penetration of solar power.
movements from direct ground observations with sky The objective of this study is to develop a suite of
cameras [7-9]. NWP models are the most popular method for generally applicable, value-based metrics for solar
forecasting solar irradiance several hours or days in advance. forecasting for a comprehensive set of scenarios (different
Mathiesen and Kleissl [4] analyzed the global horizontal time horizons, geographic locations, applications, etc.),
irradiance in the continental United States forecasted by which can assess the economic and reliability impact of
three popular NWP models: the North American Model, the improved solar forecasting. The sensitivity of proposed
Global Forecast System, and the European Centre for metrics to improved solar forecasts is also analyzed. The next
Medium-Range Weather Forecasts. Chen et al. [5] developed section presents the developed metrics for different types of
Corresponding author, email: jie.zhang@nrel.gov forecasts and applications. Section III summarizes the solar
forecast errors; (iii) uncertainty quantification and The MAE metric is also a global error measure metric, which,
propagation metrics, including standard deviation and unlike the RMSE metric, does not excessively account for
information entropy of forecast errors; (iv) ramping extreme forecast events.
characterization metrics, including swinging door algorithm The MAPE and MBE are expressed as
signal compression and heat maps; and (v) economic and 𝑁
reliability metrics, including non-spinning reserves service 1 𝑝𝑝𝑖 − 𝑝𝑖
represented by 95th percentiles of forecast errors. A detailed 𝑀𝑀𝑀𝑀 = �� � (5)
𝑁 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐
formulation and physical explanation of each metric is 𝑖=1
described in the following sections.
1 N
A. Statistical Metrics MBE = ∑ ( pˆ i − pi )
N i =1
(6)
1) Pearson’s correlation coefficient The MBE metric intends to indicate average forecast bias.
Pearson’s correlation coefficient is a measure of the Understanding the overall forecast bias (over- or under-
correlation between two variables (or sets of data). In this forecasting) would allow power system operators to better
paper, the Pearson’s correlation coefficient, ρ, is defined as allocate resources for compensating forecast errors in the
the covariance of actual and forecast solar power variables dispatch process.
divided by the product of their standard deviations, which is 4) Kolmogorov–Smirnov test integral (KSI) and OVER
mathematically expressed as: metrics
cov( p, pˆ )
ρ= (1) The KSI and OVER metrics were proposed by Espinar et
σ pσ pˆ al. [12]. The Kolmogorov–Smirnov (KS) test is a
nonparametric test to determine if two data sets are
where p and p̂ represent the actual and forecast solar significantly different. The KS statistic D is defined as the
power output, respectively. Pearson’s correlation coefficient maximum value of the absolute difference between two
is a global error measure metric; a larger value of Pearson’s cumulative distribution functions (CDFs), expressed as [12]
correlation coefficient indicates an improved solar
forecasting skill. D = max F ( pi ) − Fˆ ( pi ) (7)
2) Root mean squared error (RMSE) and normalized root
mean squared error (NRMSE) where F and F̂ represent the CDFs of actual and forecast
solar power generation data sets, respectively. The associated
The RMSE also provides a global error measure during null hypothesis is elaborated as follows: if the D statistic
the entire forecasting period, which is given by characterizing the difference between one distribution and the
reference distribution is lower than the threshold value Vc ,
1 N
RMSE =
N
∑ ( pˆ − p )
i i
2
(2) the two data sets have a very similar distribution and could
i =1
statistically be the same. The critical value Vc depends on the
where pi represents the actual solar power generation at the number of points in the forecast time series, which is
th calculated for a 99% level of confidence [12].
i time step, p̂i is the corresponding solar power
1.63
generation estimated by a forecasting model, and N is the Vc = , N ≥ 35 (8)
number of points estimated in the forecasting period. To N
compare the results from different spatial and temporal The difference between the CDFs of actual and forecast
scales of forecast errors, we normalized the RMSE using the power is defined for each interval as [12]
capacity value of the analyzed solar plants.
OVER
OVERPer (%) = ×100 (14)
ac
The parameter t is defined by [12]
D j − Vc if D j > Vc
t= (15)
0 if D j ≤ Vc
(a) Single plant (b) Denver region
As with the KSIPer metric, a smaller value of OVERPer
indicates a better performance of the solar power forecasting.
5) Skewness and kurtosis
Skewness is a measure of the asymmetry of the
probability distribution, and is the third standardized moment,
given by
e − µ 3
γ = E e
(16)
σ e
(c) State of Colorado region (d) Western Interconnection
where γ is the skewness; e is the solar power forecast Figure 1. Solar plants at different geographic locations.
error, which is equal to the forecast minus the actual solar
1) Different geographic locations
power value; and µ e and σ e are the mean and standard
deviation of forecast errors, respectively. Assuming that The data used in this work is obtained from the Western
forecast errors are equal to forecast power minus actual Wind and Solar Integration Study Phase 2 (WWSIS-2), one
power, a positive skewness of the forecast errors leads to an of the world’s largest regional integration studies to date
over-forecasting tail, and a negative skewness leads to an [15]. Details of the actual and forecast solar power data are
under-forecasting tail. The tendency to over-forecast (or summarized in Section III. Four scenarios are analyzed based
under-forecast) is important in that the system actions taken on latitude and longitude locations of solar power plants. The