Explaining When Deep Learning Models Are Better for Time Series Forecasting ^†

Abstract

There is a gap of knowledge about the conditions that explain why a method has a better forecasting performance than another. Specifically, this research aims to find the factors that can influence deep learning models to work better with time series. We generated linear regression models to analyze if 11 time series characteristics influence the performance of deep learning models versus statistical models and other machine learning models. For the analyses, 2000 time series of M4 competition were selected. The results show findings that can help explain better why a pretrained deep learning model is better than another kind of model.

1. Introduction

Different researchers have compared the performances of several forecasting methods using samples of time series to find out which method usually gives better results in forecasting regression tasks. Although the studies identified a method that tends to provide better results than others in general terms, there is no method that will always perform better [1,2,3,4]. It means that there could be an algorithm that gives better results when looking at the average of the test error, but it will not be the best for each time series. Therefore, the research must move from identifying the best algorithm at a global level to the analysis of the factors or reasons that influence an algorithm to work better under certain conditions.

On the other hand, in the field of time series, deep learning models have been gaining more relevance in different areas because of their good predictive capacity [5,6]. However, there is a gap in the knowledge about when they can provide better predictions and the conditions that favors those kinds of models. In [7], the athors found in a review of the literature that there is a lack of rigorous scientific publications that explain the benefits and limitations of the most popular algorithms for time series prediction. Hence, this research aims to identify the factors that can influence deep learning models to work better. The results could be useful to understand the behavior of deep learning in a time series. It can help to identify the conditions for using those methods, suggest transformations in the time series to improve the prediction of the model, and clarify the elements that should be investigated in learning architectures in order to overcome their weaknesses.

2. Literature Review

In this review, we describe studies that have been focused on comparing models using various time series. Most of them did not intend to understand the reason one model is better than another.

The first studies comparing the performance of statistical traditional models versus machine learning models on a larger scale were those of [3,4]. Both studies compared simple neuronal networks with other models using the M3 competition dataset. Recently, deep learning models have been gaining relevance in the field of forecasting. Therefore, some studies have emerged to compare the performance of deep learning with other machine learning models and statistical models on a larger scale. For example, [7] compared the performance of LSTM and MLP with other machine learning methods and statistical methods such as ARIMA and SARIMA. A total of 95 datasets were used for the comparisons. They used two strategies to evaluate their models: the multi-step-ahead projection with approximate iteration and the multi-step-ahead with updated iteration. Using both strategies, the LSTM was overcome by machine learning and statistical models in chaotic, stochastic, and deterministic time series.

In [8], they selected 120 intermittent and lumpy time series of M5 competition datasets to compare the performance of statistical, machine learning, and deep learning models. They found that modern deep learning methods, especially LSTMs, achieved good performance but not the best and that, with intermittent, as well as lumpy, time series, other models exceeded the deep learning models. In [9], using 1000 time series of M4 competition found that deep learning models had better results than machine learning models and statistical models. In [10], a comparative study was conducted on nine datasets (each dataset had several time series) with hourly, daily, and minute time series. The goal was to analyze the performance of a Gradient Boosting Regression Tree (GBRT) versus deep learning models like LSTM, Temporal Fusion Transformer, DeepAR, and others. The results suggest that GBRT can compete with state-of-the-art DNN models and sometimes outperform them by efficiently feature engineering the input and output structures of the GBRT. In [11], a comparison was made between LSTM and ARIMA based on a monthly financial time series. Their findings showed that LSTM outperforms the ARIMA model. Likewise, [12] found that deep learning models outperformed ARIMA. The authors predicted the aviation demand of 48 airports and multiple different aviation routes.

3. Materials and Methods

3.1. Data

We use the monthly M4 competition dataset [13]. We joined the training and testing datasets of the M4 competition, and then, 2000 times series were randomly selected for the training of deep learning models, and the other 2000 time series were selected (from the target dataset) to apply and evaluate the transfer learning. This second group of time series was called the target dataset.

3.2. Models

The deep learning models implemented were three of the most traditional methods used in time series: Long Short-Term Memory (LSTM), Temporal Convolutional Network (TCN), and Convolutional Neuronal Network (CNN). In order to make comparisons, we developed three of the most used statistical methods: ARIMA, ETS, and THETA, and another three methods of machine learning: XGBoost, Random Forest, and Support Vector Machines. All models were generated to predict 12 months into the future. In the case of the machine learning models, we used as input the past 12 months.

3.3. Training and Testing Process

The deep learning models were trained using the Adam optimizer and a stop criterion that consisted of stopping after two epochs without improvement in the validation sample’s loss function. The loss function was the mean absolute percentage error, and the batch size was equal to one, because it gave the best results.

The back-testing process with three repetitions was applied to the 2000 time series of the target dataset to transfer the knowledge of the deep learning models. Each time series was divided into two parts: one part for training and twelve future points for testing. The MAPE metric was used to compute the performance. We averaged the three MAPE results to obtain the final performance measure of each model with each metric. For the other models (machine learning and statistical models), the process was similar, but instead of using the training sample for transfer learning, it was used to train the models.

For the transfer learning, only the last layer of the model was unfrozen, and the weights were updated with a learning rate of 0.000005, which was less than the one used in the training phase. We decided to create pretrained models instead to train the models with the target time series, because it was shown that, in a monthly time series, pretrained deep learning models show better results [9].

The hyperparameters of each model were obtained with Bayesian optimization. The hyperparameters calibrated in the models during the training are in

Table 1. Grid search for the Bayesian optimization of machine learning and deep learning models.

CNN: ✓ Number of layers between 1 and 2 (Batch normalization is applied after the first layer) ✓ Filters between 12 and 132 with a step of 24 ✓ Kernel size between 2 and 12 with a step of 2 ✓ Max pooling with a size of 2 or without max pooling ✓ Activation function among linear, ReLU, and tanh ✓ Learning rate among 0.001, 0.0001, 0.00001	TCN: ✓ Filters between 12 and 132 with a step of 24 ✓ Kernel size between 2 and 12 with a step of 2 ✓ Activation function among linear, ReLU, and tanh ✓ Return sequences True or False ✓ Learning rate among 0.001, 0.0001, 0.00001 ✓ Learning rate among 0.001, 0.0001, 0.00001 ✓ Dilations between [1,2,4,8] or [1,2,4,8,16]	LSTM: ✓ Recurrent units between 12 and 132 with a step of 24 ✓ Activation function among linear, ReLU, and tanh ✓ Return sequences True or False ✓ Learning rate among 0.001, 0.0001, 0.00001
XGBoost: ✓ Max depth between 2 and 12 ✓ Learning rate between 0.01 and 1 ✓ estimators between 10 and 150	Support Vector Machines: ✓ C between 0.01 and 10 ✓ Gamma between 0.001 and 0.33 ✓ Kernel = rvf	Random Forest: ✓ estimators between 10 and 250 ✓ max_features between 1 and 15 ✓ min sample leaf between 2 and 8 ✓ mas samples between 0.70 and 0.99

.

3.4. Features for Time Series Characterization

Eleven features were chosen to characterize each time series. All the time series were standardized in a range between 2 and 3, before the estimation of the features, in order to have the same scale. The names and explanations of the features are in

Table 2. Features computed for each time series.

Name	Explanation	Package
size	Number time series points	-
stl_features_trend	The trend is a long-term change in the mean level [14]. This metric measures the strength of the trend. Higher values mean higher strength of the trend. The formula applied is analogous to [14]. $Y_t=f_t+s_{1,t}+\dots +s_{m,t}+e_t$, where $f_t$ = smoothed trend component, $s_{i,t}$ = i-th seasonal component, and $e_t$ = remainder component. Trend = $1-\frac{V{ar(e}_t)}{Var(f_t+e_t)}$.	stl_features function from package tsfeatures
alpha_tsf	Alpha parameter of simple exponential smoothing. This feature measures the relevance of recent periods. High values mean more weight to recent lags for the forecast, and lower values imply a more even distribution of weight.	tsfeatures
kurtosis_fisher	It is a measure to characterize the probability distribution of the time series. Higher values indicate more concentration around the mean.	PerformanceAnalytics
out	It averages the proportion of outliers according to two different measures. (a) The probability of being an outlier based on model-based clustering [15]. (b) Use of Student’s t-test scores to find outliers [16].	detect_outliers of package tsrobprep and scores from outliers package
pearson_test	The p-value of the normality Pearson’s test. Higher values indicate that the distribution of the time series is closer to the normal assumption.	pearson.test from nortest library
adf	The p-value of the augmented Dickey–Fuller test for the null hypothesis of a non-stationary time series. Higher values indicate that the time series is non-stationary.	adf.test from tseries package. K = 12
stl_features_linearity	Measures the linearity of a time series based on the coefficients of an orthogonal quadratic regression [17].	tsfeatures
stl_features_seasonal_strength	Measure the strength of the seasonality. Higher values mean higher strength. Following the decomposition of the time series described in the previous feature, the computation is next: seasonal = $1-\frac{V{ar(e}_t)}{Var\left(s_{i,t}+e_t\right)}.$	tsfeatures
gamma_tsf	Gamma parameter from ETS model. High values mean more weight to the recent stational period.	tsfeatures
white_noise	The p-value of the Box test. The null hypothesis is that the time series points are independently distributed. Higher values indicate the time series points are independent, and therefore, the data can be white noise.	Box.test from tseries package. We apply lag = 12
nonlinearity_tsf	It is based on Teräsvirta’s nonlinearity test, which takes larger values when the time series is nonlinear.	nonlinearity from tsfeatures
distance	We created this feature that measures the average Euclidean distances between the feature vector of a target time series and the other 2000 feature vectors of the training dataset. The purpose was to measure the distance between a target time series and the training dataset. Higher values represent higher distances.

.

3.5. Linear Regression

To analyze when deep learning models perform worst or better than other models, we computed linear regression models that used as dependent variable the percentage change in the MAPE of each deep learning model with respect to the other models. It is percentage change = $\frac{MAPEdeeplearning-MAPEothermodel}{MAPEothermodel}.$ If the result is positive, the deep learning model generates an increment in the error, but if it is negative, the deep learning model reduces the error.

For example, to compare the performance of TCN versus ARIMA, the percentage change for each of the 2000 target time series is computed. The result of this calculation is the dependent variable of the model. On the other hand, the independent variables are the features of the time series pointed out in Table 2. In this way, the coefficients and the p-values of the regression model indicate if the time series features have a significant statistical influence on the percentage change (the difference between TCN and ARIMA) and to what extent the features influence the differences between TCN and ARIMA. The models were generated using ordinary least squares (OLS). We computed a robust standard error when the linear regression models violated the assumption of homoscedasticity.

4. Results

4.1. General Results

Figure 1 shows the average percentage change of the MAPE between models. Each bar in the figure shows the percentage change between a deep learning model and a statistical model or machine learning model. For example, the first bar indicates that the MAPE in the TCN models decreased almost 20% (on average) in relation to the XGBoost models.

The results suggest that the deep learning models surpass the machine learning but not the statistical models. The fact that statistical models have the best performance is contrary to the results of [9]; however, the methodology to evaluate the models is different for both studies. They evaluate the performance of the models in seven-fold instances without updating the model for each new instance. Each fold instance consists of 12 input past values ($X_{t-12},X_{t-11}\dots X_{t-1}$) and 12 output future values ($X_{t0},X_{t+1}\dots X_{t+11}$). However, in this paper, the evaluation was on three instances with a horizon of 12, but the biggest difference was that the model was updated for each fold instance. Therefore, this result suggests that, when the models are updated for a new prediction, the statistical models tend to give the best results. Finally, in the comparison between deep learning models, the best, in general terms, was TCN, although its performance was similar to LSTM.

Table 3. Coefficients of linear regression for comparison with statistical models.

Features	ARIMA			ETS			THETA
	TCN	LSTM	CNN	TCN	LSTM	CNN	TCN	LSTM	CNN
distance	0.19	0.27	0.49	0.11	0.16	0.44	0.02	0.01	0.18
size	−0.01	−0.05	0.04	0.03	0.00	0.11	0.03	0.02	0.08
trend	0.08	0.09	0.14	0.03	0.02	0.08	0.04	0.02	0.08
alpha_tsf	0.03	0.05	0.08	0.02	0.04	0.08	−0.02	0.00	0.00
kurtosis_fisher	−0.04	−0.05	−0.21	−0.07	−0.09	−0.23	0.01	0.03	−0.09
out	−0.07	−0.11	−0.17	−0.05	−0.07	−0.14	0.00	0.00	−0.06
pearson_test	−0.07	−0.09	−0.18	−0.04	−0.06	−0.16	0.00	0.01	−0.06
adf	−0.02	0.00	−0.03	0.00	0.02	0.00	−0.03	−0.01	−0.04
linearity	−0.01	−0.04	0.07	0.00	−0.02	0.07	−0.02	−0.02	0.04
seasonal_strength	0.03	0.00	0.16	0.01	−0.01	0.13	0.06	0.06	0.18
gamma	0.05	0.15	0.15	0.07	0.18	0.18	−0.02	0.04	0.00
white_noise	−0.10	−0.15	−0.26	−0.07	−0.09	−0.24	−0.01	−0.01	−0.10
nonlinearity	−0.05	−0.08	−0.14	−0.01	−0.02	−0.09	0.02	0.03	0.01

Note: Gray shadow coefficients mean p < 0.05.

and

Table 4. Coefficients of linear regression for comparison with machine learning models.

Features	XGBoost			SVM			RF
	TCN	LSTN	CNN	TCN	LSTN	CNN	TCN	LSTN	CNN
distance	−0.15	−0.12	−0.03	−0.14	−0.14	−0.13	−0.21	−0.21	−0.14
size	0.08	0.06	0.12	0.02	0.01	0.03	0.11	0.10	0.17
trend	−0.19	−0.23	−0.22	−0.28	−0.29	−0.30	−0.23	−0.27	−0.28
alpha_tsf	0.04	0.07	0.07	0.03	0.04	0.03	0.04	0.08	0.09
kurtosis_fisher	0.08	0.07	0.01	0.09	0.10	0.08	0.10	0.11	0.05
out	0.04	0.03	0.00	0.05	0.05	0.04	0.06	0.06	0.03
pearson_test	0.04	0.03	0.00	0.06	0.06	0.06	0.06	0.06	0.03
adf	0.02	0.04	0.04	0.01	0.01	0.02	0.02	0.03	0.03
linearity	−0.02	−0.02	0.00	−0.03	−0.03	−0.03	0.01	0.00	0.03
seasonal_strength	0.05	0.05	0.12	−0.02	−0.02	0.00	0.04	0.04	0.11
gamma	0.12	0.22	0.23	0.03	0.04	0.04	0.10	0.18	0.18
white_noise	0.08	0.07	0.02	0.07	0.07	0.06	0.12	0.12	0.09
nonlinearity	0.12	0.13	0.11	0.05	0.06	0.05	0.15	0.17	0.15

Note: Gray shadow coefficients mean p < 0.05.

show the coefficients of the multiple linear regression models. Each coefficient shows how the MAPE of the deep learning model changed when the features increased by one unit. The features were standardized, which means that the change of one unit corresponded to one standard deviation. For example, the trend coefficient in Table 3 indicates that an increment of one standard deviation in the time series trend generates an increase of 8.0 percent in the MAPE of TCN in relation to XGBoost MAPE. Therefore, XGBoost tends to work better than TCN as the strength of the trend increases. It is relevant to mention that the coefficients shaded in gray are significant using p = 0.05.

4.2. Comparison between Deep Learning Models and Statistical Models

The distance (Euclidean distance) between the target time series and the training dataset was the most influential feature when we compared deep learning models to ARIMA and ETS models. For example, in Table 3, we can see that the MAPE of TCN and LSTM increase by 19% and 27% in relation to the ARIMA MAPE for each increment of one standard deviation in the distance, respectively. This result is reasonable, because the deep learning models could generate the worst results when a record was different from those used to create the model [18].

Two significant variables were the outliers and normality. Increasing the number of outliers and closeness to the normal distribution of the time series reduced the MAPE regarding the ARIMA and ETS models. On the other hand, the increment in the trend strength increased the MAPE in relation to the ARIMA MAPE, and the increment of gamma increased the MAPE in relation to the ETS MAPE. The comparisons between TCN and LSTM with the THETA models showed less significant variables. The more relevant result was that the seasonal strength favored the THETA model.

4.3. Comparison between Deep Learning Models and Machine Learning Models

We also found a result associated with the distance that is more difficult to explain. The finding suggests that the increase in distance favors deep learning models when compared against machine learning models (Table 4). This result occurred in five comparisons.

The most important factor that affects the performance of deep learning models is the trend. When we compare the performance with any of the machine learning models, the strength of the trend reduces the error. On the other hand, the increment in nonlinearity, seasonal strength, and gamma increases the error of the deep learning models versus XGBoost and Random Forest.

5. Conclusions

The machine learning models showed a lower accuracy performance than the pretrained deep learning models but could be more robust, with less predictable time series. For example, variables like nonlinearity and seasonal_strength are associated with the best performance of XGBoost and Random Forest; meanwhile, the strength of the trend is associated negatively with the three types of machine learning models executed. Deep learning models could work better with less predictable time series than statistical models. For example, variables like the strength of the trend and seasonal strength are associated positively with the performance of statistical models, and the presence of outliers is associated negatively.

Some transformations could improve the performance of pretrained models. The normal distribution test of the time series is associated positively with deep learning models; therefore, any transformation that converts the time series to a normal distribution could improve the models.

This study offers findings that can help explain better why a pretrained deep learning model is better than another kind of model. Future studies should look at new variables that can help explain the differences between models. Additionally, new models can be incorporated as transformers, hybrid models, etc. Another question to be answered is whether the model can be improved by excluding older time points if it changes the properties of the time series. The findings of these kinds of studies not only contribute to explaining when one model works better than another but can also provide evidence about possible transformations that improve the performance of deep learning models.