A Monte Carlo strategy for structure multiple-step-head time series prediction
•
1 like•548 views
The document proposes a Monte Carlo approach called SMC (Structured Monte Carlo) for multiple-step-ahead time series forecasting that takes into account the structural dependencies between predictions. It generates samples using a direct forecasting approach and weights them based on how well they satisfy dependencies identified by an iterated approach. Experiments on three benchmark datasets show the SMC approach achieves more accurate forecasts as measured by SMAPE than iterated, direct, or other comparison methods for most prediction horizons tested.
![Forecasting in probabilistic terms SMC approach Experiments References
Multiple-step-ahead forecasting
Forecasting the continuation of a time series multiple steps
forward is a relevant and challenging problem in data mining
and computational intelligence.
The complexity of this problem is due to several aspects: the
potential nonlinearity of the dependency between the past and
the future, the lack of a priori knowledge, the large noise and
the small amount of samples.
Three strategies are commonly used to tackle such task:
1 Iterated (recursive) strategy which iterates a one-step-ahead
predictor.
2 Direct strategy which decomposes the prediction in a set of
independent prediction tasks [7].
3 Multi-response regression strategies like the Joint method [6]
or the Lazy Learning MIMO [3, 2].](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
What's hot
What's hot (19)
Viewers also liked
Viewers also liked (20)
Similar to A Monte Carlo strategy for structure multiple-step-head time series prediction
Similar to A Monte Carlo strategy for structure multiple-step-head time series prediction (20)
More from Gianluca Bontempi
More from Gianluca Bontempi (8)
Recently uploaded
Recently uploaded (20)
A Monte Carlo strategy for structure multiple-step-head time series prediction
- 1. Forecasting in probabilistic terms SMC approach Experiments References A Monte Carlo strategy for structured multiple-step-ahead time series prediction Gianluca Bontempi Machine Learning Group, Interuniversity Institute of Bioinformatics in Brussels (IB)2 ULB, Université Libre de Bruxelles Boulevard de Triomphe - CP 212 Bruxelles, Belgium mlg.ulb.ac.be, ibsquare.be
- 2. Forecasting in probabilistic terms SMC approach Experiments References Multiple-step-ahead forecasting Forecasting the continuation of a time series multiple steps forward is a relevant and challenging problem in data mining and computational intelligence. The complexity of this problem is due to several aspects: the potential nonlinearity of the dependency between the past and the future, the lack of a priori knowledge, the large noise and the small amount of samples. Three strategies are commonly used to tackle such task: 1 Iterated (recursive) strategy which iterates a one-step-ahead predictor. 2 Direct strategy which decomposes the prediction in a set of independent prediction tasks [7]. 3 Multi-response regression strategies like the Joint method [6] or the Lazy Learning MIMO [3, 2].
- 3. Forecasting in probabilistic terms SMC approach Experiments References Bet: which continuation? A B t y
- 4. Forecasting in probabilistic terms SMC approach Experiments References Our contribution We propose an algorithm for multiple-step-ahead forecasting which takes into consideration the dependency between multiple predictions. The rationale is to use Monte Carlo importance sampling to sample the conditional distribution of the multivariate vector representing the continuation of the time series in a a way that takes into account the structural dependency of the series. The result is a multi-step-ahead forecasting method which combines in a probabilistic way the Iterated approach and the Direct approach. Combination is due to the fact that the outcome of the Monte Carlo importance sampling strategy reweighs the set of bootstrap predictions obtained by the Direct method by taking into consideration the constraint represented by the one-step-ahead predictor.
- 5. Forecasting in probabilistic terms SMC approach Experiments References Multiple-step-ahead forecasting in probabilistic terms A time series is the realization of a stochastic process, that is a sequence of random variables indexed by a variable t. A stochastic process is completely determined by the joint distribution of all the variables {. . . , y1, y2, . . . , yt+1, yt+2, . . . } This distribution summarizes all the dependencies between the past and the future values of the series. Forecasting at time t the next h > 0 values of the time series is then possible since the observed data {y1, . . . , yt} can provide information about the stochastic dependencies between the past and the future realizations and these dependencies are preserved with time.
- 6. Forecasting in probabilistic terms SMC approach Experiments References The NAR representation The autoregressive formalism represents the dependency by yt+1 = F(yt, yt−1, . . . , yt−p+1) + w = F(q) + w where p is the order of the model and the vector q of length p is commonly denoted as the embedding vector. The expected value of yt+1 is E[yt+1|yt, yt−1, . . . , ] = E[yt+1|yt, . . . , yt−p+1] = = F(yt , yt−1, . . . , yt−p+1) w denotes the conditional distribution of yt+1 − E[yt+1|yt, . . . , yt−p+1].
- 7. Forecasting in probabilistic terms SMC approach Experiments References Iterated vs Direct In the Iterated approach the data are used to infer the one-step-ahead dependency yt+1 = F1(yt, yt−1, . . . , yt−p+1) + w1 and the estimated model ˆF1 is used iteratively to provide the set of H predicted values ˆyt+1, ˆyt+2, . . . , ˆyt+H . In the Direct approach the multiple-step-ahead forecasting task is decomposed in a set of H independent single-output tasks yt+h = Fh(yt, yt−1, . . . , yt−p+1) + wh, h = 1, . . . , H. These approaches ignore the structured property, i.e. the H future values to be predicted are conditionally dependent.
- 8. Forecasting in probabilistic terms SMC approach Experiments References Structural dependencies The prediction H steps forward demands the estimation of H conditional expectation terms E[yt+h|q] = yt+hp(yt+h|q)dyt+h, h = 1, . . . , H where the H variables are not independent and distributed according to the conditional and multivariate distribution p(yt+H , . . . , yt+1|yt, yt−1, . . . , yt−p+1) = = p(yt+H , . . . , yt+1|q) (1) Structural constraint among the H variables: for each h = 2, . . . , H and j = 1, . . . , h − 1 p(yt+h|q) = p(yt+h|yt+j , . . . , yt+1, q)p(yt+j , . . . , yt+1|q)dyt+j . . . dyt+1
- 9. Forecasting in probabilistic terms SMC approach Experiments References Structural dependency in NAR(2) for H = 3 p(yt+3|yt, yt−1) = p(yt+3|yt+2, yt+1)p(yt+2|yt+1, yt )p(yt+1|yt, yt−1)dyt+2 . . . dyt+1, This expression shows the nature of the dependency between yt+3, yt+2 and yt+1. yt yt+1 yt+2 yt-1 yt+3
- 10. Forecasting in probabilistic terms SMC approach Experiments References Structural dependencies In the Direct approach the estimation of the H next values is done without taking into account the structural constraint: the Direct approach makes an hypothesis of conditional independence. The Iterated approach approximates the structural constraint for j = h − 1 by assuming naively that the predictions ˆyt+1, . . . , ˆyt+h−1 return an accurate estimation of the distribution of yt+1, . . . , yt+h−1 for each h = 1, . . . , H. Our paper proposes to take explicitly into consideration the structural constraint by adopting a Monte Carlo sampling strategy. This strategy allows the integration of the Direct and Iterated strategy for the sake of accuracy.
- 11. Forecasting in probabilistic terms SMC approach Experiments References Monte Carlo approach If we are able to generate R samples y (r) t+h according to the conditional distribution p(yt+h|q), the estimation of the H predictions would be easy: E[yt+h|q] ≈ ˆyt+h = 1 R R r=1 y (r) t+h Unfortunately the distribution p(yt+h|q) is complex and unknown and we cannot generate samples. A preliminary estimator is provided by using the Direct strategy. Though such estimator disregards some aspects of the conditional distribution, like the structural constraint, we could try to adjust its estimation accordingly in order to take into account the missing information.
- 12. Forecasting in probabilistic terms SMC approach Experiments References Importance sampling The idea of adjusting samples drawn from a proposal distribution in order to obtain samples from a target distribution, potentially known but impossible to be sampled directly, is the core of the importance sampling approach. We propose an importance sampling strategy to adjust the Direct approach to incorporate the structural constraint. We generate approximate samples by the Direct approach (that plays here the role of proposal distribution generator) and adjust them by weighting according to their satisfaction of the structural constraint (implemented with the Iterated approach). Inspired by the particle filter algorithm for state estimation.
- 13. Forecasting in probabilistic terms SMC approach Experiments References SMC (Structured Monte Carlo) algorithm 1 we draw R samples ˆy (r) t+h by sampling the conditional distribution p(yt+h|q) h = 1, . . . , H with the Direct approach. 2 we loop over an increasing horizon h = 2, . . . , H. For each h each sample ˆy (r) t+h is weighted by a term measuring how much this value is compliant with the structural constraint by w (r) t+h = J j=1 p(ˆy (r) t+h|ˆy (j) t+h−1, . . . , ˆy (j) t+h−p) where J is a parameter setting the number of embedding vectors taken into consideration. 3 we assemble the Direct samples with the importance weights ˆyt+h = R r=1 w (r) t+hˆy (r) t+h R r=1 w (r) t+h , h = 1, . . . , H
- 14. Forecasting in probabilistic terms SMC approach Experiments References Experiments We considered three benchmarks the 111 time series of the NN5 Competition (complete dataset) [1] measuring the daily retirement amounts from independent cash machines at different, randomly selected locations across England. We adopt five prediction horizons H = 50, 70, 90, 100, 200. the NN3 Dataset [5] is made of 111 monthly econometric time series starting at January, with a variable number of points (from 50 to 126). We consider H = 10 and H = 18. a set of 1080 series obtained by simulating 90 times (different random seeds and increasing noise variances) 12 nonlinear autoregressive models.
- 15. Forecasting in probabilistic terms SMC approach Experiments References Experiments We compared the SMC algorithm to an Iterated (IT) algorithm, a Direct (DIR) algorithm, an Averaged (AVG) version of the Direct Algorithm, a LL-MIMO algorithm [3], a linear AR and a Random Walk estimator. The SMC, IT, DIR and AVG algorithms rely on a locally constant estimator with an adaptive number of neighbors ranging between 3 and 15 and selected on the basis of the PRESS leave-one-statistic [4]. The MIMO algorithm implements the multiple output algorithm proposed in [3] with a number of neighbors ranging between 3 and 15. The AVG algorithm returns the average of R = 100 DIR estimations obtained by resampling each time two thirds of the training set. The AR forecast is implemented by the R forecast package.
- 16. Forecasting in probabilistic terms SMC approach Experiments References SMAPE results Dataset SMC IT DIR AVG MIMO AR RW NN5 (H = 50) 4.24 4.27 4.38 4.29 4.45 5.98 10.98 NN5 (H = 70) 4.52 4.67 4.60 4.55 4.67 6.38 8.67 NN5 (H = 90) 6.04 7.02 6.07 6.09 6.19 11.83 13.22 NN5 (H = 100) 5.91 6.1 5.96 5.93 6.08 8.53 12.83 NN5 (H = 200) 13.31 14.25 13.6 13.4 13.6 19.22 20.68 NN3 (H = 10) 3.18 3.28 3.25 3.20 3.32 3.00 4.39 NN3 (H = 18) 8.70 8.87 8.89 8.72 8.94 8.23 11.46 NAR (H = 10) 1.46 1.54 1.49 1.46 1.48 1.65 2.08 SMAPE = 1 H HX h=1 |ˆy − yh| (ˆyh + yh)/2 × 100 The lower the SMAPE, the more accurate the prediction. Bold notation is used to denote methods significantly different from SMC.
- 17. Forecasting in probabilistic terms SMC approach Experiments References Number of SMC Win-Losses Dataset IT DIR AVG MIMO AR RW NN5 (H = 50) 59/52 98/13 95/16 92/19 107/4 111/0 NN5 (H = 70) 54/57 78/33 91/20 84/27 105/6 108/3 NN5 (H = 90) 79/32 64/47 82/29 91/20 109/2 111/0 NN5 (H = 100) 67/44 79/32 83/28 90/21 107/4 108/3 NN5 (H = 200) 67/44 99/12 90/21 85/26 107/4 107/4 NN3 (H = 10) 62/49 72/39 61/50 54/57 43/68 65/46 NN3 (H = 18) 65/46 72/39 65/46 67/44 52/59 75/36 NAR (H = 10) 658/422 680/400 646/434 583/497 630/147 731/349 The notation W/L means that SMC has a better SMAPE than the considered technique W out of (W+L) times .
- 18. Forecasting in probabilistic terms SMC approach Experiments References Conclusion Direct strategies are able to provide good approximation of the marginal distribution of future values of the time series, on the other side Iterated methods are more effective in modeling the conditional dependency between forecasts. We aim to preserve the best of both techniques by having recourse to an importance sampling paradigm, inspired to particle filters. SMC is a robust and effective integration of Iterated and Direct strategies Future work will extend SMC to spatio-temporal and vector autoregressive forecasting.
- 19. Forecasting in probabilistic terms SMC approach Experiments References [1] Robert R. Andrawis, Amir F. Atiya, and Hisham El-Shishiny. Forecast combinations of computational intelligence and linear models for the NN5 time series forecasting competition. International Journal of Forecasting, January 2011. [2] S. Ben Taieb, G. Bontempi, A.F. Atiya, and A. Sorjamaa. A review and comparison of strategies for multi-step ahead time series forecasting based on the nn5 forecasting competition. Expert Systems with Applications, 2012. [3] G. Bontempi and S. Ben Taieb. Conditionally dependent strategies for multiple-step-ahead prediction in local learning. International Journal of Forecasting, 2011. [4] G. Bontempi, M. Birattari, and H. Bersini. A model selection approach for local learning. Artificial Intelligence Communications, 121(1), 2000. [5] Sven F. Crone, Michèle Hibon, and Konstantinos Nikolopoulos. Advances in forecasting with neural networks? empirical evidence from the nn3 competition on time series prediction. International Journal of Forecasting, 27, 2011. [6] D. M. Kline. Methods for Multi-Step Time Series Forecasting Neural Networks. IGI Global, 2004. [7] Antti Sorjamaa, Jin Hao, Nima Reyhani, Yongnan Ji, and Amaury Lendasse. Methodology for long-term prediction of time series. Neurocompuing, 70(16-18):2861–2869, 2007.