Applied Intetigence https//dotorg/101007/10480.020-01839.5, Check for A multi-layer and multi-ensemble stock trader using deep learning _ “Pts and deep reinforcement learning Salvatore Carta’ - Andrea Corriga’ - Anselmo Ferreira" ® - Alessandro Sebastian Podda’ - Diego Reforgiato Recupero! (© Springer Sciences Business Media, LIC, part of Springer Nature 2020 Abstract ‘The adoption of computer-aided stock trading methods is gaining popularity in recent years, mainly because of their ability {o process efficiently past information through machine learning to predict future market behavior. Several approaches have been proposed to this task, with the most effective ones using fusion ofa pile of classifiers decisions to predict future stock values. However, using prices information in single supervised classifiers has proven to lead to poor results, mainly because ‘market history is not enough to be an indicative of future market behavior. In this paper, We propose to tackle this issue by proposing a multi-layer and multi-ensemble stock trader. Our method starts by pre-processing data with hundreds of deep neural networks. Then, a reward-based classifier acts as a meta-learner to maximize profit and generate stock signals through different iterations. Finally, several metalearner trading decisions are fused in order to got a more cobust trading strategy. using several trading agents to take a final decision. We validate the effectiveness of the approach in a real-world trading scenario, by extensively testing it on the Standard & Poor's 500 future market and the JL. Morgan and Microsoft stocks. Experimental results show that the proposed method clearly outperforms all the considered baselines (which still performs very well in the analysed period), and even the conventional Buy-and-Hold strategy. which replicates the market behaviour. Keywords Deep learning - Deep reinforcement learning - Intraday stock trading 1 Introduction ‘The growing popularization of stock market trading has become a rich field to be explored by powerful and fast computing techniques. These tools have been widely explored, to provide better trading strategies and to maximize performance metries such as, among others, profit, economic utility, or risk-adjusted return. Indeed, ‘machine learning solutions have been proposed to stock trading even before the popularization of computers [15, 20, 21] and have potential to distribute wealth among investors with a reduced need for human intervention, “The research performod inthis paper bas been supported by the “Hlando “Aiuth per progstt di Ricerca e Sviluppo"-POR FESR 2014-2020—Aste 1, Azione 1.1.3, Strategy 2- Program 3, Project ‘AlmostAnOracle~ Aland Big Data Algorithms fr Financial Time Series Forecasting” 2 Salvatore Carta salvatore@unicai Extended author information available om the last page ofthe article, Published online: 07 September 2020 Financial forecasting techniques are usually divided into the branches of Technical Analysis (TA), which uses data acquired from the past to indicate future behavior of the market, and the Fundamentalist Analysis (FA), Which analyzes economic metrics that may influence market future trends. Machine-learning based TA approaches treat stock market forecasting as a classification problem, where classifiers use labeled time series data from historical prices jn order to assign the following trading signals: a long ‘operation (ie., betting that the price will rise in the future) ‘or a short operation (Ze., to bet the opposite) should be done in order to maximize the profit. Some solutions in machine learning-based TA have been widely proposed in the financial forecasting literature using, for example, deep learning [24, 34], Support Vector Machines [7, 45], Random Forests [18, 37], reinforcement learning [16, 23}, among others. Although not the focus of this paper, both TA and FA ccan be used together to perform financial trading [8, 17] ‘The computer-aided stock trading is basically composed of two steps: (3) analysis of past market behavior, and Gi) taking the optimal stock trading decision. To perform such tasks, time-series data from past prices are usually D springerS.cartaetal considered as input. These data are usually given by the market under different resolutions (minutes, hours, days, ete) and contain information such as open prices, close prices, among others. However, such data contains 4 lot of uncertainty, specially because stock prices are often influenced by other factors such as political events, epidemics, performance of other markets and even the behavior of other investors [4], These aspects model stock ‘markets as dynamic, non-linear, non-parametrie and chaotic environments [36]. In order to minimize such issues in machine learning, based trading, several solutions have proposed the fusion of dozens of machine learning classifiers with the aim to get more computer-aided stock traders involved in the trading decision, which in tum minimizes the error of single classifiers trained on insufficient data. These algorithms luckle the no-free lunch theorem [41] by fusing several ‘weak classifiers decisions with low correlation in order 10 yield a strong final classifier. Inthe stock market forecasting, Scenario, several literature approaches have been explored in that sense, such as using boosting [42], late fusion BI, dynamic ensembling [6], stacking [10], heterogeneous, classifiers [28] among many others. Notwithstanding, such supervised approaches still do not deal efficiently with raw past prices data and their very known limitations. Therefore, an efficient solution should be designed to better cope with the uncertainty and, atthe same time, maximize ensembling, steps in such a way to perform profitable and safer trading. In this paper, We move towards tackling these issues. by maximizing the performance of an ensemble based on Reinforcement Leaming. Our stock trader is composed of 4 multi-layer and multi-ensemble classifier that acts in three steps and use two ensembling techniques at the same time. Our approach works by firstly applying several neural networks in stock prices data in the first layer, and the output will be used in a stacking ensembling technique in the second layer. Then, the set of trading signals from the neural networks are used as Features and a reinforcement learning agent acts as a meta learner, maximizing rewards, (profits) and yielding new trading signals through different iterations. Finally, in the last layer of our trader, these new signals are fused through another ensembling technique, namely a majority voting (late fusion) approach, in order to generate more robust final decisions. Simulations of intraday trading on the Standard & Poor's 500 future market and the J.P. Morgan and Microsoft stock markets show that the proposed approach clearly outperforms not only the conventional Buy-and-Hold strategy (and, consequently, the market behaviour), but also several non-RL ensembling. techniques and literature methods, which provide strong, performance in the considered period. D springer In summary, the contributions of this paper are: 1, We propose a machine-tearning based stock trading system composed by three different layers, that leverages both on deep learning and deep reinforcement learning benefits, which were proven to be effective in several other machine learning fields [2, 43]; 2. We exploit a novel pre-processing step based on generating meta-features after converting price time- series into Gramian Angular Field images [38], in order to generate input data to be further processed in our stacking ensembling approach; 3. We propose the use of a deep reinforcement leaning meta-learner, which processes the trading signals from Convolutional Neural Networks (CNNs) to generate the final trading decisions: 4. We combine two different ensembling steps (stacking in the second layer of our agent, and majority voting in the last layer) to maximize the robustness and stability of our trading strategy: We evaluate our multilayer and multi-ensemble approach in a real-world trading scenario, comparing ‘our experimental results with several baselines and Iit- erature competitors, showing that it clearly outperforms their performance, ‘The rest of this paper is organized as follows: in Section 2, we discuss related work in. stock trading using machine learning solutions. Section 3 motivates and present our algorithm in details, Section 4 describes the experimental setup considered to validate the proposed approach, Section 5 shows experimental results and, finally, ‘Section 6 concludes this Work and discusses possible future research directions motivated by the proposed method. 2 Background and related works Several works in literature have been vastly proposed in ‘order to perform automated stock trading. One of the pioneer works in this aspect is the work of Kimoto et al [20], who used modular neural networks trained on various technical and economical information in order to determine when (0 buy and sell stocks. This work was extended Fater in [15] using recurrent neural networks that are more Suitable to time series. The work of Lee and Park [21] used a similar recurrent neural network, but trained on eleven economic indicators, Other subsequent works have used more advanced machine learning classifiers to perform stock trading. Sun et al. [34] used Long Short Term ‘Memories deep neural networks, an evolution of recurrent networks, on augmented market trading data, The work of