Algorithmic Trading using Sentiment Analysis and Reinforcement Learning

Simerjot Kaur (SUNetID: sk3391 and TeamID: 035)

Abstract the average of the profit/returns generated at time t normalized
This work presents a novel algorithmic trading system based on over the standard deviation of the profit/returns generated at time
reinforcement learning. We formulate the trading problem as a t. Thus for this trading system, the performance criterion will be
Markov Decision Process (MDP). The formulated MDP is solved based on Sharpe Ratio. The oracle [1] in terms of Sharpe Ratio is
using Q-Learning. To improve the performance of Q-Learning, that usually, any Sharpe ratio greater than 1 is considered
we augment MDP states with an estimate of current market/asset acceptable to good by investors. A ratio higher than 2 is rated as
trend information and sentiment analysis of news articles of each very good, and a ratio of 3 or higher is considered excellent.
asset which are estimated using Neural Networks. Our proposed In our first approach, described in Section 4, we formulated the
algorithm is able to achieve a Sharpe Ratio of 2.4 when we invest MDP consisting of historical price, number of stocks, cash in
$10,000 as compared to a baseline Sharpe Ratio of -0.2 and an hand as states and solved using reinforcement learning. In section
estimated oracle Sharpe Ratio of 3.0. 5, we extend our approach by augmenting the states with current
1. Introduction market/industry trend information and then solve the MDP using
reinforcement learning. In Section 6, we further extend our
Algorithmic trading also called as automated trading is the
approach by augmenting the states with Sentiment Analysis
process of using computers programmed to follow a defined set
performed on news articles. Section 7 provides a summary and
of instructions for placing a trade in order to generate profits at a
comparison of our results. Section 8 further identifies the current
speed and frequency that is impossible for a human trader. An
limitations of our model. Finally, section 9 summarizes and
important factor affecting the trading decisions is the ability of
concludes this paper and lays foundation for our future work.
predicting stock market movement. The prediction of stock
markets movement is considered to be a challenging task of 2. Literature Review
financial time series prediction due to the complexity of the stock Various different techniques have been implemented in literature
market with its noisy and volatile environment, considering the to train ML agents to do automated trading using machine
strong connection to numerous stochastic factors such as political learning techniques. For instance, [Pierpaolo G. Necchi,
events, newspapers as well as quarterly and annual reports. 2016][2], [David W. Lu, 2017][3], [Xin Du, Jinjian Zhai, Koupin
In this project, I want to explore the possibility of building a Lv, 2009][4], [Jae Won Lee, Jangmin O][5] all describe different
machine learning agent that tries to learn an optimal trading machine learning techniques like deep q-learning, recurrent
policy/strategy using several machine learning techniques like reinforcement learning, etc to perform algorithmic trading.
reinforcement learning. The problem we are trying to solve in [James Cumming, 2015][6] also wrote a book on the use of
this project can be summarized as: “Train a ML Agent to learn different reinforcement learning techniques within the
an optimal trading strategy based on historical data and stock Algorithmic Trading Domain. The major strength of these
market news in order to maximize the generated profits.” researches is that they are trying to investigate the best possible
In particular, the project would involve the following sub-steps, learning algorithm so that automated trading can be performed
as shown in Figure 1 below: with minimum human intervention.
1. Formulate the trading problem as a Markov Decision Process On the other hand, a large number of implementations have been
(MDP) which is then solved using Q-learning with done to perform sentiment analysis on news articles and twitter
functional approximations. The states of the MDP not only feeds. The paper of [Gyozo Gidofalvi, 2011][7] predicts the
involve historical price, number of stocks, cash in hand but short-term stock price movements using financial news articles.
also trend information (step 2) and sentiment score (step 3). The paper of [Naveed Ahmad, Aram Zinzalian, 2010][8]
2. Perform Trend Analysis using technical market indicators as explores stock volatility forecasting from quarterly earnings call
input to neural networks and obtain stock market trend as the transcripts of the 30 DOW component stocks. The paper of
output signal. [Qicheng Ma, 2008][9] presents a scheme in which 29 stock
3. Perform Sentiment Analysis on news articles of each asset. names are picked from DJI and related articles and are classified
Rather than maximizing profits, most modern fund managers as positive or negative using Naive Bayes and Maximum Entropy
attempt to maximize risk-adjusted return as advocated by Modern classifier. The paper [Ravi Parikh, Matin Movassate, 2009][10]
Portfolio Theory. The Sharpe Ratio is the most widely-used uses Unigram and Multi-nomial Bigram Naive Bayes to detect
measure of risk-adjusted return (Sharpe) and can be calculated as sentiment. The paper of [Gabriel Fung, Jeffrey Yu, Hongjun Lu,

Figure 1: Proposed Approach

1
2005][11] predicts the movements of stock prices based on the Rewards: The reward at any point is calculated as the
contents of the news stories. All these researches are trying to [Current Value of the Portfolio - Initial Investment].
classify the news more accurately, thereby trying to obtain Discount Factor: In this project, the discount factor is assumed
significant accuracy. to be 1 always.
In this project we follow a novel approach of combining both the Solving the MDP:
implementations where we gather information from various
The MDP described above was solved using vanilla Q-Learning
sources that have big impact on the stock market and supplying
algorithm with functional approximations[17].
this additional information to better learn an optimal
policy/strategy. Thus the ML agent not only just learns an optimal The algorithm can be described as: 

trading strategy based on historical prices but also on additional On each (s, a, r, 𝑠ʹ):
information based on sentiment and trend of the market to make 𝑄$%& 𝑠, 𝑎 ← 1 − 𝜂 𝑄$%& 𝑠, 𝑎 + 𝜂(𝑟 + 𝛾𝑉$%& 𝑠′ )
an informed decision.
where: 𝑉$%& 𝑠′ = 𝑚𝑎𝑥 𝑄$%& 𝑠′, 𝑎′
89∈;<&=$>?(? @ )
3. Dataset Used s = current state, a = action being taken, 𝑠′ = next state, 𝛾 = discount
5-years of daily historical closing prices (2011-2016) for various factor, r = reward, 𝜂 = exploration probability.
stocks were obtained from Yahoo Finance[12] to form the As Q-learning doesn’t generalize to unseen states/actions,
dataset. For this project, the results are restricted to a portfolio function approximation has been used which parameterizes 𝑄$%&
consisting of 2 stocks [Qualcomm (QCOM), Microsoft (MSFT)]. by a weight vector and feature vector which can be described as:
10-years of news articles have also been obtained from Reuters On each (s, a, r, 𝑠ʹ):
Key Development Corpus[13] for both Qualcomm and 𝑤 ← 𝑤 − 𝜂 𝑄$%& 𝑠, 𝑎; 𝑤 − 𝑟 + 𝛾𝑉$%& 𝑠′ 𝜙(𝑠, 𝑎)
Microsoft. 5-years of these news articles (2006-2010) are used as
training dataset and the remaining (2011-2016) are used as test where: 𝑄$%& 𝑠, 𝑎; 𝑤 = prediction and 𝑟 + 𝛾𝑉$%& 𝑠′ = target
dataset to perform Sentiment Analysis. The headlines were For our problem, we used the following feature vectors: (a)
manually classified to obtain the ground truth sentiment score: +1 Number of Stocks of each asset, (b) Current Stock Price of each
if the news provides a positive sentiment and -1 if the news asset and, (c) Cash in Hand.
provides a negative sentiment.
Also, in order to deploy the tradeoff between exploration and
4. Problem Formulation exploitation, epsilon-greedy algorithm has been used which
This section describes how the problem of choosing when to explores with probability ε and exploits with probability 1−ε. An
BUY/HOLD/SELL a portfolio of stocks is formulated as a exploration probability of 0.2 has been chosen for this project.
Markov Decision Process (MDP). The section further elaborates Results:
on how the MDP is solved by learning an optimal policy using This section discusses the performance of the above implemented
Q-learning with functional approximations. It may be noted that Q-learning system. The Q-learning system was run on a dataset
since the state space with real valued stock prices is really huge, containing 5-year stock prices and number of trials as 10,000.
the state space has been discretized for the stock prices.
The plot below shows how the Sharpe Ratio evolves with
MDP Formulation: increasing historical time period. As can be observed, the ML
MDP[16] can be formulated by describing its States, Actions, agent slowly but steadily learns an optimal strategy that increases
Transition Probability, Rewards, and discount factor. the Sharpe Ratio as more historical data is provided, thus
States: [(#of Stocks for each asset), (Current Stock Price for each displaying incremental performance and is successful in
asset), Cash in Hand] achieving the Sharpe Ratio of 0.85.
The first part of the state consists of a tuple containing number of
stocks for each asset. The second part of the state consists of a
tuple containing the daily closing stock price for each asset.
Finally, the third part of the state consists of cash in hand which
is evaluated at every time step based on the action performed.
Initial State: [(0, 0...), (𝑆1, 𝑆2...), $10,000] i.e. the agent has 0
stocks for each asset and only $10,000 as an initial investment.
Actions: At any point the agent chooses from three actions: BUY,
SELL, and HOLD. Action BUY buys as many stocks for each
asset as possible based on the current stock price and cash in
hand. Action SELL sells all the stocks in portfolio and adds the Figure 2: Sharpe Ratio vs historical time period using Q-Learning
generated cash to cash in hand. Action HOLD, does nothing, i.e. To validate that the algorithm was effective in learning the
neither sells nor buys any stock of an asset. optimal strategy, Monte-Carlo simulation has been performed as
Transition Probability: The transition probability is chosen to baseline. In this simulation, the agent is forced to choose an
be 1 always as whenever the action is BUY/SELL we are sure to action at random. As can be observed in the plot below, the ML
BUY/SELL the stocks of an asset. Here the randomness in the agent generates negative Sharpe Ratio, hence validating the Q-
system comes from the fact that the stock price changes just after learning algorithm for this problem.
a stock is bought or sold i.e. after every time step.
2
 4. Stochastic D
𝐾% 𝑖 − 2 + 𝐾% 𝑖 − 1 + 𝐾% 𝑖
𝐷% 𝑖 =
3

5. Relative Strength Index

100
𝑅𝑆𝐼 = 100 − ( )
1 + 𝑅𝑆
𝐴𝑣𝑒𝑟𝑎𝑔𝑒 𝑜𝑓 𝑡 𝑑𝑎𝑦 9 𝑠 𝑢𝑝 𝑐𝑙𝑜𝑠𝑒𝑠
𝑤ℎ𝑒𝑟𝑒 𝑅𝑆 =
𝐴𝑣𝑒𝑟𝑎𝑔𝑒 𝑜𝑓 𝑡 𝑑𝑎𝑦𝑠 9 𝑠𝑑𝑜𝑤𝑛 𝑐𝑙𝑜𝑠𝑒𝑠
6. Larry William’s R%
𝐻& − 𝑐𝑝 𝑖
𝑅% 𝑖 = ∗ 100
𝐻& − 𝐿&
Figure 3: Sharpe Ratio vs historical time period using MCS
Where cp(i) is the closing price, 𝐿& is the lowest price of last t
5. Problem Formulation Augmented with Trend Analysis
days, 𝐻& is the highest price of last t days. Also, the six technical
When a human trader makes a trading decision, in addition to the indicator values represent continuous values in different ranges.
stock price, his decision is also based on his view of the So the input data is scaled in the range 0–1 using the min max
market/industry’s trend so that he is able to profit from the normalization as follows:
upward movement and avoid the downfall of a stock. In this 𝑥 − 𝑥f=>
section, we are trying to capture the market’s current trend and 𝑦 =
𝑥f8g − 𝑥f=>
use this trend information as an additional feature in the state Where y = normalized value, x = value , 𝑥f=> = minimum value
definition of our MDP formulation. The practical implementation and 𝑥f8g = maximum value of the series to be normalized.
of the above approach entails two steps: (A) Finding the current
The scaled technical indicators are then inputted into the neural
trend information (B) Augmenting the trend information into
network. An inbuilt neural network toolbox in python, HP-
MDP formulation.
ELM[14], has been used in this project to get the output trading
5.A Using Market Technical Indicators and Neural Network signal (OTS) in range 0-1. The uptrend and down trend are
to find Trend Information classified using the following methodology:
To identify the trend information, six popular market technical If OTS > Mean (OTS), predicted trend is up (1) else down (0).
indicators[15] calculated from the historical stock price are used
as the input features for the neural network. The output of this The chart below shows historical price (blue) and predicted trend
neural network is used to perform the classification into two states(orange) (~2 years) by the neural network for test data
states, 1 if an upward trend is observed and 0 if downward trend (QCOM stock).
is observed. 80 1.2
1
60
0.8
40 0.6
0.4
20
0.2
0 0
1

129
161
193
225
257
289
321
353
385
417
449
481
33
65
97

Figure 5: Trend States vs Historical Price for QCOM stock

5.B Augmenting the trend information into MDP formulation

The above section described how we can identify current trend in
stock data using neural networks. To provide this information to
Figure 4: Trend Analysis performed using neural networks our ML agent, we augmented the state of the system with this
The six technical indicators that are used as input for the neural information and the new state implemented was [(#of Stocks for
network can be calculated as shown below: each asset), (Current Stock Price for each asset), Cash in Hand,
1. Simple Moving Average: t=15 (trend of each asset)]. All other aspects of the MDP definition are
1 & same as before.
𝑀𝐴& = 𝑐𝑝(𝑖)
𝑡 =JK The MDP was still solved using Q-learning approach as in
2. Moving Average Convergence and Divergence section 4, with the only difference being that the trend of asset is
𝑀𝐴𝐶𝐷 = 𝐸𝑀𝐴KO − 𝐸𝑀𝐴OP also being used as feature in our function approximation based
2
𝑤ℎ𝑒𝑟𝑒 𝐸𝑀𝐴 𝑖 = 𝑐𝑝 𝑖 − 𝐸𝑀𝐴 𝑖 − 1 ∗ Q-Learning.
𝑡+1
+ 𝐸𝑀𝐴(𝑖 − 1) Similar to Section 4, MDP was run over the data-set consisting
3. Stochastic K of 5-year time period and 10,000 trials and Sharpe Ratio was
𝑐𝑝 𝑖 − 𝐿& reported for each data point. The plot below shows that the
𝐾% 𝑖 = ∗ 100
𝐻& − 𝐿& Sharpe Ratio of 1.4 is obtained and is much higher and stable as
compared to Section 4.
3
 networks that are trained to reconstruct linguistic contexts of
words. Word2vec takes as its input a large corpus of text and
produces a vector space, typically of several hundred dimensions,
with each unique word in the corpus being assigned a
corresponding vector in the space. Word vectors are positioned
in the vector space such that words that share common contexts
in the corpus are located in close proximity to one another in the
space.
We use the pre-trained Word2Vec model to obtain a 100-
dimensional vector for each of the headline obtained for
Qualcomm and Microsoft (2006-2016) from Reuters Key
Figure 6: Sharpe Ratio vs historical time period using Q-Learning
Development Corpus.
Similar to section 4, Monte-Carlo Simulation was considered as
a baseline. The plot below shows that providing the trend
information also does not help in improving the performance
obtained from random actions and Monte-Carlo Simulation.

Figure 7: Sharpe Ratio vs historical time period using MCS

6. Problem Formulation Augmented with Sentiment Analysis

and Trend Information
Due to the volatility of the stock market, price fluctuations based Figure 8: Sentiment Analysis using Word2Vec Model and Neural Networks
on sentiment and news reports are common. As humans, we can Neural Network:
easily interpret the sentiment of a news article. However, the
problem lies when we have multiple stocks in our portfolio, most Subsequently, we use these 100-dimensional vectors as input to
of which are being mentioned frequently in the news or social a neural network with a single hidden layer (120 dimensions) and
media sites. The average adult reads 300 wpm and takes cross entropy loss. The sigmoid function has been used as
approximately 2 minutes to read an article with 600 words. activation for the hidden layer, and softmax function for the
Within 2 minutes, 10 of the portfolio stocks generate over 100 output layer. The forward-propagation and back-propagation[20]
articles. It is impossible for humans to keep up with the massive for the cross entropy loss with regularization (λ=0.0001) are
data overload which is where sentiment analysis comes into play. implemented to perform the classification.
The model was trained on news from year 2006 to 2010. The
In this section, sentiment analysis is being performed on the news
news headlines from year 2011 to 2016 acted as a test set. The
articles of each stock and use this sentiment score as an additional
figure 9 below shows the training and test set accuracy using the
feature in the state definition. The practical implementation of the
proposed method. As is evident, the model is able to achieve a
above approach entails two steps: (A) Perform sentiment analysis
training accuracy of ~100% with the test set accuracy being
(B) Augmenting the sentiment score into MDP formulation.
~80%. It may be noted that finding the sentiment from just the
6.A Sentiment Analysis using News Articles of each asset headline is rather a difficult task as headlines can often be
The sentiment analysis has been performed in two steps, as ambiguous.
shown in the figure 8 below. The first step involves passing the
news articles through a pre-trained Word2Vec model and
obtaining N-Dimensional Vector for each headline. The second
step involves passing these vectors as inputs to a neural network
and obtaining the classification also known as sentiment score.
Word2Vec Model:
As a first step, we used a pre-trained Word2Vec Model for
Reuters News Corpus (using NLTK Library[18]). Word2Vec[19]
is a group of related models that are used to produce word
embeddings. These models are shallow, two-layer neural
Figure 9: Accuracy of training(red) and test(blue) set vs the number of epochs
4
6.B Augmenting the sentiment score into MDP formulation achieved when we use Q-Learning to solve MDP augmented with
The above section described how we can obtain the sentiment trend and sentiment analysis. This is expected because adding the
score using Word2Vec model and neural networks. trend information and sentiment scores to our state helps the ML
Consequently, we augmented the state of the system with this agent to make much more informed decisions.
information and the new state implemented was [(# of Stocks for 8. Challenges and Error Analysis
each asset), (Current Stock Price for each asset), Cash in Hand, We observed that while our proposed approach works well in a
(trend of each asset), (sentiment score of each asset)]. All other lot of scenarios, there are still significant challenges that need to
aspects of the MDP definition are same as before. be solved to build high performing autonomous trading system.
The MDP was still solved using Q-learning approach as in Some of the major challenges/limitations in our current approach
section 4 and 5, with the only difference being that the trend of are:
asset and sentiment scores are also being used as feature in our a. Large State Space: The exponential increase in the state-
function approximation based Q-Learning. action pair space as the number of assets in our portfolio
Similar to Section 4 and 5, MDP was run over the data-set increases. Since our state definition consists of (number of
consisting of 5-year time period and 10,000 trials and Sharpe stocks for each asset, stock prices of each asset, cash in
Ratio was reported for each data point. The plot below shows how hand, trend path followed by each asset, sentiment score of
the Sharpe Ratio evolves with increasing historical time period. each asset), as the number of assets increase, the action space
It may be noted that the Sharpe Ratio of 2.4 is obtained and is increases exponentially. This increases the run-time of our
much higher and stable as compared to Section 5. system considerably, and limits the performance of the
system.
b. Accuracy Improvement of Sentiment Analysis: The
accuracy of our proposed method of estimation of sentiment
score can be improved by taking the content of news articles
rather than just taking the headlines as headlines can often be
ambiguous.
9. Conclusion and Future Work
In this paper, we implemented reinforcement learning algorithm
particularly Q-Learning with functional approximations, to train
a ML agent to develop an optimal strategy to perform automated
Figure 10: Sharpe Ratio vs historical time period using Q-Learning trading. In the first section, the ML agent was successful in
learning the optimal policy, achieving a Sharpe Ratio of 0.85. In
Similar to previous sections, Monte-Carlo Simulation is order to improve the learning efficiency of the agent, Trend
considered as a baseline. The plot below shows that providing the Analysis using Neural Networks was performed and the MDP
trend and sentiment information also does not help in improving was augmented with trend information. The agent performed
the performance obtained from random actions and MCS. better and was successful in achieving the Sharpe Ratio of 1.4.
Furthermore, in order to capture the volatility in the market due
to news reports, Sentiment Analysis using Word2Vec model and
Neural Networks was performed. The sentiment score was then
augmented to the MDP formulation and solved. The agent
performed significantly better and achieved a Sharpe Ratio of 2.4.
However, although the average profit generated in 10,000 trials
is closer to Oracle (Sharpe Ratio = 3.0 being excellent), we still
see a significant run-to-run variation. The run-to-run variations
could be because of limited data set, i.e. we only have stock data
and news articles test data for 5-year time period.
Figure 11: Sharpe Ratio vs historical time period using MCS
Finally, our current work can be extended in various ways. For
7. Summary of Results instance, while performing Sentiment Analysis, only the news
The table below summarizes the Sharpe Ratio achieved from headlines have been used. This can be further extended to include
10,000 trials using the various proposed algorithms. the content of news article which will help the model to classify
Methods Sharpe Ratio the news more accurately, thereby increasing the accuracy.
Q-Learning 0.85 Further, given the large state space, it might be worthwhile to try
RL to use policy gradient methods which directly try to estimate the
MCS -0.5
Q-Learning 1.4 best policy without really generating the Q-Value. Deep Q-
RL + Trend Learning and Deep Reinforcement learning methods have
MCS -0.2
Q-Learning 2.4 recently been applied to many different problems and have
RL + Trend +
achieved very good performance. Based on the initial results
Sentiment Score MCS -0.2
from this report, we feel it could be a good idea to try to
Table 1: Summary of Results from various proposed algorithms
implement Deep-Reinforcement learning methods for this
While all algorithms are better than our baseline Monte-Carlo
problem too.
simulations (MCS), as expected, the best performance is
5
10. References
[1] https://www.investopedia.com/ask/answers/010815/what-good-
sharpe-ratio.asp
[2] Pierpaolo G. Necchi, Reinforcement Learning For Automated
Trading, 2016
[3] David W. Lu, Agent Inspired Trading Using Recurrent
Reinforcement Learning and LSTM Neural Networks, July 2017.
[4] Xin Du, Jinjian Zhai, Koupin Lv, Algorithm Trading using Q -
Learning and Recurrent Reinforcement Learning, Dec 2009
[5] Jae Won Lee, Jangmin O, A Multiagent Approach to Q-Learning
for Daily Stock Trading, Nov 2007
[6] James Cumming, An Investigation into the Use of Reinforcement
Learning Techniques within the Algorithmic Trading Domain, June
2018
[7] Győző Gidófalvi, Using News Articles to Predict Stock Price
Movements, University of California, San Diego La Jolla, CA
92037, 2001
[8] Naveed Ahmad, Aram Zinzalian, Predicting Stock Volatility from
Quarterly Earnings Calls and Transcript Summaries using Text
Regression, June 2010
[9] Qicheng Ma, Stock Price Prediction Using News Articles, CS224N
Final Report, 2008
[10] Ravi Parikh, Matin Movassate, Sentiment Analysis of User-
Generated Twitter Updates using Various Classification
Techniques, June 2009
[11] Gabriel Fung, Jeffrey Yu, Hongjun Lu, The Predicting Power of
Textual Information on Financial Markets, IEEE Intelligent
Informatics Bulletin, Vol. 5. No. 1, June 2005.
[12] Yahoo Finance, https://finance.yahoo.com/
[13] Reuters Key Development Corpus,
https://www.reuters.com/finance/stocks/QCOM.O/key-
developments
[14] http://hpelm.readthedocs.io/en/latest/index.html, HP-ELM
[15] Rajashree Dash and Pradipta Kishore Dash, A hybrid stock trading
framework integrating technical analysis with machine learning
techniques, March 2016.
[16] Markov Decision Process, Lecture Notes
[17] Q-Learning and Epsilon Greedy Algorithm, Lecture Notes
[18] NLTK Library, http://www.nltk.org/
[19] Word2Vec Model, https://en.wikipedia.org/wiki/Word2vec
[20] Forward-propagation and back-propagation, Lecture Notes