Multivariate Index Relationship Exploration Using Advanced Machine Learning Techniques

11 IV April 2023
https://doi.org/10.22214/ijraset.2023.51011
International Journal for Research in Applied Science & Engineering Technology (IJRASET)
ISSN: 2321-9653; IC Value: 45.98; SJ Impact Factor: 7.538
Volume 11 Issue IV Apr 2023- Available at www.ijraset.com
Multivariate Index Relationship Exploration Using

Advanced Machine Learning Techniques
Jay Patel1, Siddhanth Jain2, U. M. Prakash3
1
Department of Computer Science & Engineering, SRM institute of Science and Technology, Kattankulathur, Chennai, India,
603203
2
Department of Computer Science & Engineering, SRM institute of Science and Technology, Kattankulathur, Chennai, India,
603203
3
Department of Computing Technologies, SRM institute of Science and Technology, Kattankulathur, Chennai, India, 603203
Abstract: Stock market forecasts have always attracted the attention of many analysts and researchers. Popular theory suggests
that stock markets are inherently straightforward, and trying to predict them is a trivial matter. It is a difficult problem in itself.
The market behaves like a voting machine in the short term, but in the long term it behaves like a pair of scales. Therefore, there
is room to predict market movements over a longer period of time. Applying machine learning techniques and other algorithms
to stock price analysis and forecasting is a promising area. This white paper begins with a brief overview of the stock market and
a taxonomy of stock market forecasting methods. We then highlight some research findings on resource analysis and
forecasting. We discuss technical, fundamental, short- term and long-term approaches to equity analysis. Finally, we present
some challenges and research opportunities in this area.
Keywords: crash prediction, machine learning, python, correlation, drawdown
I. INTRODUCTION
Stock market forecasting and analysis is an attempt to determine the future value of an exchange-traded company's stock or other
financial instrument. The stock market is an important part of the country's economy and plays an important role in the growth of
the country's industry and trade which ultimately affects the country's economy. Both investors and industry are involved in the
stock market and want to know whether stocks will rise or fall over a period of time. The stock market is the primary source for
companies to raise funds to expand their operations. It is based on the concept of supply and demand. If demand for a company's
shares is high, the company's share price will rise, and if demand for the company's shares is low, the company's share price will
fall.
National Stock Exchange of India Limited (NSE) is India's leading stock exchange based in Mumbai. NSE was founded in 1992 as
the first demutualized electronic exchange in the country. NSE is the first stock exchange in the country to offer a fully automated,
modern, screen-based electronic trading system that offers easy trading opportunities to investors across the country.
NIFTY 50 Index is the National Stock Exchange of India's broad benchmark equity market index for the Indian equity market. It
represents a weighted average of 50 Indian corporate stocks across 12 sectors and is one of two major stock indices used in India,
the other being the BSE Sensex contains very large datasets that are difficult to extract information from and manually analyze work
development as they involve many industries and companies. The application developed in this project not only helps predict the
future movement of stocks in the market, but also automates data search, trend analysis, predictive analysis and stock insight
generation at the click of a button. Stock market analysis and forecasting reveals market patterns and predicts when to buy stocks.
Correctly predicting the future price of stocks can yield big profits.
.
II.LITERATURE SURVEY
1) Research on Stock Price Prediction Method Based on Convolutional Neural Networks, IEEE 2019 - Sayavong Lounnapha et al.
The dataset is taught and tested in relation to the behavior of convolutional neural networks and the Thai stock market. The
results show that models based on convolutional neural networks can effectively detect and imagine stock price trends. It
provides an important indicator for stock price prediction. The accuracy of the forecast is high, which could also facilitate the
financial sector.
©IJRASET: All Rights are Reserved | SJ Impact Factor 7.538 | ISRA Journal Impact Factor 7.894 | 3775
2) Enhancing Profit by Predicting Stock Prices using Deep Neural Networks, IEEE 2019-Soheila Abrishami, et al., Economic time
series forecasting is a very difficult task and has attracted the attention of many scientists. , is very important for investors. This
white paper focuses on introducing a deep learning system that predicts stock values using a set of facts about a subset of stocks
on the NASDAQ stock exchange. This model has been trained on the minimum data for a particular stock and in a few steps
accurately estimates the final value of that stock. It consists of an autoencoder that removes noise and uses time-series data
techniques to syndicate advanced features with original features. These new functions are fed into a stacked LSTM autoencoder
to make multi-level predictions of final stock values. Additionally, this estimate is used by the profit maximization approach to
provide guidance on the right time to buy or sell a particular stock. The results show that the proposed framework outperforms
state-of-the-art time series forecasting methods in terms of analytical accuracy and effectiveness.
3) LSTM Method for Bitcoin Price Prediction: A Case Study Yahoo Finance Stock Market, IEEE 2019- Ferdiansyah et al.,
Bitcoin is a type of cryptocurrency and currently a type of stock market asset. There are some risks in the stock market
And Bitcoin is a kind of virtual currency, and in recent years it has gone up and down violently, and it may plummet without
knowing the impact on the stock market. Due to its volatility, automated tools are needed to predict Bitcoin on the stock market.
This research study explores how LSTMs can be used to make modal predictive Bitcoin stock market predictions. Before
confirming the results, the paper tries to measure the results using RMSE (Root Mean Square Error). RMSE is always greater
than or equal to MAE. The RMSE metric evaluates how well the model computes continuous values. The method applied in
this study to forecast Bitcoin on the Yahoo Finance stock market can predict results above US$12600 for days after the
forecast.
4) Stock Prediction Using Machine Learning Techniques, IEEE 2019-Jeevan B et al., Everyone is talking about the stock market
these days, and science and business people are interested. This paper mainly uses his RNN (Recurrent Neural Network) and
his LSTM (Long Short Term Memory) on the National Stock Exchange, using many factors such as current market prices and
anonymous events. It deals with an approach to predicting stock prices using This white paper also describes a recommendation
system along with a model based on the RNN and LSTM methods used to select companies.
5) Stock Market Forecasting Using Machine Learning Techniques, IEEE 2020 – Naadun Sirimevan et al. Stock market prices play
an important role in today's economy. Researchers have found that social media platforms such as Twitter and internet news
tend to influence each individual's decision-making process. This study takes into account behavioral reflexes to web messages
in order to reduce the gap and make predictions more accurate. Here, accurate predictions were made for 1 day, 1 week, and 2
weeks later.
6) Machine Learning Stock Market Forecasting, IEEE 2018 – Ishita Parmar, Ridam Arora, Lokesh Chouhan, Navanshu Agarwal,
Shikhin Gupta, Sheirsh Saxena, Himanshu Dhiman. In this article, we examine a method [3] for predicting stock prices using
regression and LSTM-based machine learning. The factors that are measured are opening price, closing price, low price, high
price, and volume. This paper was an attempt to use machine learning techniques to improve accuracy and reliability in
determining the future price of a company's stock. The LSTM algorithm produced positive results with higher accuracy in stock
price prediction.
7) Stock Prediction Using Machine Learning Techniques, IEEE 2018 – Jeevan B, Naresh E, Vijaya kumar BP, Prashanth Kambli.
This paper mainly used [5] long-short-term memory (LSTM) and recurrent neural network (RNN) to predict stock prices, using
various factors such as current market prices, price-earnings ratios, and baselines. based on an approach to estimating stock
prices for NSE data. and other anonymous events. Model efficiency is analyzed by comparing true and predicted data using
RNN plots. Machine learning for predicting stock prices as the model can predict stock prices very close to the actual price.
This model captures detailed features and uses different strategies to make predictions. The model trains all NSE data from the
web, recognizes and groups inputs, and provides inputs according to user configuration. This RNN-based architecture is highly
efficient in stock price forecasting by changing the configuration accordingly and using the backpropagation mechanism to
avoid data mixing during data collection and grouping. It has been proven.
8) Stock Market Forecasting Using Machine Learning Techniques, IEEE 2016 - Mehak Usmani, Syed Hasan Adil, Kamran Raza,
Syed Saad Azhar Ali. The main objective of this study is to predict the market performance of the Karachi Stock Exchange
(KSE) at the end of the day using [6] machine learning algorithms. Predictive models are used to predict various attributes as
inputs and predictive markets as positive and negative. Features used in the model include oil prices, gold and silver prices,
interest rates, foreign exchange rates (FEX), news, and social media feeds. Machine learning algorithms including Single Layer
Perceptron (SLP), Multilayer Perceptron (MLP), Radial Basis Function (RBF), and Support Vector Machine (SVM) are
compared. The MLP algorithm, a multi-layer perceptron, performed best compared to other methods. The main feature that
helped predict the market was the oil price attribute. The final results of this study confirm that machine learning techniques can
predict stock market trends. A multi-layer perceptron machine learning algorithm predicted 70% accurate market performance.
9) Development of Predictive Models for Stock Analysis, IEEE 2017 – R. Yamini Nivetha, Dr. C. Dhaya. A comparative study of
three algorithms, namely multiple linear regression (MLR), support vector machine (SVM), and artificial neural network
(ANN), is the main purpose of this work. Forecasts are determined by monthly and daily forecasts to forecast the market for the
next day. Predict stock prices with sentiment analysis with the best prediction algorithms. A less sophisticated algorithm is the
multiple linear regression algorithm that calculates the correlation between volume and price. After research, I found that deep
learning algorithms are more developed than his MLR and SVM algorithms.
10) Stock Prediction Based on Information Entropy and Artificial Neural Networks, IEEE 2019 - Zang Yeze, Wang Yiying
One of the most important elements of the financial system is the stock market. Funds are provided by affiliated investors to
support activities and development. Together with information theory and artificial neural networks (ANNs) form a
combination of machine learning frameworks. Informational entropy for nonlinear causality and inventory association is also
creatively used in this method to facilitate ANN time series modeling. The feasibility of this machine learning framework is
analyzed at Amazon, Apple, Google, and Facebook prices. In this paper, we review time series analysis techniques based on
information theory and LSTMs for modeling stock price dynamics. Transfer entropy between related variables supporting
LSTM time series forecasting is integrated into this modeling infrastructure, so the accuracy of hypothetical results is nearly
guaranteed. Although the modeled and actual stock prices are highly correlated, they differ slightly in terms of mean absolute
error (MAE) and mean squared error (RMSE), which are examined using the results.
III. METHODOLOGY
The first step in this process is collecting financial data and identifying crashes. We've been looking for daily price information from
major stock markets with low correlation. Low cross-correlation is important for effective cross- validation and model testing. To
avoid overfitting, we avoided including two datasets with a cross-correlation greater than 0.5 in the collection. Overfitting occurs
when a machine learning model tries to cover all data points or exceeds the required data points for a given dataset. Because of this,
the model starts caching noise and inaccurate values in the dataset, all of which reduce the efficiency and accuracy of the model. To
identify clashes in each dataset, we first calculate the markdown. A drawdown is a continuous drop in price over several consecutive
days from the last high to the next low. Then use Emily Jacobson's methodology. In this methodology, a crash in any market he
defines as a drawdown at the 99.5% quantile. A quantile defines a specific portion of a data set. H. The quantile determines whether
the values in the distribution are above or below certain limits.
Fig - 1: Block diagram of our project
For our project, we collected data from Yahoo Finance. Our dataset consists of historical data for the following major indices:
(India), DAX (Europe), SMI (Switzerland), MXX (Mexico), BVSP (Brazil). Select a dataset to investigate further and later use to
develop algorithms to predict crashes. The datasets used should not exhibit strong cross-correlations to avoid overfitting to specific
patterns or biased test sets.
Fig - 2: Correlation between datasets
The correlation matrix shows that the three US indices (S&P, DJ, NDX), DAX and SMI, and MXX and BVSP are highly correlated.
To avoid overfitting when training a predictive model, we need to avoid correlations above 0.5 for any two datasets. Therefore, SJ,
NDX, DAX, and MXX are excluded for further analysis.
Fig - 3: Datasets with correlation <0.5
The correlation matrices of the remaining data sets show no correlation between any two data sets above 0.5. Now, I will plot the
distribution of prices, daily returns and drawdowns.
Fig - 4: Price Vs Time Graph of S&P
Fig - 5: Price Vs Time Graph of N225
Fig - 6: Price Vs Time Graph of SSE
Fig - 7: Price Vs Time Graph of HSI
Fig - 8 Price Vs Time Graph of BSESN
Fig - 9: Price Vs Time Graph of SMI
Fig - 10: Price Vs Time Graph of BVSP
The time series plots give an impression of the performance of the different markets over the past 50-20 years.
Now I’ll plot the Daily Return Graph for the above mentioned Index.
Fig - 11: Daily Return Vs Time Graph of S&P
Fig - 12: Daily Return Vs Time Graph of N225
Fig - 13: Daily Return Vs Time Graph of SSE
Fig - 14: Daily Return Vs Time Graph of HIS
Fig - 15: Daily Return Vs Time Graph of BSESN
Fig - 16: Daily Return Vs Time Graph of SMI
Fig - 17: Daily Return Vs Time Graph of BSVP
The amplitude of daily returns over time for all datasets give an impression of the volatility in the different markets with the
Brazilian market showing the largest daily gains/losses.
Fig - 18: Correlation of daily return of all Datasets
The autocorrelation of daily returns approaches zero when the lag exceeds one day, indicating that daily returns are not strong
predictors of next-day price movements.
Now we will identify crashes. crashes as the 99.5% empirical quantile of the drawdowns (as suggested by Jacobsson, E., Stockholm
University, in 'How to predict crashes in financial markets with the Log-Periodic Power Law', 2009).
Fig - 19: All crashes in SSE since 1966
Fig - 20: All crashes in N225 since 1965
Fig - 21: All crashes in HSI since 1987
Fig - 22: All crashes in S&P since 1950
Fig - 23: All crashes in BSESN since 1997
Fig - 24: All crashes in SMI since 1990
Fig - 25: All crashes in BSVP since 2002
Now we’ve prepared our dataset and we’ll train out Models after finding the best parameter for each of the Models.
After training each model, we will test the model for each of the indices we are considering and then finally we can predict any
index we want.
Datasets that we are using for training of Linear Regression are as follows:
S&P500 (USA), Nikkei225 (Japan), SSE (Shanghai/China), HSI (Hong Kong), BSESN (India), SMI (Switzerland), BVSP (Brazil).
Datasets that we are using for training of Logistic Regression are as follows:
S&P500 (USA), Nikkei225 (Japan), SSE (Shanghai/China), HSI (Hong Kong), BSESN (India), SMI (Switzerland), BVSP (Brazil).
Datasets that we are using for training of Support Vector Machine are as follows:
S&P500 (USA), Nikkei225 (Japan), SSE (Shanghai/China), HSI (Hong Kong), BSESN (India), SMI (Switzerland), BVSP (Brazil)
Fig - 26: Prediction of SVM for S&P
IV. CONCLUSION
We found that returns are based on duration and drawdown and are not dependent on the index's daily returns.
To identify the decline in each dataset, we first calculated the price decline. A drawdown is a continuous drop in price for several
days in a row from the last high to the next low.
Simple price patterns defined by long-term price movements and volatility changes seem to occur regularly before crashes. The best
models were able to learn these patterns and predict crashes much better than comparable random models. For example, the best
regression model for 3-month crash prediction achieved an accuracy of 0.15 and a recall of 0.59 on the test set, whereas an
equivalent random model with no predictive power achieved an accuracy of 0.04 and a recall of 0.16. Achieved. The results for 1-
month and 6-month crash predictions are similar, with the highest F-beta scores for 6-month prediction and the worst for 1-month
prediction. Whether these results are sufficient to optimize investment strategies is debatable.
Looking at the test data during the crashes and the price index chart of the crash predictor indicator, some crashes were detected
very well, while others occurred with little or no warning from the crash predictor.
REFERENCES
[1] Chiu, DY, & Chen, P.J. (2009). Dynamically explore the internal mechanisms of the stock market through fuzzy-based support vector machines with high-
dimensional input spaces and genetic algorithms. Expert Systems with Applications, 36(2), 1240-1248.
[2] Chong, E., Han, C., & Park, F.C. (2017). Deep Learning Networks for Analysis of Stock Market and Forecasting: Methodology, Data Plots, and Case Studies.
Expert Systems with Applications, 83, 187-205.
[3] Dai, W., Wu, J.Y., and Lu, C.J. (2012). A combination of non-linear independent component analysis and neural networks for predicting Asian stock market
indices. Expert Systems with Applications, 39(4),-4452.
[4] Das, S.P., & Padhy, S. (2012). Supporting vector machines for predicting Indian stock market futures prices. International Journal of Computer Applications,
41(3).
[5] Dash R. & Dash P.K. (2016). A hybrid stock trading framework that integrates technical analysis and machine learning techniques. Journal of Finance and Data
Science, 2(1), 42-57.
[6] Enke, D. & Thawornwong, S. (2005). Predict stock market returns using data mining and neural networks. Expert Systems with Applications, 29(4), 927-940.
[7] Goodhue, D.L., & Thompson, R. L (1995). Adaptability of task technology and individual performance MIS Quarterly, 213-236.
[8] Guresen, E., Kayakutlu, G., & Daim, TU (2011). Use of artificial neural network models in prediction of the stock market. Expert Systems with Applications,
38(8), 10389-10397.
[9] Kim, K.J. & Lee, W.B. (2004). Stock market forecasting using artificial neural networks with optimal feature transformations. Neural Computing and
Applications, 13(3), 255-260.
[10] Jasic, T. & Wood, D. (2004). Profitability of Daily Stock Market Indices Based on Neural Network Predictions: A Case Study of the S&P 500, DAX, TOPIX,
and FTSE from 1965 to 1999. Applied Finance, 14(4), 285-297.
[11] Kim, H.J. & Shin, K. S (2007). A hybrid approach on study of neural networks and genetic algorithms to identify temporal patterns in stock markets. Applied
Soft Computing, 7(2), 569-576.
[12] Kim, K. J. & Han, I. (2000). A Genetic Algorithm Approaches Feature Discretization in Artificial Neural Networks for Stock Index Prediction Expert Systems
with Applications, 19(2), 125-132.

Multivariate Index Relationship Exploration Using Advanced Machine Learning Techniques

Uploaded by

Copyright:

Available Formats

Multivariate Index Relationship Exploration Using Advanced Machine Learning Techniques

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Multivariate Index Relationship Exploration Using Advanced Machine Learning Techniques

Uploaded by

Copyright:

Available Formats

11 IV April 2023

Multivariate Index Relationship Exploration Using

Fig - 1: Block diagram of our project

Fig - 2: Correlation between datasets

Fig - 3: Datasets with correlation <0.5

Fig - 4: Price Vs Time Graph of S&P

Fig - 5: Price Vs Time Graph of N225

Fig - 6: Price Vs Time Graph of SSE

Fig - 7: Price Vs Time Graph of HSI

Fig - 8 Price Vs Time Graph of BSESN

Fig - 9: Price Vs Time Graph of SMI

Fig - 10: Price Vs Time Graph of BVSP

Fig - 11: Daily Return Vs Time Graph of S&P

Fig - 12: Daily Return Vs Time Graph of N225

Fig - 13: Daily Return Vs Time Graph of SSE

Fig - 14: Daily Return Vs Time Graph of HIS

Fig - 15: Daily Return Vs Time Graph of BSESN

Fig - 16: Daily Return Vs Time Graph of SMI

Fig - 17: Daily Return Vs Time Graph of BSVP

Fig - 18: Correlation of daily return of all Datasets

Fig - 19: All crashes in SSE since 1966

Fig - 20: All crashes in N225 since 1965

Fig - 21: All crashes in HSI since 1987

Fig - 22: All crashes in S&P since 1950

Fig - 23: All crashes in BSESN since 1997

Fig - 24: All crashes in SMI since 1990

Fig - 25: All crashes in BSVP since 2002

Fig - 26: Prediction of SVM for S&P

Fig - 27: Prediction of SVM for S&P

Fig - 28: Prediction of SVM for S&P

You might also like