A Natural Language Processing For Sentiment Analysis From Text Using Deep Learning Algorithm
A Natural Language Processing For Sentiment Analysis From Text Using Deep Learning Algorithm
language processing, such as sentiment analysis.In order to reviews from various online and social networking platforms,
consider the context of each and every word in a sentence, including those for products, services, entertainment, and
BERT uses a transformer-based architecture. This is crucial politics. First, we improved the BERT, then we used the
for sentiment analysis because the surrounding words can aggregating layer as the text embedding, and finally, we
affect how a word is perceived. The first step in using BERT created a highly integrated model called CNN-BiLSTM for
for sentiment analysis is to hone the pre-trained model using a decision-making. Due to its high accuracy in tasks requiring
set of labelled text data. This entails teaching the model to natural language processing and its capacity to comprehend
anticipate the tone of a sentence given its surrounding context. the context of each word in a sentence, BERT is, overall, an
effective tool for sentiment analysis of text data.
By taking into account the other words in a sentence,
BERT is able to capture contextual word representations. This 2. LITERATURE SURVEY
enables BERT to grasp subtleties and clear out any potential
ambiguities by understanding words in the context of the Hou.Y et. al [1] proposed Sentiment analysis in 2015
entire phrase. Such contextual information is not captured by on the Chinese poetry,” Analyzing sentiment in classical
conventional models like Bag-of-Words or lexicon-based Chinese poetry”. These studies show how effective BERT is
techniques. for sentiment analysis while also highlighting ongoing work to
enhance and broaden this methodology. A corpus of
An N-gram model relay on the occurrence of traditional Chinese poetry is examined using natural language
sequence of words but not on the relationship among the processing techniques in order to extract sentiment-related
words but BERT relies on the relationship among the words components. The poems are then divided into several
and also it has bidirectional attention. sentiment categories using machine learning models that we
train and test.
Ghosh.A et. al [2] proposed ‘‘SemEval-2015 task 11:
Sentiment analysis of a figurative language in the Twitter’’ in
2015 used a series of heuristics, this first finds tweets that use
figurative language, and then it uses machine learning
techniques to identify the sentiment communicated in the
tweet. They experimented with lexical, syntactic, and
sentiment-specific feature sets, among others. This study
contributes to the development of sentiment analysis
techniques for social media content because figurative
language is frequently used in it.
Sreeja.P.S et. al [3] has proposed “Emotion
recognition from poems by maximum posterior probability”,
2016. It has the potential for practical applications like
sentiment analysis in poetry collections or recommendation
systems for readers based on their emotional preferences. The
suggested method for emotion identification from poems
utilising the Maximum Posterior Probability methodology
yields encouraging results.
Kaur.J et. al [4] proposed ‘‘Punjabi poetry
classification: The test of 10 machine learning algorithms” in
2017.The study stresses the significance of feature extraction
techniques in obtaining high classification accuracy and offers
insights into the efficacy of several machine learning
algorithms for categorising Punjabi poetry. The findings can
be used to guide the creation of automated systems for the
study and preservation of Punjabi poetry ten machine learning
algorithms are tested on how well they categorize Punjabi
Figure 1.1 Block Diagram of Proposed Method poems into six groups based on author and genre such as
A probability score that indicates the likelihood that a Decision Tree Algorithm (DT), Random Forest Algorithm
given sentence expresses a specific sentiment, such as a (RF), K-Nearest Neighbour Algorithm (KNN), Naive Bayes
positive, negative, or neutral sentiment, is the usual output of Algorithm (NB), Logistic Regression Algorithm (LR),
BERT for sentiment analysis. Text data can be categorized Support Vector Machine Algorithm (SVM), Gradient
into various sentiment categories using this score. This Boosting Machine Algorithm (GBM), XGBoost Algorithm
involves analysing a sizable sentiment dataset that contains (XGB), Multilayer Perceptron Algorithm (MLP), and
Convolutional Neural Network (CNN).
Mohanty.G et. al [5] has proposed a study to compile varioustechniques for the sentiment analysis and machine
a corpus of Odia poetry with information on sentiment learning algorithms are outperformed on several datasets.
polarity in 2018,”Sad or glad? Corpus creation for an Odia
Raza.M.R et. al [12] has proposed a study between
poetry with the sentiment polarity information“, specifically
the deep learning approaches and the importance of
indicating whether the poem expresses happiness or sadness.
implementing the sentimental analysis for text in cloud
The corpus is produced by gathering a wide range of Odia
computing by comparing with the existing approaches.
poems from various sources and manually annotating them for
sentiment polarity. The study assesses how well various
machine learning algorithms predict the sentiment polarity of Mahmud M.S et. al [13] has proposed a model and
the poems as well as the inter-annotator agreement among the evaluated the sentimental analysis to detect the opinions from
the reviews of text using the DistillBert algorithm and it
annotators. Researchers interested in examining the sentiment
outperformed when compared to the other approaches by
and emotions expressed in Odia poetry may find the corpus
achieving the 98% of highest accuracy.
useful for sentiment analysis tasks as well.
Devlin.J et. al [6] ”BERT: Deep Bidirectional Table 2.1: Overview of Survey
Transformers for Language Understanding Pre-training”.
Human Language Technologies: Proceedings of the 2019 S.No Authors Year of Technique
Conference of the Association for Computational Linguistics' publication
North American Chapter (NAACL-HLT 2019). The BERT 1 Hou.Y and 2015 Natural language
model was introduced in this paper, which also demonstrated Frank.A processing
how it performed better than earlier cutting-edge models at a
number of NLP tasks, including sentiment analysis. 2 Ghosh.A, Li.G, 2015 machine learning
Xiong.C et. al [7] Using BERT to Create an Veale.T,
Auxiliary Sentence for Aspect-Based Sentiment Rosso.P,
Analysis.2019 Conference on Empirical Methods in Natural Shutova.E,
Language Processing and the 9th International Joint Barnden.J, and
Conference on Natural Language Processing This study Reyes.A
suggests a novel approach for aspect-based sentiment analysis 3 Sreeja.P.S and 2016 Maximum
using BERT and auxiliary sentence construction. Mahalakshmi.G. Posterior
S Probability
Carbonell.J.G et. al [8] ” Generalized Autoregressive
4 Kaur.J and 2017 Decision Tree
Pretraining for Language Understanding is an XLNet
Saini.J.R Algorithm,
technology”. In Neural Information Processing Systems:
Random Forest
Advances (NeurIPS 2019). The XLNet model, which this
Algorithm, K-
paper proposed, is similar to BERT but has some advantages
Nearest Neighbor
over it due to the permutation-based training method it
Algorithm, Naive
employs. The researchers demonstrated that XLNet produced
Bayes Algorithm,
cutting-edge outcomes on a number of benchmark datasets,
Logistic
including sentiment analysis.
Regression
Chen.Y et. al [9] ” Classification of Sentences Using Algorithm,
BERT Fine-Tuning”. Reports from the 2020The sixth Support Vector
international conference on robotics, automation, and control Machine
(ICCAR 2020). This study offered a thorough examination of Algorithm,
various BERT fine-tuning strategies for sentiment analysis Gradient
and suggested a brand-new strategy based on data Boosting
augmentation. Machine
Algorithm,
LeCun.Y et. al [10] “Convolutional networks at the XGBoost
character level for text classification”. In the 28th Algorithm,
International Conference on Machine Learning Proceedings Multilayer
(ICML 2020). The new sentiment analysis model proposed in Perceptron
this paper outperformed BERT on several benchmark datasets
Algorithm, and
and was based on character-level convolutional neural
Convolutional
networks (CNNs). Neural Network
Rahat, A. Met. al [11] developed an area where Algorithm.
people can easily detect their choices using the sentimental 5 Mohanty.G and 2018 creating a corpus
analysis. This study offered a thorough examination of Mishra.P of Odia poetry
3. METHODOLOGY
CLS:
The word means classification. It is included to
signify classification at the sentence level. In order for it to
fully represent the sentence,it is typically added at the
beginning of a sequence.
SEP:
When predicting the following statement, the SEP is
utilized. It acts as a delimiter token to inform the model about
the next sentence. It is merely added at the conclusion of one
sentence. During the MLM phase, this token is utilized. It
serves as the model's representation of the masked word.
Token embeddings:
Token embeddings procedure involves pre training of Figure 3.1 BERT Model
different words and these pre trained embeddings are created
Therefore, a different embedding known as the
using Word Piece Tokenisation which is a segmentation
position embeddings are applied to solve it.The absolute
algorithm.This algorithm merges set of different characters to
position embedding is used to explain how a token at one
build vocabulary which is used as language model.New words
place attends to another token at a different position. Up to
can be created by using different combinations of characters
512 charactersinput sequences can be processed by BERT.
initialized which increases the accuracy of the training
The authors used BERT to train a vector representation for
data.Each word fragment token can be transformed into a 768-
each point, taking advantage of the input sequences' sequential
dimensional vector representation using the token embeddings
nature. In position embeddings layer, the vector representation
layer. By using basic elements built in the language model, it
of any word in the first position for first row and the second
can define the size of number of word units to be reached or a
position for second row, and so on, is basically a lookup table
certain threshold can be used.
with a range of (512, 768) rows.For BERT to support
Segment embeddings: temporal ordering, segment and position embeddings are
required. The total embeddings such as Position,
This is followed by token embeddings we can know
Segmentation and token embeddings are called as input
by inserting a fixed token into sentences, a process known as
embeddings. In order to create a single representation with
segment embedding, we can determine if a given token
shape, the elements from all three of the aforementioned
belongs to a given sentence or not. It encodes a sentence into a
embeddings are added together (1, n, 768). The input
vector and has 2 vector representations in this layer.We assign
representation used by BERT's encoder layer is this. One of
these vectors to the words based on sentence number. For
the most important factors contributing to the model's speed
example, if a word belongs to sentence 1 an index 0 is
and exceptional performance is its BERT embeddings.
and Figure 4.2 representing the accuracy of the model and loss 5. CONCLUSION
of the model during training and testing. The accuracy of the
Sentiment analysis, often known as opinion mining, is a
model during initial training was very low as number of
methodwhich is used in natural language processing (NLP) to
training epochs increased the accuracy of the model has been
determine the emotional undertone of a document. It can be
increased which shows the increase in confidence level of the
used to determine whether a piece of writing conveys positive,
model in predicting the statement. Also the loss the model
neutral, or negative emotions. Since BERT was trained on a
during initial stage of training was very low and it is increased
huge text corpus, the architecture or model was able to learn a
when number of training epochs increased and during
variety of data patterns, comprehend language better, and
validation the loss has become minimal due to training it large
effectively generalize to a variety of NLP applications. The
number of epochs.The model accuracy has also been
refined BERT model can outperform other conventional
gradually increasing during testing. This shows the model is
machine learning models and deep learning models on a
able to predict the sentiment correctly of the given sentence as
variety of sentiment analysis datasets. Organizations
the training set increases and also by running more number of
frequently employ this strategy to identify and group concepts
epochs the graph during validation proves this.
related to a particular product, service, or idea. BERT is an
effective language model that can capture intricate verbal
representations and patterns, making it suitable for sentiment
analysis tasks.The following are some advantages of
sentiment analysis: gathering a lot of unstructured data from
several sources. Monitor the moment of customer mood and
feedback regarding a company's name, goods, and services.
Giving suggestions on how to make products, services, and
the consumer experience better.
6. REFERENCES
[1] Y. Hou and A. Frank, ‘‘Analyzing sentiment in classical Chinese
poetry,’’ in Proc. 9th SIGHUM Workshop Lang. Technol. Cultural
Heritage, Social Sci., Hum. (LaTeCH), 2015, pp. 15–24.
[5] G. Mohanty and P. Mishra, ‘‘Sad or glad? Corpus creation for Odia
poetry with sentiment polarity information,’’ in Proc. 19th Int. Conf.
Comput. Linguistics Intell. Text Process. (CICLing), Hanoi, Vietnam,
2018
[6] Devlin, J., Toutanova, K., & Chang, M.W. (2018). “BERT: Deep
Bidirectional Transformers for Language Understanding Pre-
training”.
[7]Xiong, C., Sun, C., Huang, L., and Huang, L. (2019).” Using BERT to
Create an Auxiliary Sentence for Aspect-Based Sentiment Analysis”.
[8] Carbonell, J.G., Salakhutdinov, R., Yang, Z., Dai, Z., Yang, Y., & Le,
Q.V. (2019).” Generalized Autoregressive Pretraining for Language
Understanding is an XLNet technology”.
[9] Chen Y., Chen X., Chen Z., and Wang, J. (2020). “Classification of
Sentences Using BERT Fine-Tuning”.
Figure 4.2 A sample line graph shows model loss versus epochs during [10] LeCun, Y., Zhao, L., and Zhang, X. (2020). “Convolutional networks
training and validation. at the character level for text classification”.