ISSN (Print) : 0974-6846

Indian Journal of Science and Technology, Vol 10(17), DOI: 10.17485/ijst/2017/v10i17/106493, May 2017 ISSN (Online) : 0974-5645

Evaluation of Unsupervised Learning based

Extractive Text Summarization Technique for Large
Scale Review and Feedback Data
Jai Prakash Verma1* and Atul Patel2
1
Department of CE, Institute of Technology, Nirma University, Ahmedabad – 382481, Gujarat, India;
jaiprakash.verma@nirmauni.ac.in
2
CMPICA, CHARUSAT University, Changa – 388421, Gujarat, India; atulpatel.mca@charusat.ac.in

Abstract
Background/Objectives: Supervised techniques uses human generated summary to select features and parameter for
summarization. The main problem in this approach is reliability of summary based on human generated parameters
and features. Many researches have shown the conflicts in summary generated. Due to diversity of large scale datasets,
supervised techniques based summarization also fails to meet the requirements. Big data analytics for text dataset also
recommends unsupervised techniques than supervised techniques. Unsupervised techniques based summarization
systems finds representative sentences from large amount of text dataset. Methods/Statistical Analysis: Co-selection
based evaluation measure is applied for evaluating the proposed research work. The value of recall, precision, f-measure
and similarity measure are determined for concluding the research outcome for the respective objective. Findings: The
algorithms like KMeans, MiniBatchKMeans, and Graph based summarization techniques are discussed with all technical
details. The results achieved by applying Graph Based Text Summarization techniques with large scale review and
feedback data found improvement over previously published results based on sentence scoring using TF and TF-IDF. Graph
based sentence scoring method is much efficient than other unsupervised learning techniques applied for extractive text
summarization. Application/Improvements: The execution of graph based algorithm with Spark’s Graph X programming
environment will secure execution time for this types of large scale review and feedback dataset which is considered under
Big Data Problem.

Keywords: Extractive Text Summarization, Sentence Scoring Methods, Unsupervised Learning

1. Introduction sion making. Such Text Summarization Systems can be

categorized as per following categories.
Content based Recommendation system for large amount Extractive and Abstractive Summarization Systems2–4
of text data generated by different stake-holders for an are the techniques to summarize large amount of text
organization in the form of review and feedback. These using computer programs. Extractive summarization
types of text data are generated from different types of technique is based on selection of representative text
computerized automated feedback and review system or from the given large text data. Abstractive summarization
extracted from web. This is a type of Big Data Analytics is generating summary, based on the sense and feeling of
problem because of data volume, velocity and variety1. An the text document. Here in abstractive summary, we may
extractive text summarization based recommendation use new words for sensing the large text but in extrac-
system model is proposed for analyzing and summarizing tive summarization program will identify representative
these large amount of text data known as Big Data. The set of words and sentences. Single and Multi-Document
system helps finding actionable insights for better deci- Summarization Systems5,6 categorize text summariza-

*Author for correspondence

 Evaluation of Unsupervised Learning based Extractive Text Summarization Technique for Large Scale Review and Feedback
Data

tion systems based on approach in which number of Decision Trees, Hidden Markov Model, Log-linear
documents are selected for analyzing the dataset. Multi Models, and Neural Networks. Algebraic Approaches2–4
document summarization is more complex than single such as Latent Semantic Analysis (LSA), Non-negative
document summarization but it recommends for review Matrix Factorization (NMF), and Semi-discrete Matrix
and feedback summarization. Generic and Query-Based Decomposition (SDD) are also used for text summariza-
Summarization Systems7–9 categories summarization tech- tion.
niques between a specific request based and generic query Remaining sections of the paper comprise as follows.
based summarization. Generic summary is based on main In section 2 sentence scoring based text summariza-
topics covered in the text but query based summarization tion techniques are discussed. In section 3 unsupervised
specifies the request or question for summary. Another learning based text summarization techniques are dis-
categorization of text summarization systems are based cussed. In section 4 evaluation methodologies for text
on the Supervised and Unsupervised Techniques10–12. summary generated are discussed. Section 5 produces the
Supervised techniques use dataset which are annotated by proposed research work, its techniques and approaches.
humans before applying the algorithm but unsupervised In section 6, experimental study is described for proposed
techniques do not use this type of human annotations research work. Section 7 discusses performance analy-
with dataset. Unsupervised techniques use the linguistic sis based on evaluation methodologies. In Section 8, we
and statistical information generated from the dataset present conclusion and future extension in the research
for text summarization. Another categorization in text work proposed.
summarization systems are based on Surface-Level and
Deeper-Level Summarization Systems13. Surface-Level
and Deeper-Level Summarization Systems summarize
2. Sentence Scoring Based Text
the text as per the purpose of summary. Generally this Summarization
type of summarizations are used for news articles, scien- Sentence scoring methods discussed in many research
tific text etc. papers basically emphasize on word score, sentence score
Extractive Text Summarization selects representa- and graphs, where word score and sentence scores are
tive sentences from available large scale text dataset. counted based on the frequencies of word in given text
These sentences are selected based on different methods. dataset. The graph based sentence scoring is based on
One method is based on Surface Level Approaches2,3 in relationship between the sentences. The focus of many
which sentences are selected based on the most frequent researches is on analysis of large scale text available or
words. This type of method gives good results for query written with print media. Research work in this paper
and purpose based summarizations but summariza- is focused on analyzing the large amount data extracted
tion for reviews and feedback is not appropriate for it. from web in the form of review and feedback about an
Another method is based on Statistical Approaches2,14 enterprise or organization for their products and services.
which gives the summary based on relevance of informa-
tion extracted from dictionaries. For finding relevance
information about the selected text classifier, algorithms 3. Unsupervised Learning Based
like Bayesian classifier are used. Another type of method Text Summarization
is based on Text Connectivity Approaches2,4. In this
approach, text summarization is generated by the con- Supervised techniques use human generated summary
nectivity of sentences and text based on lexical chains and to select features and parameters for summarization.
Rhetorical Structure. Another type of method is based on The main problem in this approach is reliability of sum-
Graph Based Approaches4. The nodes of directed graph mary based on human generated parameters and features.
represent sentences of text, and edges represent the simi- Many researches have shown the conflicts in summary
larity between these sentences. Summary is generated generated. Due to diversity of large scale datasets, super-
by selection of sentences with highest similarity associ- vised techniques based summarization are also not fitted.
ated. Another method is based on Machine Learning Study and research on big data analytics for text dataset
Based Approaches13. The machine learning based sum- also recommends unsupervised techniques and their
marization algorithms use techniques like Naïve-Bayes, acceptance than supervised techniques15–17. Unsupervised

2 Vol 10 (17) | May 2017 | www.indjst.org Indian Journal of Science and Technology
 Jai Prakash Verma and Atul Patel

techniques based summarization systems find represen-

tative sentences from large amount of text dataset. | {relevant terms} ∩ {retrieved terms} |
Recall = (1)
| {relevant terms} |

4. Evaluation Methodology | {relevant terms} ∩ {retrieved terms} |

Precision = (2)
Evaluation methodologies for summary generated by | {retrieved terms}|
different techniques are mainly comparing computer gen-
erated summary with human made summary. Here few F-Measure = (3)
methods are discussed for evaluating computer gener-
ated summary based on their effectiveness and usability. 4.2 Content-Based Evaluation
Text Quality Evaluation: The text should not contain
any grammatical error such as incorrect words or punc- Co-selection based evaluation measures the number of
tuation errors. Co-Selection Evaluation: Where extracted match terms or sentences between both the summaries. It
summaries are compared with ideal summaries. ignores the fact that both the terms and sentences contain
Content-Based Evaluation: Compare extracted and the same information even though both are written differ-
ideal summaries, even though they do not share sentence. ently. Content- Based Evaluation measure can overcome
For content-based evaluations, measures such as cosine these issues. Cosine Similarity is a basic content-based
similarity, longest common subsequence, pyramids, evaluation measure (equation-4), where X and Y repre-
and ROUGE scores are used. Task-Based Evaluation: sent sentences or terms in both the summaries.
Compared according to their performance of accom-
plishing the given task. CosineSim(X, Y) = (4)

4.1 Co-Selection Evaluation

5. Proposed Research Work
Co-Selection Text Summary Evaluation technique is
based on comparison of gold summary with computer Research work in the area of extractive text summariza-
generated summary. The main metrics for this method tion with unsupervised learning approach is proposed
are recall, precision, and f-measure. Recall is the number in this paper. An experimental analysis of unsupervised
of terms in both the summaries divided by total number techniques with python programming language is imple-
of terms in gold summary (relevant terms) (equation 1). mented and discussed. All the steps like data extraction,
Precision is the total number of terms in both the summa- cleaning, preprocessing, and analyzing for text summary
ries divided by number of terms in computer generated generation are discussed and implemented. As per Figure
summary (equation 2). F-measure is a composite mea- 1 proposed work highlights the expected data sources,
sure of recall and precision. Due to contradiction between preprocessing steps, and analysis processes. Due to large
recall and precision measure, researchers recommend amount of data, this work also recommends open source
harmonic average of both the measures for performance solutions for handling the data in effectively and effi-
evaluation as f-measure (equation 3). ciently.

Figure 1. Process flow for proposed research work.

Vol 10 (17) | May 2017 | www.indjst.org Indian Journal of Science and Technology 3
 Evaluation of Unsupervised Learning based Extractive Text Summarization Technique for Large Scale Review and Feedback
Data

6. Experimental Analysis 1) Remove Punctuation characters

2) Lowercase Conversion
In this work, a recommendation system model is pro- 3) Remove numerals
posed to find actionable insights from the text data 4) Spelling Corrections
extracted from different web pages and different com- 5) Singularization
puter automated review or feedback systems. 6) Converting all words in Base Form
The steps for the proposed model are 7) Stop-words Removal.

6.1 Dataset Selection Data Preprocessing Steps Execution for presentation:

Following dataset was used in this research work for As shown in Figure 2 one text file demoTest.txt is used as
experimental study. Input text (python program implementation).
Opinion Dataset 1.04: This dataset contains sentences
extracted from reviews on a given topic. The Opinion 6.3 Tokenization and Vectorization
dataset also comes with human composed summaries for Tokenization is a process for breaking a text stream in
evaluation. Feedback and Review data for an educational words, sentences, phrases and other meaningful objects
institution. called tokens. Vectorization converts these text docu-
Primary Source: Different feedback systems. ments to matrix of token based on their occurrences.
Secondary Source: Data Extracted from Facebook and Following are the methods (implemented in python) used
website. Amazon’s Review and Feedback data for different for both the task:
products (Size: 34 GB). hasher = TfidfVectorizer(input = dataset, max_df=0.5,
min_df=2, stop_words=’english’,use_idf=1)
6.2 Data Preprocessing vectorizer = make_pipeline(hasher,
Following are the data pre-processing steps selected for TfidfTransformer())
this research work. X = vectorizer.fit_transform(dataset)

Figure 2. Data preprocessing steps for extractive text summarization.

4 Vol 10 (17) | May 2017 | www.indjst.org Indian Journal of Science and Technology
 Jai Prakash Verma and Atul Patel

Feature Selection: In machine learning and statis- km = MiniBatchKMeans(n_clusters=8, init=’k-

tics, feature selection is the process of selecting a subset means++’, n_init=1,init_size=1000, batch_size=1000,
of terms from the large amount of terms generated by verbose=0)
tokenization and vectorization step for use in model con- Graph Based Text Summarization: Python program
struction. is implemented for graph based text summarization
Cluster and Summary Analysis: Three approaches are where sentences are represented by vertices and similar-
selected for Summary Analysis. ity between sentences are represented by edges between
a) Sentence Score based on word TF. vertices. It is an unsupervised learning based approach
b) Sentence Score using Word Base Form’ TF. for extractive text summarization by automatic sen-
c) Sentence Score using Word Base Form’ TF-IDF. tence extraction using graph based ranking algorithms.
The results achieved by applying Graph Based Text
Following are the python implementation code that Summarization techniques with large scale review and
have used for clustering analysis. MiniBatchKMeans is feedback data found improvement with previously pub-
modified version of KMeans algorithm which is more lished results based on sentence scoring using TF and
efficient for large scale web data. Due to large size and TF-IDF. In short, a graph-based ranking algorithm is a
time constraint for cluster MiniBatchKMeans perform way of deciding on the importance of a vertex within a
more effectively compare to KMeans. Suppose we have a graph, in this research work the vertexes represent the
dataset of 500000 reviews and feedbacks, and objective is review or feedback given by individual, by taking into
to divide them into 100 clusters. The complexity of the account global information recursively computed from
original K-Means clustering algorithm is O(n*K*I*f), the entire graph, rather than relying only on local vertex-
where n is the number of records, K is the number of specific information. Participating review and feedback
clusters, I is the number of iterations and f is the num- in summary sentences are well connected to other
ber of features. It can be clearly seen that this will take a sentences. The connectivity of the sentences which is
lifetime for the original algorithm to cluster data. In this represented by vertexes is based on similarity with other
research work review or feedback given by individual is sentences. Similarity measure like TF-IDF can be selected
considered a document. Due to large size the small size as per performance of the system. Graph G (V, E), where
subsets are selected form original dataset and then apply V: set of sentences and E: similarity between sentences.
the algorithms for clustering. The algorithm takes small A threshold value is decided for similarity between the
batches (randomly chosen) of the dataset for each itera- sentences. Sentence score is calculated based on the Rank
tion. It then assigns a cluster to each data point in the of sentences which is estimated by their degree. Top k
batch, depending on the previous locations of the cluster sentences are selected for summarizing sentences.
centroids.
For Kmeans:
km = KMeans(n_clusters=8, init=’k-means++’, max_ 7. Performance Evaluation
iter=100, n_init=1,verbose=0) As per Table 1, the outcome from above experimental
For MiniBatchKMeans: analysis shows that unsupervised techniques for extrac-
Table 1. Comparison of Unsupervised extractive text summarization techniques
Algorithm ID Average_Precision Average_Recall Average_f-measure Average_Similarity
Sentence Score Based on 0.137966126452 0.206132266328 0.164604495226 0.224801884677
word TF
Sentence Score using Word 0.140276737872 0.309796632534 0.191952080092 0.239164737438
Base Form’ TF
Sentence Score using Word 0.123298003326 0.266903172868 0.167818834179 0.220944272061
Base Form’ TF-IDF
Kmeans 0.147341297188891 0.324066229588176 0.201706815726618 0.153989154237332
MiniBatchKMeans 0.151333386161 0.331919322357 0.207004641971 0.144447037715
GraphBased Summarization 0.155983580672 0.339218973418 0.212496582933 0.375529302366

Vol 10 (17) | May 2017 | www.indjst.org Indian Journal of Science and Technology 5
 Evaluation of Unsupervised Learning based Extractive Text Summarization Technique for Large Scale Review and Feedback
Data

tive text summarization improve the recall, precision, and 5. Ittoo A. Text analytics in industry: Challenges, desiderata
f-measure. MiniBatchKMeans improves the result than and trends. Comput Industry. 2016. Crossref.
K-Means. Graph Based Text Summarization improves the 6. Khan A, Salim N, Obasa AI. An Optimized Semantic
results with recall, precision, and f-measure. Here we are Technique for Multi- Document Abstractive
comparing unsupervised learning techniques with sen- Summarization. Indian Journal of Science and Technology.
2015 Nov; 8(32):1–11. Crossref
tence scoring methods for extractive text summarization.
7. Lloret E, Palomar M. Tackling redundancy in text sum-
marization through different levels of language analysis.
8. Conclusion Computer Standards & Interfaces. 2013; 35:507–18.
8. Bridge D, Healy P. The GhostWriter-2.0 Case-Based
An unsupervised learning based extractive text summari- Reasoning system for making content suggestions to the
zation system is implemented and evaluated with different authors of product reviews. Knowledge-Based Systems.
algorithms. Graph based sentence scoring method is 2012; 29:93–103. Crossref
implemented and evaluated with traditional sentence 9. Online Shopping touched new heights in India in 2012.
scoring methods. Programming with Spark program- Hindustan Times, 31 December 2012. 2014 July; 3(7):1–7,
ming framework on Hadoop Distributed File System Retrieved on 31 December 2012.
10. Bing LI, Keith CC, Chan. A Fuzzy Logic Approach for
storage is better for efficient execution when compared
Opinion Mining on Large Scale Twitter Data. IEEE/
to other Map Reduce with Hadoop environment. Graph
ACM 7th International Conference on Utility and Cloud
based sentence scoring method gives comparatively bet- Computing, 2014. p. 652–7.
ter result than other unsupervised learning techniques 11. Ghorpade T, Ragha L. Hotel Reviews using NLP and
applied for extractive text summarization. Analyzing Bayesian Classification. International Conference on
Amazon’s Review and feedback dataset can provide the Communication, Information & Computing Technology
future enhancement in this work. (ICCICT), Mumbai: 2012 Oct 19-20; 84(6):17–22.
12. Khan A, Baharudin B. Sentiment Classification Using
Sentence-level Semantic Orientation of Opinion Terms
9. References from Blogs IEEE. IEEE. 2011; 1–17.
1. Verma JP, Patel B, Patel A. Big Data Analysis. 13. Thiago S, Guzella, Walmir M, Caminhas. A review of
Recommendation System with Hadoop Framework, machine learning approaches to Spam filtering. Elsevier
IEEE International Conference on Computational Journal - Expert Systems with Applications. 2009;
Intelligence & Communication Technology. 2015. p. 1–6. 36:10206–22. Crossref
PMCid:PMC4410521 14. Sheshasaayee A, Jayanthi R. A Text Mining Approach to
2. Ferreira R, Cabral LS, Lins RD, Silva GP, Freitas F, George Extract Opinions from Unstructured Text. Indian Journal
DC, Cavalcanti A, Lima RA, Steven J, Simske B, Favaro L. of Science and Technology. 2015 Dec; 8(36):1–4. Crossref
Assessing sentence scoring techniques for extractive text 15. Nomoto T, Matsumoto Y. A New Approach to Unsupervised
summarization. Expert Systems with Applications. 2013; Text Summarization. SIGIR’01, Septe, New Orleans,
40:5755–64. Crossref Louisiana, USA: 2001. p. 1–9.
3. Xiang Z, Schwartz Z, John H, Gerdes J, Uysal M. What can 16. Sulthana AR, Subburaj R. An Improvised Ontology
big data and text analytics tell us about hotel guest experi- based K-Means Clustering Approach for Classification
ence and satisfaction. International Journal of Hospitality of Customer Reviews, Indian Journal of Science and
Management. 2015; 44:120–30. Crossref Technology. 2016 Apr; 9(15):1–6. Crossref
4. Ganesan K, Zhai C, Han. Opinosis. A Graph Based 17. Anuradha G, Varma DJ. Fuzzy Based Summarization of
Approach to Abstractive Summarization of Highly Product Reviews for Better Analysis. Indian Journal of
Redundant Opinions. Proceedings of the 23rd International Science and Technology. 2016 Aug; 9(31):1–9. Crossref
Conference on Computational Linguistic, Beijing, China:
2010. p. 1–9.

6 Vol 10 (17) | May 2017 | www.indjst.org Indian Journal of Science and Technology