International Journal of Engineering and Technical Research (IJETR)

ISSN: 2321-0869 (O) 2454-4698 (P), Volume-7, Issue-4, April 2017

An Approach to Analysis and Classification of Data

from Big Data by Using Apriori Algorithm
Mr. Nagesh Sharma, Mr. Ram Kumar Sharma

Abstract A small amount of data can be easy to manage and specific or general and private or public. Big data are
straightforward analysis can be gleaned from it. After all, characterized by 3 Vs: Volume, Velocity, and Variety.
theres only so much data to consider. But when data gets big, Volume -the size of data now is larger than terabytes and peta
big problems can arise. Most companies are swimming in more
data than they know what to do with. Unfortunately, too many bytes. The large scale and rise of size makes it difficult to
of them associate that drowning phenomenon with big data store and analyze using traditional tools.
itself. Big data has rapidly developed into a hot topic that Velocity big data should be used to mine large amount of
attracts extensive attention from academia, industry, and data within a pre defined period of time. The traditional
governments around the world. Data has become an methods of mining may take huge time to mine such a volume
indispensable part of every economy, industry, organization,
of data.
business function and individual. Big Data is a term used to
identify the datasets that whose size is beyond the ability of Variety Big data comes from a variety of sources which
typical database software tools to store, manage and analyze. includes both structured and unstructured data. Traditional
The Big Data introduce unique computational and statistical database systems were designed to address smaller volumes
challenges, including scalability and storage bottleneck, noise of structured and consistent data whereas Big Data is
accumulation, spurious correlation and measurement errors. geospatial data, 3D data, audio and video, and unstructured
Enormous amount of data are generated every minute.
text, including log files and social media. This heterogeneity
Index Terms Apriori Algoritthm, EMV, HIVE, of unstructured data creates problems for storage, mining and
Heterogeneity KDD, MangoDB, NoSQL, PIG, C4.5, SLIQ, analyzing the data.
Hadoop, Volume, Velocity, Variety New technologies are required to store unstructured large data
sets and processing methods such as Hadoop and Map Reduce
have greater importance in big data analysis. To process large
I. INTRODUCTION volumes of data from different sources quickly, Hadoop is
Data is the collection of values and variables related in some used. Hadoop is a free, Java-based programming framework
sense and differing in some other sense. In recent years the that supports the processing of large data sets in a distributed
sizes of databases have increased rapidly. This has lead to a computing environment.
growing interest in the development of tools capable in the
automatic extraction of knowledge from data. II. ISSUES AND CHALLENGES
Data are collected and analyzed to create information suitable Big Data: Emerging Challenges of Big Data and Techniques
for making decisions. Hence data provide a rich resource for for handling Big data analysis is the process of applying
knowledge discovery and decision support. A database is an advanced analytics and visualization techniques to large data
organized collection of data so that it can easily be accessed, sets to uncover hidden patterns and unknown correlations for
managed, and updated. Data mining is the process effective decision making. Big Data Data comes mainly in
discovering interesting knowledge such as associations, two forms- 1. Structured Data 2. Unstructured Data (there
patterns, changes, anomalies and significant structures from are also semi-structured data eg. XML) structured data
large amounts of data stored in databases, data warehouses or has semantic meaning attached to it whereas unstructured data
other information repositories. has no latent meaning. The growth in data that we are
The extracted knowledge is very useful and the mined referring is most unstructured data. Below are few examples
knowledge is the representation of different types of patterns of unstructured data.
and each pattern corresponds to knowledge. Data mining The analysis of Big Data involves multiple distinct phases
(DM), also called Knowledge Discovery in Databases (KDD) which include data acquisition and recording, information
or Knowledge Discovery and Data Mining, is the process of extraction and cleaning, data integration, aggregation and
searching large volumes of data automatically for patterns representation, query processing, data modeling and analysis
such as association rules and Interpretation. Each of these phases introduces
Big data refers to the use of large data sets to handle the challenges. Heterogeneity, scale, timeliness, complexity and
collection or reporting of data that serves businesses or other privacy are certain challenges of big data mining.
recipients in decision making. The data may be enterprise 2.1 Heterogeneity and Incompleteness
The difficulties of big data analysis derive from its large scale
as well as the presence of mixed data based on different
Mr. Nagesh Sharma, Assistant Professor, Noida Institute of patterns or rules (heterogeneous mixture data) in the collected
Engineering and Technology Greater Noida (201306) and stored data. In the case of complicated heterogeneous
Mr. Ram Kumar Sharma, Assistant Professor, Noida Institute of
Engineering and Technology Greater Noida (201306) mixture data, the data has several patterns and rules and the

10 www.erpublication.org
 An Approach to Analysis and Classification of Data from Big Data by Using Apriori Algorithm

properties of the patterns vary greatly. Data can be both insights from large repositories of unstructured and structured
structured and unstructured. 70% of the data generated by data that resides in multiple sources such as file systems,
organizations are unstructured. databases, streams, APIs, and other platforms and
2.2. Scale and complexity applications.
Managing large and rapidly increasing volumes of data is a iv. Stream analytics: software that can filter, aggregate,
challenging issue. Traditional software tools are not enough enrich, and analyze a high throughput of data from multiple
for managing the increasing volumes of data. Data analysis, disparate live data sources and in any data format.
Computer Science & organization, retrieval and modeling are v. In-memory data fabric: provides low-latency access and
also challenges due to scalability and complexity of data that processing of large quantities of data by distributing data
needs to be analysed. across the dynamic random access memory (DRAM), Flash,
2.3 Timeliness or SSD of a distributed computer system.
The flip side of size is speed. The larger the data set to be vi. Distributed file stores: a computer network where data is
processed, the longer it will take to analyze. The design of a stored on more than one node, often in a replicated fashion,
system that effectively deals with size is likely also to result in for redundancy and performance.
a system that can process a given size of data set faster. vii. Data virtualization: a technology that delivers
However, it is not just this speed that is usually meant when information from various data sources, including big data
one speaks of Velocity in the context of Big Data. As the size sources such as Hadoop and distributed data stores in
of the data sets to be processed increases, it will take more real-time and near-real time.
time to analyse. In some situations results of the analysis is viii. Data integration: tools for data orchestration across
required immediately. solutions such as Amazon Elastic MapReduce (EMR),
2.4 Privacy Apache Hive, Apache Pig, Apache Spark, MapReduce,
The privacy of data is another huge concern, and one that Couchbase, Hadoop, and MongoDB.
increases in the context of Big Data. However, there is great ix. Data preparation: software that eases the burden of
public fear regarding the inappropriate use of personal data, sourcing, shaping, cleansing, and sharing diverse and messy
particularly through linking of data from multiple sources. data sets to accelerate datas usefulness for analytics.
x. Data quality: products that conduct data cleansing and
Heterogeneity enrichment on large, high-velocity data sets, using parallel
operations on distributed data stores and databases.
So, Various Algorithms are used to analysis and classification
of data for above parameter.
Scale
Privacy Big A. ID3 algorithm
Data
The ID3 algorithm (Quinlan86) is a decision tree building
algorithm which determines the classification of objects by
testing the values of the their properties. It builds the tree in a
Timeliness top down fashion, starting from a set of objects and a
specification of properties. At each node of the tree, a
Fig 1 .Challenges and Issues of Big Data property is tested and the results used to partition the object
set. This process is recursively done till the set in a given
subtree is homogeneous with respect to the classification
criteria - in other words it contains objects belonging to the
III. III PROPOSED ALGORITHM FOR BIG DATA same category. This then becomes a leaf node. At each node,
CLASSIFICATION AND ANALYSIS the property to test is chosen based on information theoretic
criteria that seek to maximize information gain and minimize
There are many techniques available for data management. entropy. In simpler terms, that property is tested which
The Big Data handling techniques and tools include Hadoop, divides the candidate set in the most homogeneous subsets.
MapReduce, Simple DB, Google BigTable, Not Only SQL B. C4.5 algorithm
(NoSQL), Data Stream Management System (DSMS),
MemcacheDB, and Voldemort. Out of these, Hadoop is one This algorithm was proposed by Quinlan (1993). The C4.5
of the most widely used technologies. algorithm generates a classification-decision tree for the given
i. Predictive analytics: software and/or hardware solutions data-set by recursive partitioning of data. The decision is
that allow firms to discover, evaluate, optimize, and deploy grown using Depth-first strategy. The algorithm considers
predictive models by analyzing big data sources to improve all the possible tests that can split the data set and selects a test
business performance or mitigate risk. that gives the best information gain. For each discrete
ii. NoSQL databases: key-value, document, and graph attribute, one test with outcomes as many as the number of
databases. distinct values of the attribute is considered. For each
iii. Search and knowledge discovery: tools and technologies continuous attribute, binary tests involving every distinct
to support self-service extraction of information and new values of the attribute are considered. In order to gather the

11 www.erpublication.org
 International Journal of Engineering and Technical Research (IJETR)
ISSN: 2321-0869 (O) 2454-4698 (P), Volume-7, Issue-4, April 2017
entropy gain of all these binary tests efficiently, the training { for all candidates c Ck contained int do
data set belonging to the node in consideration is sorted for c:count++ }
the values of the continuous attribute and the entropy gains of Lk = { c Ck | c:count >= min-support}
the binary cut based on each distinct values are calculated in }
one scan of the sorted data. This process is repeated for each Answer := k Lk
continuous attributes. end
It makes multiple passes over the database. In the first pass,
C. SLIQ algorithm
the algorithm simply counts item occurrences to determine the
frequent 1-itemsets (itemsets with 1 item). A subsequent pass,
SLIQ (Supervised Learning In Quest) developed by IBM's say pass k, consists of two phases. First, the frequent itemsets
Quest project team, is a decision tree classifier designed to Lk-1 (the set of all frequent (k-1)-itemsets) found in the (k-1)th
classify large training data [1]. It uses a pre-sorting technique pass are used to generate the candidate itemsets Ck, using the
in the tree-growth phase. This helps avoid costly sorting at apriori-gen() function. This function first joins Lk-1 with Lk-1,
each node. SLIQ keeps a separate sorted list for each the joining condition being that the lexicographically ordered
continuous attribute and a separate list called class list. An first k-2 items are the same. Next, it deletes all those itemsets
entry in the class list corresponds to a data item, and has a from the join result that have some (k-1)-subset that is not in
class label and name of the node it belongs in the decision Lk-1 yielding Ck.Frequent itemset generation after
tree. An entry in the sorted attribute list has an attribute value classification of the Apriori algorithm
and the index of data item in the class list. SLIQ grows the k =1
decision tree in breadth-first manner. For each attribute, it Lk = {i|i I ({i}) N minsup}.
scans the corresponding sorted list and calculates entropy {Find all frequent 1-itemsets}
values of each distinct values of all the nodes in the frontier of repeat k=k+1.
the decision tree simultaneously. After the entropy values Ck = apriori-gen(Lk-1) ).
have been calculated for each attribute, one attribute is chosen {Generate candidate itemsets}
for a split for each nodes in the current frontier, and they are for each transection t T do
expanded to have a new frontier. Then one more scan of the Ct = subset( Ck , t )
sorted attribute list is performed to update the class list for the {Identify all candidates that belong to t}
new nodes. for each candidate itemset c Ct do
While SLIQ handles disk-resident data that are too large to fit (c) = (c)+1.
in memory, it still requires some information to stay {Increment support count}
memory-resident which grows in direct proportion to the end for
number of input records, putting a hard-limit on the size of end for Lk = {c|c Ck (c) N x minsup
training data. The Quest team has recently designed a new {Extract the frequent kitemsets}
decision-tree-based classification algorithm, called SPRINT until Lk =
(Scalable Parallelizable Induction of decision Trees) that for Result = Lk
the removes all of the memory restrictions. IV Comparison and Result Analysis
D. Apriori Algorithm .There are currently 4 kinds of classical algorithm, frequently
An association rule mining algorithm, Apriori has been of use is the most widely used of these two algorithm, the
developed for rule mining in large transaction databases. following two algorithms will be compared With above given
A itemset is a non-empty set of items. 10 parameters :-
They have decomposed the problem of mining association Comparison
Algorithm Application Processing Aims
rules into two parts
Name
Find all combinations of items that have transaction
Search the
support above minimum support. Call those combinations database to
frequent itemsets. Association Frequent obtain
Use the frequent item sets to generate the desired Apriori
rule item set candidate set
rules. The general idea is that if, say, ABCD and AB are of suppose
frequent item sets, then we can determine if the rule AB CD item sets
holds by computing the ratio r = Machine
support(ABCD)/support(AB). The rule holds only if r >= Learning and Supervised
C4.5 Tuple
minimum confidence. Note that the rule will have minimum data mining learning
support because ABCD is frequent. The Apriori algorithm classification
used in Quest for finding all frequent item sets is given below Table1.The Comparison of Algorithms
Procedure AprioriAlg() The Fig. 2 shows the all above parameter it would appear from
begin this graph and comparison between to algorithm Apriori and
L1 :={frequent1-itemsets}; C4.5 that are used in the Big Data Analytics and
for ( k := 2; Lk-1 0; k++ ) do classification.
{ Ck= apriori-gen(Lk-1) ; // new candidates
for all transactions t in the dataset do

12 www.erpublication.org
 An Approach to Analysis and Classification of Data from Big Data by Using Apriori Algorithm

Nagesh Sharma, Department Name Information Technology, Noida

Institute of Engineering & Technology Greater Noida
MobileNo.-9999100436,(e mail:iamnageshsharma@gmail.com

Ram Kumar Sharma, Department Name - Information

Technology, Noida Institute of Engineering & Technology Greater Noida
MobileNo.-9654624322

Fig 2 .Comparison Graph of Apriori and C4.5 Algorithm

V. CONCLUSION
These concepts include Big Data characteristics, challenges
and techniques for handling big data and Big Data Mining.
Big data discloses the limitations of existing data mining
techniques, resulted in a series of new challenges related to
big data mining. We are show the comparison of two different
algorithm for classifying and analysis of the data with above
ten parameters and find the Apriori algorithm is best on any
data itemset.

VI. FUTURE SCOPE

As there are huge volumes of data that are produced every
day, so such large size of data it becomes very challenging to
achieve effective processing using the existing traditional
techniques. Big data is data that exceeds the processing
capacity of conventional database systems. In this paper
fundamental concepts about Big Data are presented. Apriori
Algorithm is implemented with Map reduce for Classification
of Structured, semi structured and unstructured data. We can
also implement Apriori Algorithm for analysis of terabyte
data with frequent itemset.

REFERENCE
[1].Jianwei DaiZhaolin WuMingdong ZhuJianhua Gonget al.Data
engineering theory and technology.National defence industry press
2010p.180-220.
[2]. Changqing Ji, Yu Li, Wenming Qiu, Uchechukwu Awada, Keqiu Li,
Big Data Processing in Cloud Computing Environments,
International Symposium on Pervasive Systems, Algorithms and
Networks,2012
[3].Mukherjee, A.; Datta, J.; Jorapur, R.; Singhvi, R.; Haloi, S.; Akram, W.,
(18-22 Dec.,2012) , Shared disk big data analytics with Apache
Hadoop
[4].S.Vikram Phaneendra and E.Madhusudhan Reddy, Big Data-
solutions for RDBMS problems- A survey, IEEE/IFIP Network
Operations & Management Symposium (NOMS 2010),Osaka Japan,
Apr 19-23 2013.
[5]. Ashish R. Jagdale, Kavita V. Sonawane & Shamsuddin S. Khan, Data
Mining and Data Pre-processing for Big Data, Vol 5, Issue 7, July
2014.
[6]. M. Chen, S. Mao, and Y. Liu, Big data: a survey, Mobile Networks and
Applications, vol. 19, no. 2, pp. 171209, 2014.
[7].V.Jude Nirmal and D.I. George Amalarethinam, Parallel Implementation
of Big Data Pre-Processing Algorithms for Sentiment Analysis of Social
Networking Data, IJFMA Vol. 6, No. 2, 2015, 149-159, ISSN: 2320
3242 (P), 2320 3250 (online)Published on22 January 2015.

13 www.erpublication.org