CSE 2027-Fundamental of Data Analysis

Module 1-Introduction to Data Analysis

Introducing Data, overview of data analysis: Data

in the Real World, Data vs. Information, Many “Vs”
of Data, Structured Data and Unstructured Data,
Types of Data, Data Analysis Defined, Types of
Variables, Central Tendency of Data, Scales of Data,
Sources of Data, Data preparation: Cleaning the
data, Removing variables, Data Transformations.
 Introducing Data
• Facts and statistics collected together for reference or analysis

• Data has to be transformed into a form that is efficient for movement or

processing.

2
Over view of Data
Analysis
• Data analysis is defined as a process of cleaning,
transforming, and modeling data to discover
useful information for business decision-making.
• The purpose of Data Analysis is to extract useful
information from data and taking the decision
based upon the data analysis.
• A simple example of Data analysis is whenever
we take any decision in our day-to-day life is by
thinking about what happened last time or what
will happen by choosing that particular decision.

4
• This is nothing but analyzing our past or future
and making decisions based on it.
• For that, we gather memories of our past or
dreams of our future.
• So that is nothing but data analysis. Now same
thing analyst does for business purposes, is
called Data Analysis.

5
Data Analysis Tools

6
Data in the Real World

7
8
Many v’s of Data
 A. Volume
• The term Volume is meant for the Magnitude or Scale of data.

• Massive amounts of data generated from multiple resources are not

possible to handle through the traditional ways like a database.

• This large volume data is a composition of multiple data types, which is

unstructured in nature.
• This kind of data can be either in the form of audio, video, tweets, likes
etc.
 B. Velocity
• Velocity refers to the speed at which the gigantic amount of data is
being generated, collected and scrutinized.

• With every flip of second, data is being searched on the internet.

• On a day to day basis, social networking sites like Facebook, Twitter,

Linkedln etc, are sharing a large amount of data.

• For easy analysis of this high amount of constantly generating data

with keeping an eye on it speed and easy access.
 C. Variety
• In terms of Big data, term Variety of data pretends to be a composition of
structured and unstructured kind of data.

• The data collected from different sources like mobile phones, laptops etc is not
homogenous in nature.

• Apart from text, audio ,video files, there may be some log files ,clicks or likes or
dislikes etc.
 D. Value
• Value refers to convert our investigated data into values.
• Value is one of the most important characteristics of Big data with
a composition of collection and analyzing the same in order to
boost the performance of any organization along with a better
understanding of customers.
• With the access to this useful data, one must analyze great values
in order to get amazing benefits.
 E. Variability
• Variability refers to unpredictable changes in the data.

• It may happen because of multiple data types & the speed with
which data is generating and being loaded into the database.
 F. Veracity
• Veracity refers to the term trustworthiness with reference to accurate data.

• If the data is accurate, only then you could think of meaningful data.

• For example, consider a dataset of thirty students on which we have to make an

analysis about the reason they got distinction.

• Being an analyzer, you can ask questions like:

• what are the methodology you adopted to get good marks in all the subjects?
• How much time you devote to individual subject?

• Do you learn some subjects with the help of daily life activities like sports etc?

• Have you ever been a scholar?

• Be getting answers like this it would be easier to determine the accuracy of

information which could easily be maintained in statistical form.
 G. Validity
• Two terms of big data veracity and validity seems to be alike but
are quite different.

• validity is meant for an accurate analysis in order to get optimized

results.
 H. Vulnerability
• Vulnerability is one of the major challenge in big data as the data
generated from multiple sources with such an erratic speed has high
chances of being harmed by any intruder.

• Currently, in a case of facebook, where the Belgium court has

threatened to fine a high amount on breaking privacy recently.
 I. Volatility
• Volatility refers to how long the perceived data remains to be
useful for us and how it is to be kept.

• For analyzing the same, it is necessary to develop some new rules

and techniques through which rapid access to information is
possible.
 J. Visualization
• Data Visualization is one of the most complex challenge in big data.

• In this information age, data is not only going beyond the limits but also
is composed of different data types.

• So, there is a need of communicate the information by visualizing it

through some special ways with special functionalities like a web-based
approach, statistical analysis etc.
• Traditional tools of data visualization face severe challenges
like low response time, complex methods of scalability,
precision in reporting time etc.

• So, it is a challenge to work with the concept which way of

communication with data is most suitable in order to make
visualization more effective.
22
 Typical human-generated unstructured
data includes
• Text files: Word processing, spreadsheets, presentations, email, logs.
• Email: Email has some internal structure thanks to its metadata, and we sometimes refer to
it as semi-structured. However, its message field is unstructured and traditional analytics
tools cannot parse it.
• Social Media: Data from Facebook, Twitter, LinkedIn.
• Website: YouTube, Instagram, photo sharing sites.
• Mobile data: Text messages, locations.
• Communications: Chat, IM, phone recordings, collaboration software.
• Media: MP3, digital photos, audio and video files.
• Business applications: MS Office documents, productivity applications

23
 Typical machine-generated unstructured
data includes:
• Satellite imagery: Weather data, land forms, military movements.

• Scientific data: Oil and gas exploration, space exploration, seismic imagery,
atmospheric data.

• Digital surveillance: Surveillance photos and video.

• Sensor data: Traffic, weather, oceanographic sensors.

24
25
26
27
Types of Digital Data

28
Data Analysis-Types
• There are several types of Data Analysis techniques that
exist based on business and technology. However, the
major Data Analysis methods are:
• Text Analysis
• Statistical Analysis
• Diagnostic Analysis
• Predictive Analysis
• Prescriptive Analysis

29
30
Text Analysis
• Text Analysis is also referred to as Data Mining. It
is one of the methods of data analysis to discover
a pattern in large data sets using databases or
data mining tools.
• It used to transform raw data into business
information. Business Intelligence tools are
present in the market which is used to take
strategic business decisions. Overall it offers a
way to extract and examine data and deriving
patterns and finally interpretation of the data.

31
 Statistical Analysis

• Statistical Analysis shows "What happen?" by

using past data in the form of dashboards.
Statistical Analysis includes collection, Analysis,
interpretation, presentation, and modeling of
data. It analyses a set of data or a sample of data.
• There are two categories of this type of Analysis -
Descriptive Analysis
Inferential Analysis.

32
Descriptive Analysis
• Analyses complete data or a sample of
summarized numerical data. It shows mean and
deviation for continuous data whereas
percentage and frequency for categorical data.
Inferential Analysis
• Analyses sample from complete data. In this type
of Analysis, you can find different conclusions
from the same data by selecting different
samples.

33
Diagnostic Analysis
• Diagnostic Analysis shows "Why did it happen?"
by finding the cause from the insight found in
Statistical Analysis. This Analysis is useful to
identify behavior patterns of data. If a new
problem arrives in your business process, then
you can look into this Analysis to find similar
patterns of that problem. And it may have
chances to use similar prescriptions for the new
problems.

34
Predictive Analysis
• Predictive Analysis shows "what is likely to happen" by using
previous data. The simplest data analysis example is like if last
year I bought two dresses based on my savings and if this year
my salary is increasing double then I can buy four dresses. But
of course it's not easy like this because you have to think about
other circumstances like chances of prices of clothes is
increased this year or maybe instead of dresses you want to
buy a new bike, or you need to buy a house!
• So here, this Analysis makes predictions about future
outcomes based on current or past data. Forecasting is just an
estimate. Its accuracy is based on how much detailed
information you have and how much you dig in it.

35
Prescriptive Analysis
• Prescriptive Analysis combines the insight from
all previous Analysis to determine which action
to take in a current problem or decision. Most
data-driven companies are utilizing Prescriptive
Analysis because predictive and descriptive
Analysis are not enough to improve data
performance. Based on current situations and
problems, they analyze the data and make
decisions.

36
Types of Variable
 Categorical (qualitative) variables have values that
can only be placed into categories, such as “yes”
and “no.”

 Numerical (quantitative) variables have values that

represent quantities.
 Discrete variables arise from a counting process
 Continuous variables arise from a measuring process

37
Types of Variables
Variables

Categorical Numerical

Examples:
 Marital Status Discrete Continuous
 Political Party
 Eye Color
Examples: Examples:
(Defined categories)
 Number of Children  Weight
 Defects per hour  Voltage
(Counted items) (Measured characteristics)

Copyright ©2011
2-38
Pearson Education
Central Tendency-Mode
• The mode is the most commonly reported value
for a particular variable.
• It is illustrated using the following variable
whose values are: 3, 4, 5, 6, 7, 7, 7, 8,8,9
• The mode would be the value 7 since there are
three occurrences of 7 (more than any other
value).
• The following values, both 7 and 8 are reported
three times: 3, 4, 5, 6, 7,7, 7, 8, 8, 8, 9 The mode
may be reported as {7, 8} or 7.5.

39
Median
• The median is the middle value of a variable
once it has been sorted from low to high. For
variables with an even number of values, the
mean of the two values closest to the middle is
selected (sum the two values and divide by 2).
• The following set of values will be used to
illustrate: 3, 4, 7, 2, 3, 7,4, 2, 4, 7, 4.
• Before identifying the median, the values must
be sorted: 2, 2, 3, 3, 4, 4, 4, 4, 7,7, 7

40
Mean

41
Source of Data
• Surveys or polls
A survey or poll can be useful for gathering data to answer
specific questions
• Experiments:
Experiments measure and collect data to answer a specific
question in a highly controlled manner. The data collected
should be reliably measured, that is, repeating the
measurement should not result in different values.
Experiments attempt to understand cause and affect
phenomena by controlling other factors that may be
important.

42
• Observational and other studies: In certain
situations it is impossible on either logistical or
ethical grounds to conduct a controlled
experiment. In these situations, a large number
of observations are measured and care taken
when interpreting the results.
• Operational databases: These databases contain
ongoing business transactions. They are accessed
constantly and updated regularly. Examples
include supply chain management systems,
customer relationship management (CRM)
databases and manufacturing production
databases.

43
• Data warehouses: A data warehouse is a copy of data
gathered from other sources within an organization that
has been cleaned, normalized, and optimized for making
decisions. It is not updated as frequently as operational
databases.
• Historical databases: Databases are often used to house
historical polls, surveys and experiments.
• Purchased data: In many cases data from in-house
sources may not be sufficient to answer the questions
now being asked of it. One approach is to combine this
internal data with data from other sources.

44
Scales of Data
• Nominal: Scale describing a variable with a
limited number of different values. This scale is
made up of the list of possible values that the
variable may take. It is not possible to determine
whether one value is larger than another.
• Ordinal: This scale describes a variable whose
values are ordered; however, the difference
between the values does not describe the
magnitude of the actual difference.

45
• Interval: Scales that describe values where the
interval between the values has meaning.
• Ratio: Scales that describe variables where the
same difference between values has the same
meaning (as in interval) but where a double,
tripling, etc. of the values implies a double,
tripling, etc. of the measurement.

46
Table

47
Cleaning the Data
• Since the data available for analysis may not have been
originally collected with this project’s goal in mind, it is
important to spend time cleaning the data.
• It is also beneficial to understand the accuracy with
which the data was collected as well as correcting any
errors.
• For variables measured on a nominal or ordinal scale
(where there are a fixed number of possible values), it is
useful to inspect all possible values to uncover mistakes
and/or inconsistencies.
• Any assumptions made concerning possible values that
the variable can take should be tested.

48
• For example, a variable Company may include a
number of different spellings for the same
company such as:
• General Electric Company
• General Elec. Co
• GE
• Gen. Electric Company
• General electric company
• G.E. Company

49
• These different terms, where they refer to the
same company, should be consolidated into one
for analysis.
• In addition, subject matter expertise may be
needed in cleaning these variables.
• For example, a company name may include one
of the divisions of the General Electric Company
and for the purpose of this specific project it
should be included as the ‘‘General Electric
Company.’’

50
Removing Variables
• On the basis of an initial categorization of the
variables, it may be possible to remove variables
from consideration at this point.
• For example, constants and variables with too
many missing data points should be considered
for removal.
• Further analysis of the correlations between
multiple variables may identify variables that
provide no additional information to the analysis
and hence could be removed.

51
Data Transformation
Normalization
• Normalization is a process where numeric columns are transformed
using a mathematical function to a new range. It is important for two
reasons.
• First, analysis of the data should treat all variables equally so that
one column does not have more influence over another because the
ranges are different.
• For example, when analyzing customer credit card data, the Credit
limit value is not given more weightage in the analysis than the
Customer’s age.
• Second, certain data analysis and data mining methods require the
data to be normalized prior to analysis, such as neural networks or k-
nearest neighbors

52
Problem

53
Solution

54