31/01/2021

Lecture 2:
Getting to Know Your Data
Lecturer: Dr. Nguyen, Thi Thanh Sang
(nttsang@hcmiu.edu.vn)

References:
Chapter 2 in Data Mining: Concepts and Techniques (Third Edition), by Jiawei
Han, Micheline Kamber

1
 31/01/2021

Getting to Know Your Data

 Data Objects and Attribute Types

 Basic Statistical Descriptions of Data

 Measuring Data Similarity and Dissimilarity

 Summary

Types of Data Sets

 Record

 Relational records

 Data matrix, e.g., numerical matrix, crosstabs

 Document data: text documents: term-frequency vector

 Transaction data

 Graph and network

 World Wide Web

 Social or information networks

 Molecular Structures

 Ordered

 Video data: sequence of images TID Items

 Temporal data: time-series 1 Bread, Coke, Milk

 Sequential Data: transaction sequences 2 Beer, Bread
 Genetic sequence data 3 Beer, Coke, Diaper, Milk
 Spatial, image and multimedia:
4 Beer, Bread, Diaper, Milk
 Spatial data: maps
5 Coke, Diaper, Milk

 Image data:

 Video data: 4

2
 31/01/2021

Important Characteristics of Structured Data

 Dimensionality
 Curse of dimensionality
 Sparsity

 Only presence counts

 Resolution

 Patterns depend on the scale

 Distribution
 Centrality and dispersion
5

Data Objects

 Data sets are made up of data objects.

 A data object represents an entity.
 Examples:
 sales database: customers, store items, sales
 medical database: patients, treatments
 university database: students, professors, courses
 Also called samples , examples, instances, data points, objects,
tuples.
 Data objects are described by attributes.
 Database rows -> data objects; columns ->attributes.

3
 31/01/2021

Attributes

 Attribute (or dimensions, features, variables): a data

field, representing a characteristic or feature of a data
object.
 E.g., customer _ID, name, address
 Types:
 Nominal
 Binary
 Numeric: quantitative
 Interval-scaled
 Ratio-scaled

Attribute Types

 Nominal: categories, states, or “names of things”

 Hair_color = {auburn, black, blond, brown, grey, red, white}
 marital status, occupation, ID numbers, zip codes
 Binary
 Nominal attribute with only 2 states (0 and 1)
 Symmetric binary: both outcomes equally important
 e.g., gender
 Asymmetric binary: outcomes not equally important.
 e.g., medical test (positive vs. negative)
 Convention: assign 1 to most important outcome (e.g., HIV
positive)
 Ordinal
 Values have a meaningful order (ranking) but magnitude between
successive values is not known.
 Size = {small, medium, large}, grades, army rankings

4
 31/01/2021

Numeric Attribute Types

 Quantity (integer or real-valued)

 Interval
 Measured on a scale of equal-sized units
 Values have order
• E.g., temperature in C˚or F˚, calendar dates
 No true zero-point
 Ratio
 Inherent zero-point
 We can speak of values as being an order of magnitude
larger than the unit of measurement (10 K˚ is twice as high
as 5 K˚).
• e.g., temperature in Kelvin, length, counts, monetary
quantities
9

Discrete vs. Continuous Attributes

 Discrete Attribute
 Has only a finite or countably infinite set of values
 E.g., zip codes, profession, or the set of words in a collection of documents

 Sometimes, represented as integer variables

 Note: Binary attributes are a special case of discrete
attributes
 Continuous Attribute
 Has real numbers as attribute values
 E.g., temperature, height, or weight

 Practically,real values can only be measured and

represented using a finite number of digits
 Continuous attributes are typically represented as
floating-point variables
10

10

5
 31/01/2021

Activities

 Identify attribute types in datasets

11

Getting to Know Your Data

 Data Objects and Attribute Types

 Basic Statistical Descriptions of Data

 Measuring Data Similarity and Dissimilarity

 Summary

12

12

6
 31/01/2021

Basic Statistical Descriptions of Data

 Motivation
 To
better understand the data: central tendency,
variation and spread
 Data dispersion characteristics
 median, max, min, quantiles, outliers, variance, etc.
 Numerical dimensions correspond to sorted intervals
 Data dispersion: analyzed with multiple granularities
of precision
 Boxplot or quantile analysis on sorted intervals
 Dispersion analysis on computed measures
 Folding measures into numerical dimensions
 Boxplot or quantile analysis on the transformed cube
13

13

Measuring the Central Tendency

n
1
 Mean (algebraic measure) (sample vs. population): x 
n

i 1
xi
Note: n is sample size and N is population size. n

wx i i

 Weighted arithmetic mean: x  i 1

n

w

i
x
 Trimmed mean: chopping extreme values i 1

N
 Example:
 Suppose we have the following values for salary (in
thousands of dollars), shown in increasing order: 30,
36, 47, 50, 52, 52, 56, 60, 63, 70, 70, 110.

 Mean = ?

14

14

7
 31/01/2021

Measuring the Central Tendency (cont.)

 Median:
 Middle value if odd number of values, or average of the middle two values
otherwise
 Estimated by interpolation (for grouped data):
n / 2  ( freq )l
median  L1  ( ) width
freqmedian
 Mode Median
interval

 Value that occurs most frequently in the data

 Unimodal, bimodal, trimodal

 Empirical formula:

mean  mode  3  (mean  median)

15

Measuring the Central Tendency (cont.)

 Example:
 Suppose we have the following values for salary (in thousands of dollars),
shown in increasing order: 30, 36, 47, 50, 52, 52, 56, 60, 63, 70, 70, 110.

 Median = ?

 Mode = ?

16

8
 31/01/2021

Symmetric vs. Skewed Data

 Median, mean and mode of symmetric,

positively and negatively skewed data symmetric

positively skewed negatively skewed

17

January 31, 2021 Data Mining: Concepts and Techniques

17

Measuring the Dispersion of Data

 Quartiles, outliers and boxplots

 Quartiles: Q1 (25th percentile), Q3 (75th percentile)
 Inter-quartile range: IQR = Q3 – Q1
 Five number summary: min, Q1, median, Q3, max
 Boxplot: ends of the box are the quartiles; median is marked; add whiskers, and
plot outliers individually
 Outlier: usually, a value higher/lower than 1.5 x IQR beyond the quartiles.
 Variance and standard deviation (σ)
 Variance: (algebraic, scalable computation)

 Standard deviation σ is the square root of variance σ2

18

18

9
 31/01/2021

Boxplot Analysis

 Five-number summary of a distribution

 Minimum, Q1, Median, Q3, Maximum
 Boxplot
 Data is represented with a box
 The ends of the box are at the first and third
quartiles, i.e., the height of the box is IQR
 The median is marked by a line within the box
 Whiskers: two lines outside the box extended
to Minimum and Maximum
 Outliers: points beyond a specified outlier
threshold, plotted individually
19

19

Fig. 2.3. Boxplot for the unit price data for items sold at four
branches of AllElectronics during a given time period.

For branch 1, Median = $80, Q1 is $60, and Q3 =

Max $100. Notice that two outlying observations for this
branch were plotted individually, as their values of
175 and 202 are more than 1.5 times the IQR here of
40.
Q3

Median

Q1

Min

20

10
 31/01/2021

Measuring the Dispersion of Data

 Example:
 Suppose we have the following values for salary (in thousands of
dollars), shown in increasing order: 30, 36, 47, 50, 52, 52, 56, 60,
63, 70, 70, 110.

 Q1 = ?; Q3 = ?

 IQR = ?

 Variance 2 = ?; Standard deviation  = ?

21

Visualization of Data Dispersion: 3-D Boxplots

22

22

11
 31/01/2021

Properties of Normal Distribution Curve

 The normal (distribution) curve

 From μ–σ to μ+σ: contains about 68% of the measurements
(μ: mean, σ: standard deviation)
 From μ–2σ to μ+2σ: contains about 95% of it
 From μ–3σ to μ+3σ: contains about 99.7% of it

23

23

Graphic Displays of Basic Statistical Descriptions

 Boxplot: graphic display of five-number summary

 Histogram: x-axis are values, y-axis repres. frequencies

 Quantile plot: each value xi is paired with fi indicating that

approximately 100 fi % of data are  xi

 Quantile-quantile (q-q) plot: graphs the quantiles of one univariant

distribution against the corresponding quantiles of another

 Scatter plot: each pair of values is a pair of coordinates and plotted

as points in the plane

24

24

12
 31/01/2021

Histogram Analysis

 Histogram: Graph display of tabulated

frequencies, shown as bars 40
 It shows what proportion of cases fall 35
into each of several categories 30
 Differs from a bar chart in that it is the 25
area of the bar that denotes the 20
value, not the height as in bar charts,
15
a crucial distinction when the
10
categories are not of uniform width
5
 The categories are usually specified as
0
non-overlapping intervals of some 10000 30000 50000 70000 90000
variable. The categories (bars) must
be adjacent
25

25

Quantile Plot
 Displays all of the data (allowing the user to assess both
the overall behavior and unusual occurrences)
 Plots quantile information
 For a data xi data sorted in increasing order, fi
indicates that approximately 100 fi% of the data are
below or equal to the value xi

26

26

13
 31/01/2021

Quantile-Quantile (Q-Q) Plot

 Graphs the quantiles of one univariate distribution against the

corresponding quantiles of another
 View: Is there is a shift in going from one distribution to another?
 Example shows unit price of items sold at Branch 1 vs. Branch 2 for
each quantile. Unit prices of items sold at Branch 1 tend to be
lower than those at Branch 2.

27

27

Scatter plot

 Provides a first look at bivariate data to see clusters of

points, outliers, etc.
 Each pair of values is treated as a pair of coordinates
and plotted as points in the plane

28

28

14
 31/01/2021

Example: Histogram and Scatter

29

Positively and Negatively Correlated Data

 The left half fragment is positively

correlated

 The right half is negative correlated

30

30

15
 31/01/2021

Uncorrelated Data

31

31

Getting to Know Your Data

 Data Objects and Attribute Types

 Basic Statistical Descriptions of Data

 Measuring Data Similarity and Dissimilarity

 Summary

32

32

16
 31/01/2021

Similarity and Dissimilarity

 Similarity
 Numerical measure of how alike two data objects are
 Value is higher when objects are more alike
 Often falls in the range [0,1]
 Dissimilarity (e.g., distance)
 Numerical measure of how different two data objects
are
 Lower when objects are more alike
 Minimum dissimilarity is often 0
 Upper limit varies
 Proximity refers to a similarity or dissimilarity
33

33

Data Matrix and Dissimilarity Matrix

 Data matrix
 n data points with p  x 11 ... x 1f ... x 1p 
dimensions  
 ... ... ... ... ... 
 Two modes x ... x if ... x ip 
 i1 
 ... ... ... ... ... 
x ... x nf ... x np 
 n1 
 Dissimilarity matrix
 0 
 n data points, but  d(2,1) 
0
registers only the  
distance  d(3,1 ) d ( 3 ,2 ) 0 
 
A triangular matrix  : : : 
 d ( n ,1 ) d ( n,2 ) ... ... 0 
 Single mode
34

34

17
 31/01/2021

Proximity Measure for Nominal Attributes

 Can take 2 or more states, e.g., red, yellow, blue, green

(generalization of a binary attribute)

 Method: Simple matching

 m: # of matches, p: total # of variables

d ( i , j )  p p m

35

35

Proximity Measure for Binary Attributes

Object j
 A contingency table for binary data
Object i

 Distance measure for symmetric binary

variables:

 Distance measure for asymmetric

binary variables:

 Jaccard coefficient (similarity measure

for asymmetric binary variables):

 Note: Jaccard coefficient is the same as “coherence”:

36

36

18
 31/01/2021

Dissimilarity between Binary Variables

q t
 Example Asymmetric att.

Name Gender Fever Cough Test-1 Test-2 Test-3 Test-4

Jack M Y N P N N N
Mary F Y N P N P N
Jim M Y P N N N N
 Gender is a symmetric attribute
 The remaining attributes are asymmetric binary
 Let the values Y and P be 1, and the value N 0
0  1 based only on
d ( jack , mary )   0 . 33 the asymmetric
2  0  1
attributes.
1  1
d ( jack , jim )   0 . 67
1  1  1
1  2
d ( jim , mary )   0 . 75
1  1  2
37

37

Standardizing Numeric Data


z  x
 Z-score:

 X: raw score to be standardized, μ: mean of the population, σ: standard

deviation
 the distance between the raw score and the population mean in units of
the standard deviation
 negative when the raw score is below the mean, “+” when above
 An alternative way: Calculate the mean absolute deviation
s f  1n (| x1 f  m f |  | x2 f  m f | ... | xnf  m f |)
where
m f  1n (x1 f  x2 f  ...  xnf )
.

xif  m f
 standardized measure (z-score): zif  sf
 Using mean absolute deviation is more robust than using standard deviation

38

38

19
 31/01/2021

Example:
Data Matrix and Dissimilarity Matrix

x2 x4
Data Matrix
point attribute1 attribute2
4 x1 1 2
x2 3 5
x3 2 0
x4 4 5
2 x1
Dissimilarity Matrix
(with Euclidean Distance)
x3
0 2 4 x1 x2 x3 x4
x1 0
x2 3.61 0
x3 2.24 5.1 0
x4 4.24 1 5.39 0
39

39

Distance on Numeric Data: Minkowski Distance

 Minkowski distance: A popular distance measure

where i = (xi1, xi2, …, xip) and j = (xj1, xj2, …, xjp) are

two p-dimensional data objects, and h is the order
(the distance so defined is also called L-h norm)
 Properties
 d(i, j) > 0 if i ≠ j, and d(i, i) = 0 (Positive definiteness)
 d(i, j) = d(j, i) (Symmetry)
 d(i, j)  d(i, k) + d(k, j) (Triangle Inequality)
 A distance that satisfies these properties is a metric

40

40

20
 31/01/2021

Special Cases of Minkowski Distance

 h = 1: Manhattan (city block, L1 norm) distance
 E.g., the Hamming distance: the number of bits that are different
between two binary vectors
d (i, j) | x  x |  | x  x | ... | x  x |
i1 j1 i2 j 2 ip jp

 h = 2: (L2 norm) Euclidean distance

d (i, j)  (| x  x |2  | x  x |2 ... | x  x |2 )
i1 j1 i2 j 2 ip jp

 h  . “supremum” (Lmax norm, L norm) distance.

 This is the maximum difference between any component
(attribute) of the vectors

41

41

Example: Minkowski Distance

Dissimilarity Matrices
point attribute 1 attribute 2 Manhattan (L1)
x1 1 2
L x1 x2 x3 x4
x2 3 5 x1 0
x3 2 0 x2 5 0
x4 4 5 x3 3 6 0
x4 6 1 7 0
Euclidean (L2)
L2 x1 x2 x3 x4
x1 0
x2 3.61 0
x3 2.24 5.1 0
x4 4.24 1 5.39 0

Supremum
L x1 x2 x3 x4
x1 0
x2 3 0
x3 2 5 0
x4 3 1 5 0
42

42

21
 31/01/2021

Ordinal Variables

 An ordinal variable can be discrete or continuous

 Order is important, e.g., rank
 Can be treated like interval-scaled
 replace x by their rank rif  {1,..., M f }
if

 map the range of each variable onto [0, 1] by

replacing i-th object in the f-th variable by
r if  1
z if 
M f  1

 compute the dissimilarity using methods for interval-

scaled variables

43

43

Example: Ordinal variables

 Dissimilarity matrix:

44

22
 31/01/2021

Attributes of Mixed Type

 A database may contain all attribute types
 Nominal, symmetric binary, asymmetric binary, numeric,
ordinal
 One may use a weighted formula to combine their effects
 pf  1 ij( f ) dij( f )
d (i, j) 
 pf  1 ij( f )
f is binary or nominal:
dij(f) = 0 if xif = xjf , or dij(f) = 1 otherwise
f is numeric: use the normalized distance
f is ordinal
zif 
r 1
if
 Compute ranks rif and
M 1 f
 Treat zif as numeric

45

45

Example: Mixed Type

 Dissimilarity matrix:
 For test-3: ; for the data:

46

23
 31/01/2021

Cosine Similarity

 A document can be represented by thousands of attributes, each recording

the frequency of a particular word (such as keywords) or phrase in the
document.

 Other vector objects: gene features in micro-arrays, …

 Applications: information retrieval, biologic taxonomy, gene feature
mapping, ...
 Cosine measure: If d1 and d2 are two vectors (e.g., term-frequency
vectors), then
cos(d1, d2) = (d1  d2) /||d1|| ||d2|| ,
where  indicates vector dot product, ||d||: the length of vector d

47

47

Example: Cosine Similarity

 cos(d1, d2) = (d1  d2) /||d1|| ||d2|| ,

where  indicates vector dot product, ||d||: the length of vector d

 Ex: Find the similarity between documents 1 and 2.

d1 = (5, 0, 3, 0, 2, 0, 0, 2, 0, 0)
d2 = (3, 0, 2, 0, 1, 1, 0, 1, 0, 1)

d1d2 = 5*3+0*0+3*2+0*0+2*1+0*1+0*1+2*1+0*0+0*1 = 25
||d1||= (5*5+0*0+3*3+0*0+2*2+0*0+0*0+2*2+0*0+0*0)0.5=(42)0.5 = 6.481
||d2||= (3*3+0*0+2*2+0*0+1*1+1*1+0*0+1*1+0*0+1*1)0.5=(17)0.5 = 4.12
cos(d1, d2 ) = 0.94

48

48

24
 31/01/2021

Activities

 Do exercises in Text [1]: 2.2.a-f, 2.4.a-b, 2.6.a-c, 2.8.a

49

Getting to Know Your Data

 Data Objects and Attribute Types

 Basic Statistical Descriptions of Data

 Measuring Data Similarity and Dissimilarity

 Summary

50

50

25
 31/01/2021

Summary

 Data attribute types: nominal, binary, ordinal, interval-

scaled, ratio-scaled
 Many types of data sets, e.g., numerical, text, graph,
Web, image.
 Gain insight into the data by:
 Basic statistical data description: central tendency,
dispersion, graphical displays
 Datavisualization: map data onto graphical primitives
 Measure data similarity
 Above steps are the beginning of data preprocessing
 Many methods have been developed but still an active
area of research

51

52

January 31, 2021 Data Mining: Concepts and Techniques

52

26