Data Preprocessing

January 20, 2015

Data Mining: Concepts and Techniques

Data Preprocessing

Why preprocess the data?

Data cleaning

Data integration and transformation

Data reduction

Discretization and concept hierarchy

generation

Summary

January 20, 2015

Data Mining: Concepts and Techniques

Data Preprocessing

Why preprocess the data?

Data cleaning

Data integration and transformation

Data reduction

Discretization and concept hierarchy

generation

Summary

January 20, 2015

Data Mining: Concepts and Techniques

Why Data Preprocessing?

Data in the real world is dirty

incomplete: lacking attribute values,
lacking certain attributes of interest, or
containing only aggregate data

noisy: containing errors or outliers

e.g., occupation=
e.g., Salary=-10

inconsistent: containing discrepancies in

codes or names

January 20, 2015

e.g., Age=42 Birthday=03/07/1997

e.g., Was rating 1,2,3, now rating A, B, C
e.g., discrepancy between duplicate records
Data Mining: Concepts and Techniques

Why Is Data Dirty?

Incomplete data may come from

Noisy data (incorrect values) may come from

Faulty data collection instruments

Human or computer error at data entry
Errors in data transmission

Inconsistent data may come from

Not applicable data value when collected

Different considerations between the time when the data
was collected and when it is analyzed.
Human/hardware/software problems

Different data sources

Functional dependency violation (e.g., modify some linked
data)

Duplicate records also need data cleaning

January 20, 2015

Data Mining: Concepts and Techniques

Why Is Data Preprocessing

Important?

No quality data, no quality mining results!

Quality decisions must be based on quality data

e.g., duplicate or missing data may cause incorrect or

even misleading statistics.

Data warehouse needs consistent integration of

quality data

Data extraction, cleaning, and transformation

comprises the majority of the work of building a
data warehouse

January 20, 2015

Data Mining: Concepts and Techniques

Multi-Dimensional Measure of Data

Quality

Measures for data quality: A multidimensional view

Accuracy: correct or wrong, accurate or not

Completeness: not recorded, unavailable,

Consistency: some modified but some not,

dangling,

Timeliness: timely update?

Believability: how trustable the data are correct?

Interpretability: how easily the data can be

understood?

Major Tasks in Data Preprocessing

Data cleaning

Data integration

Fill in missing values, smooth noisy data, identify or

remove outliers, and resolve inconsistencies
Integration of multiple databases, data cubes, or files

Data reduction

Dimensionality reduction

Numerosity reduction

Data compression

Data transformation and data discretization

Normalization

Concept hierarchy generation

Forms of Data Preprocessing

January 20, 2015

Data Mining: Concepts and Techniques

Data Preprocessing

Why preprocess the data?

Data cleaning

Data integration and transformation

Data reduction

Discretization and concept hierarchy

generation

Summary

January 20, 2015

Data Mining: Concepts and Techniques

10

Data Cleaning

Importance
Data cleaning is one of the three biggest
problems in data warehousingRalph
Kimball
Data cleaning is the number one problem in
data warehousingDCI survey

January 20, 2015

Data Mining: Concepts and Techniques

11

Data Cleaning

Importance
Data cleaning is one of the three biggest
problems in data warehousingRalph Kimball
Data cleaning is the number one problem in
data warehousingDCI survey

Data cleaning tasks

Fill in missing values

Identify outliers and smooth out noisy data

Correct inconsistent data

Resolve redundancy caused by data integration

January 20, 2015

Data Mining: Concepts and Techniques

12

Incomplete (Missing) Data

Data is not always available

13

E.g., many tuples have no recorded value for

several attributes, such as customer income in
sales data

Incomplete (Missing) Data

Data is not always available

14

E.g., many tuples have no recorded value for several

attributes, such as customer income in sales data

Missing data may be due to

equipment malfunction

inconsistent with other recorded data and thus

deleted

data not entered due to misunderstanding

certain data may not be considered important at the

time of entry

not register history or changes of the data

Missing data may need to be inferred

How to Handle Missing Data?

Ignore the tuple: usually done when class label is

missing (when doing classification)not effective
when the % of missing values per attribute varies
considerably

Fill in the missing value manually: tedious +

infeasible?

15

How to Handle Missing Data?

Ignore the tuple: usually done when class label is

missing (when doing classification)not effective when
the % of missing values per attribute varies considerably

Fill in the missing value manually: tedious + infeasible?

Fill in it automatically with

16

a global constant : e.g., unknown, a new class?!

the attribute mean

the attribute mean for all samples belonging to the

same class: smarter

the most probable value: inference-based such as

Bayesian formula or decision tree

Noisy Data

Noise: random error or variance in a measured

variable

Incorrect attribute values may due to

faulty data collection instruments
data entry problems
data transmission problems
technology limitation
inconsistency in naming convention

January 20, 2015

Data Mining: Concepts and Techniques

17

How to Handle Noisy Data?

Binning
first sort data and partition into (equalfrequency) bins
then one can smooth by bin means, smooth by
bin median, smooth by bin boundaries, etc.

January 20, 2015

Data Mining: Concepts and Techniques

18

Simple Discretization Methods:

Binning

Equal-width (distance) partitioning

Divides the range into N intervals of equal size: uniform grid

if A and B are the lowest and highest values of the attribute, the
width of intervals will be: W = (B A)/N.

The most straightforward, but outliers may dominate

presentation

Skewed data is not handled well

Equal-depth (frequency) partitioning

Divides the range into N intervals, each containing

approximately same number of samples

Good data scaling

Managing categorical attributes can be tricky

January 20, 2015

Data Mining: Concepts and Techniques

19

Binning Methods for Data

Smoothing
Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24,
25, 26, 28, 29, 34
* Partition into equal-frequency (equi-depth) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34

January 20, 2015

Data Mining: Concepts and Techniques

20

Binning Methods for Data

Smoothing
Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24,
25, 26, 28, 29, 34
* Partition into equal-frequency (equi-depth) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34
* Smoothing by bin means:
- Bin 1: 9, 9, 9, 9
- Bin 2: 23, 23, 23, 23
- Bin 3: 29, 29, 29, 29

January 20, 2015

Data Mining: Concepts and Techniques

21

Binning Methods for Data

Smoothing
Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25,
26, 28, 29, 34
* Partition into equal-frequency (equi-depth) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34
* Smoothing by bin means:
- Bin 1: 9, 9, 9, 9
- Bin 2: 23, 23, 23, 23
- Bin 3: 29, 29, 29, 29
* Smoothing by bin boundaries:
- Bin 1: 4, 4, 4, 15
- Bin 2: 21, 21, 25, 25
- Bin 3: 26, 26, 26, 34

January 20, 2015

Data Mining: Concepts and Techniques

22

How to Handle Noisy Data?

Binning
first sort data and partition into (equalfrequency) bins
then one can smooth by bin means, smooth by
bin median, smooth by bin boundaries, etc.
Regression
smooth by fitting the data into regression
functions

January 20, 2015

Data Mining: Concepts and Techniques

23

Regression
y
Y1

y=x+1

Y1

X1

January 20, 2015

Data Mining: Concepts and Techniques

24

How to Handle Noisy Data?

Binning
first sort data and partition into (equalfrequency) bins
then one can smooth by bin means, smooth by
bin median, smooth by bin boundaries, etc.
Regression
smooth by fitting the data into regression
functions
Clustering
detect and remove outliers

January 20, 2015

Data Mining: Concepts and Techniques

25

Cluster Analysis

January 20, 2015

Data Mining: Concepts and Techniques

26

How to Handle Noisy Data?

Binning
first sort data and partition into (equal-frequency)
bins
then one can smooth by bin means, smooth by
bin median, smooth by bin boundaries, etc.
Regression
smooth by fitting the data into regression functions
Clustering
detect and remove outliers
Combined computer and human inspection
detect suspicious values and check by human
(e.g., deal with possible outliers)

January 20, 2015

Data Mining: Concepts and Techniques

27

Problems
3.3 Suppose that the data for analysis includes the
attribute age. The age values for the data tuples
are (in increasing order)
13,15,16,16,19,20,20,21,22,22,25,25,25,25,30,33,3
3,35,35,35,35,36,40,45,46,52,70.
i. Use smoothing by bin means and bondaries to
smooth the data, using a bin depth of 3. Illustrate
your steps.
ii. How might you determine the outliers?

January 20, 2015

Data Mining: Concepts and Techniques

28

Data Cleaning as a Process

Data discrepancy detection

Use metadata (e.g., domain, range, dependency, distribution)
Check field overloading
Check uniqueness rule, consecutive rule and null rule
Use commercial tools

Data scrubbing: use simple domain knowledge (e.g., postal

code, spell-check) to detect errors and make corrections

Data auditing: by analyzing data to discover rules and

relationship to detect violators (e.g., correlation and
clustering to find outliers)
Data migration and integration
Data migration tools: allow transformations to be specified
ETL (Extraction/Transformation/Loading) tools: allow users to
specify transformations through a graphical user interface
Integration of the two processes
Iterative and interactive (e.g., Potters Wheels)
29