Data Preprocessing Techniques

Data Preprocessing Techniques

Data preprocessing describes any type of processing performed on raw data to prepare it for
another processing procedure. Commonly used as a preliminary data mining practice, data
preprocessing transforms the data into a format that will be more easily and effectively
processed for the purpose of the user.

In a customer relationship management (CRM) context, data preprocessing is a component of

Web mining. Web usage logs may be preprocessed to extract meaningful sets of data called
user transactions, which consist of groups of URL references. User sessions may be tracked
to identify the user, the Web sites requested and their order and the length of time spent on
each one. Once these have been pulled out of the raw data, they yield more useful
information that can be put to the user's purposes, such as consumer research, marketing, or
personalization.

1.1 Why preprocess the data?

Incomplete, noisy, and inconsistent data are commonplace properties of large, real-world
databases and data warehouses. Incomplete data can occur for a number of reasons.
Attributes of interest may not always be available, such as customer information for sales
transaction data. Other data may not be included simply because it was not considered
important at the time of entry. Relevant data may not be recorded due to a misunderstanding,
or because of equipment malfunctions. Data that were inconsistent with other recorded data
may have been deleted. Furthermore, the recording of the history or modifications to the data
may have been overlooked. Missing data, particularly for tuples with missing values for some
attributes, may need to be inferred.

Data can be noisy, having incorrect attribute values, owing to the following. The data
collection instruments used may be faulty. There may have been human or computer errors
occurring at data entry. Errors in data transmission can also occur. There may be technology
limitations, such as limited buffer size for coordinating synchronized data transfer and
consumption. Incorrect data may also result from inconsistencies in naming conventions or
data codes used. Duplicate tuples also require data cleaning.

Christ University Bangalore Page| 1

Data Preprocessing Techniques

1.2 Data cleaning

Real-world data tend to be incomplete, noisy, and inconsistent. Data cleaning routines
attempt to fill in missing values, smooth out noise while identifying outliers, and correct
inconsistencies in the data. In this section, you will study basic methods for data cleaning.

1.2.1 Missing values

Imagine that you need to analyze AllElectronics sales and customer data. You note that many
tuples have no recorded value for several attributes, such as customer income. How can you
go about filling in the missing values for this attribute? Let's look at the following methods.

1. Ignore the tuple: This is usually done when the class label is missing. This method is
not very effective, unless the tuple contains several attributes with missing values. It
is especially poor when the percentage of missing values per attribute varies
considerably.
2. Fill in the missing value manually: In general, this approach is time-consuming and
may not be feasible given a large data set with many missing values.
3. Use a global constant to fill in the missing value: Replace all missing attribute
values by the same constant, such as a label like “Unknown", or - ~. If missing values
are replaced by, say, “Unknown", then the mining program may mistakenly think that
they form an interesting concept, since they all have a value in common, that of
“Unknown". Hence, although this method is simple, it is not recommended.
4. Use the attribute mean to fill in the missing value: For example, suppose that the
average income of AllElectronics customers is $28,000. Use this value to replace the
missing value for income.
5. Use the attribute mean for all samples belonging to the same class as the given
tuple: For example, if classifying customers according to credit risk, replace the
missing value with the average income value for customers in the same credit risk
category as that of the given tuple.
6. Use the most probable value to fill in the missing value: This may be determined
with inference-based tools using a Bayesian formalism or decision tree induction. For

Christ University Bangalore Page| 2

Data Preprocessing Techniques

example, using the other customer attributes in your data set, you may construct a
decision tree to predict the missing values for income.
1.2.2 Noisy data
“What is noise?" Noise is a random error or variance in a measured variable. Given a numeric
attribute such as, say, price, how can we “smooth" out the data to remove the noise? Let's
look at the following data smoothing techniques.

1. Binning methods: Binning methods smooth a sorted data value by consulting the
“neighborhood", or values around it. The sorted values are distributed into a number
of 'buckets', or bins. Figure 1.2 illustrates some binning techniques. In this example,
the data for price are first sorted and partitioned into equi-depth bins (of depth 3). In
smoothing by bin means, each value in a bin is replaced by the mean value of the
bin. For example, the mean of the values 4, 8, and 15 in Bin 1 is 9. Therefore, each
original value in this bin is replaced by the value 9. Similarly, smoothing by bin
medians can be employed, in which each bin value is replaced by the bin median. In
smoothing by bin boundaries, the minimum and maximum values in a given bin are
identified as the bin boundaries. Each bin value is then replaced by the closest
boundary value.

 Sorted data for price (in dollars): 4, 8, 15, 21, 21, 24, 25, 28, 34
 Partition into (equi-width) bins:
-Bin 1: 4, 8, 15
-Bin 2: 21, 21, 24
-Bin 3: 25, 28, 34
 Smoothing by bin means:
-Bin 1: 9, 9, 9,
-Bin 2: 22, 22, 22
-Bin 3: 29, 29, 29
 Smoothing by bin boundaries:
-Bin 1: 4, 4, 15
-Bin 2: 21, 21, 24
-Bin 3: 25, 25, 34

Figure 3.2: Binning methods for data smoothing.

Christ University Bangalore Page| 3

Data Preprocessing Techniques

2. Clustering: Outliers may be detected by clustering, where similar values are

organized into groups or “clusters". Intuitively, values which fall outside of the set of
clusters may be considered outliers.

3. Combined computer and human inspection: Outliers may be identified through a

combination of computer and human inspection. In one application, for example, an
information-theoretic measure was used to help identify outlier patterns in a
handwritten character database for classification. The measure's value reflected the
“surprise" content of the predicted character label with respect to the known label.
Outlier patterns may be informative. Patterns whose surprise content is above a
threshold are output to a list. A human can then sort through the patterns in the list to
identify the actual garbage ones.

4. Regression: Data can be smoothed by fitting the data to a function, such as with
regression. Linear regression involves finding the “best" line to fit two variables, so
that one variable can be used to predict the other. Multiple linear regressions are an
extension of linear regression, where more than two variables are involved and the
data are fit to a multidimensional surface. Using regression to find a mathematical
equation to fit the data helps smooth out the noise.

1.2.3 Inconsistent data

There may be inconsistencies in the data recorded for some transactions. Some data
inconsistencies may be corrected manually using external references. For example, errors
made at data entry may be corrected by performing a paper trace. This may be coupled with
routines designed to help correct the inconsistent use of codes. Knowledge engineering tools
may also be used to detect the violation of known data constraints. For example, known
functional dependencies between attributes can be used to find values contradicting the
functional constraints.

Christ University Bangalore Page| 4

Data Preprocessing Techniques

1.3 Data integration and transformation

1.3.1 Data integration

It is likely that your data analysis task will involve data integration, which combines data
from multiple sources into a coherent data store, as in data warehousing. These sources may
include multiple databases, data cubes, or flat files. Redundancy is another important issue.
An attribute may be redundant if it can be “derived" from another table, such as annual
revenue. Inconsistencies in attribute or dimension naming can also cause redundancies in the
resulting data set. Some redundancies can be detected by correlation analysis. For example,
given two attributes, such analysis can measure how strongly one attribute implies the other,
based on the available data. The correlation between attributes A and B can be measured by
P (A ^ B)
P (A) P (B)

If the resulting value of Equation is greater than 1, then A and B are positively correlated.
The higher the value, the more each attribute implies the other. Hence, a high value may
indicate that A (or B) may be removed as a redundancy. If the resulting value is equal to 1,
then A and B are independent and there is no correlation between them. If the resulting value
is less than 1, then A and B are negatively correlated. This means that each attribute
discourages the other.
A third important issue in data integration is the detection and resolution of data value
conflicts. Careful integration of the data from multiple sources can help reduce and avoid
redundancies and inconsistencies in the resulting data set.

1.3.2 Data transformation

In data transformation, the data are transformed or consolidated into forms appropriate for
mining. Data transformation can involve the following:
1. Normalization, where the attribute data are scaled so as to fall within a small specified
range, such as -1.0 to 1.0, or 0 to 1.0.
2. Smoothing, this works to remove the noise from data. Such techniques include binning,
clustering, and regression.

Christ University Bangalore Page| 5

Data Preprocessing Techniques

3. Aggregation, where summary or aggregation operations are applied to the data. For
example, the daily sales data may be aggregated so as to compute monthly and annual total
amounts. This step is typically used in constructing a data cube for analysis of the data at
multiple granularities.
4. Generalization of the data, where low level or 'primitive' (raw) data are replaced by higher
level concepts through the use of concept hierarchies. For example, categorical attributes, like
street, can be generalized to higher level concepts, like city or county. Similarly, values for
numeric attributes, like age, may be mapped to higher level concepts, like young, middle-
aged, and senior.

Min-max normalization performs a linear transformation on the original data. Suppose that
minA and maxA are the minimum and maximum values of an attribute A. Min-max
normalization maps a value v of A to V’ in the range [new_minA; new_maxA] by computing

V’= (v - minA/ maxA – mina)(new_maxA - new_minA) + new_minA

In z-score normalization (or zero-mean normalization), the values for an attribute A are
normalized based on the mean and standard deviation of A. A value v of A is normalized to
V’ by computing
V’ = v - meanA /stand devA
Where meanA and stand devA are the mean and standard deviation, respectively, of attribute
A. This method of normalization is useful when the actual minimum and maximum of
attribute A are unknown, or when there are outliers which dominate the min-max
normalization.
Normalization by decimal scaling normalizes by moving the decimal point of values of
attribute A. The number of decimal points moved depends on the maximum absolute value of
A. A value v of A is normalized to v’ by computing
V’ = v/10j’
where j is the smallest integer such that Max(jv0j) < 1.

Christ University Bangalore Page| 6

Data Preprocessing Techniques

1.4 Data reduction

Data reduction techniques can be applied to obtain a reduced representation of the data set
that is much smaller in volume, yet closely maintains the integrity of the original data. That
is, mining on the reduced data set should be more efficient yet produce the same analytical
results.
Strategies for data reduction include the following.
1. Data cube aggregation, where aggregation operations are applied to the data in the
construction of a data cube.
2. Dimension reduction, where irrelevant, weakly relevant or redundant attributes or
dimensions may be detected and removed.
3. Data compression, where encoding mechanisms are used to reduce the data set size.
4. Numerosity reduction, where the data are replaced or estimated by alternative, smaller data
representations such as parametric models or nonparametric methods such as clustering,
sampling, and the use of histograms.
5. Discretization and concept hierarchy generation, where raw data values for attributes are
replaced by ranges or higher conceptual levels. Concept hierarchies allow the mining of data
at multiple levels of abstraction, and are a powerful tool for data mining.

1.4.1 Data cube aggregation

Data cubes store multidimensional aggregated information. Concept hierarchies may exist for
each attribute, allowing the analysis of data at multiple levels of abstraction. For example, a
hierarchy for branch could allow branches to be grouped into regions, based on their address.
Data cubes provide fast access to precompiled, summarized data, thereby benefiting on-line
analytical processing as well as data mining. The cube created at the lowest level of
abstraction is referred to as the base cuboids. A cube for the highest level of abstraction is the
apex cuboids. Data cubes created for varying levels of abstraction are sometimes referred to
as cuboids, so that a “data cube" may instead refer to a lattice of cuboids. Each higher level of
abstraction further reduces the resulting data size.

1.4.2 Dimensionality reduction

Christ University Bangalore Page| 7

Data Preprocessing Techniques

Dimensionality reduction reduces the data set size by removing such attributes (or
dimensions) from it. Typically, methods of attribute subset selection are applied. The goal of
attribute subset selection is to find a minimum set of attributes such that the resulting
probability distribution of the data classes is as close as possible to the original distribution
obtained using all attributes. Mining on a reduced set of attributes has an additional benefit. It
reduces the number of attributes appearing in the discovered patterns, helping to make the
patterns easier to understand.

1.4.3 Data compression

In data compression, data encoding or transformations are applied so as to obtain a reduced or
“compressed" representation of the original data. If the original data can be reconstructed
from the compressed data without any loss of information, the data compression technique
used is called lossless. If, instead, we can reconstruct only an approximation of the original
data, then the data compression technique is called lossy. In this section, we instead focus on
two popular and effective methods of lossy data compression: wavelet transforms, and
principal components analysis.

Wavelet transforms:-
The discrete wavelet transform (DWT) is a linear signal processing technique that, when
applied to a data vector D, transforms it to a numerically different vector, D’, of wavelet
coefficients. The two vectors are of the same length.
The DWT is closely related to the discrete Fourier transform (DFT), a signal processing
technique involving sins and cosines. There is only one DFT, yet there are several DWTs.
The general algorithm for a discrete wavelet transform is as follows:-
1. The length, L, of the input data vector must be an integer power of two. This
condition can be met by padding the data vector with zeros, as necessary.

2. Each transform involves applying two functions. The first applies some data
smoothing, such as a sum or weighted average. The second performs a weighted
difference.

Christ University Bangalore Page| 8

Data Preprocessing Techniques

3. The two functions are applied to pairs of the input data, resulting in two sets of data of
length L=2. In general, these respectively represent a smoothed version of the input
data, and the high-frequency content of it.

4. The two functions are recursively applied to the sets of data obtained in the previous
loop, until the resulting data sets obtained are of desired length.

5. A selection of values from the data sets obtained in the above iterations are designated
the wavelet coefficients of the transformed data.

Principal components analysis

Suppose that the data to be compressed consists of N tuples or data vectors, from k-
dimensions. Principal components analysis (PCA) searches for c k-dimensional orthogonal
vectors that can best be used to represent the data, where c << N. The original data is thus
projected onto a much smaller space, resulting in data compression. PCA can be used as a
form of dimensionality reduction.
The basic procedure is as follows:-
1. The input data are normalized, so that each attribute falls within the same range. This
step helps ensure that attributes with large domains will not dominate attributes with
smaller domains.

2. PCA computes N orthonormal vectors which provide a basis for the normalized input
data. These are unit vectors that each point in a direction perpendicular to the others.
These vectors are referred to as the principal components. The input data are a linear
combination of the principal components.

3. The principal components are sorted in order of decreasing “significance" or strength.

The principal components essentially serve as a new set of axes for the data,
providing important information about variance.

Christ University Bangalore Page| 9

Data Preprocessing Techniques

4. Since the components are sorted according to decreasing order of “significance", the
size of the data can be reduced by eliminating the weaker components, i.e., those with
low variance.

1.4.4 Numerosity reduction

“Can we reduce the data volume by choosing alternative, `smaller' forms of data
representation?" Techniques of Numerosity reduction can indeed be applied for this purpose.
These techniques may be parametric or non-parametric. For parametric methods, a model is
used to estimate the data, so that typically only the data parameters need be stored, instead of
the actual data. (Outliers may also be stored). Log-linear models, which estimate discrete
multidimensional probability distributions, are an example. Non-parametric methods for
storing reduced representations of the data include histograms, clustering, and sampling.

1.5 Discretization and concept hierarchy generation

Discretization techniques can be used to reduce the number of values for a given continuous
attribute, by dividing the range of the attribute into intervals. Interval labels can then be used
to replace actual data values. Reducing the number of values for an attribute is especially
beneficial if decision tree-based methods of classification mining are to be applied to the
preprocessed data. These methods are typically recursive, where a large amount of time is
spent on sorting the data at each step. Hence, the smaller the number of distinct values to sort,
the faster these methods should be. Many Discretization techniques can be applied
recursively in order to provide a hierarchical, or multiresolution partitioning of the attribute
values, known as a concept hierarchy. Concept hierarchies are useful for mining at multiple
levels of abstraction. A concept hierarchy for a given numeric attribute defines a
Discretization of the attribute. Concept hierarchies can be used to reduce the data by
collecting and replacing low level concepts (such as numeric values for the attribute age) by
higher level concepts (such as young, middle-aged, or senior). Although detail is lost by such
data generalization, the generalized data may be more meaningful and easier to interpret, and
will require less space than the original data. Mining on a reduced data set will require fewer
input/output operations and be more efficient than mining on a larger, ungeneralized data set.
More than one concept hierarchy can be defined for the same attribute in order to
accommodate the needs of the various users.

Christ University Bangalore Page| 10

Data Preprocessing Techniques

CONCLUSION

Data preparation is an important issue for both data warehousing and data mining, as real-
world data tends to be incomplete, noisy, and inconsistent. Data preparation includes data
cleaning, data integration, data transformation, and data reduction.
Data cleaning routines can be used to fill in missing values, smooth noisy data,
identify outliers, and correct data inconsistencies. Data integration combines data from
multiples sources to form a coherent data store. Metadata, correlation analysis, data conflict
detection, and the resolution of semantic heterogeneity contribute towards smooth data
integration. Data transformation routines conform the data into appropriate forms for mining.
For example, attribute data may be normalized so as to fall between a small range, such as 0
to 1.0. Data reduction techniques such as data cube aggregation, dimension reduction, data
compression, Numerosity reduction, and Discretization can be used to obtain a reduced
representation of the data, while minimizing the loss of information content. Concept
hierarchies organize the values of attributes or dimensions into gradual levels of abstraction.
They are a form a Discretization that is particularly useful in multilevel mining. Automatic
generation of concept hierarchies for categorical data may be based on the number of distinct
values of the attributes defining the hierarchy. For numeric data, techniques such as data
segmentation by partition rules, histogram analysis, and clustering analysis can be used.
Although several methods of data preparation have been developed, data preparation
remains an active area of research.

Christ University Bangalore Page| 11

Data Preprocessing Techniques

REFERENCES

[1] S. Agarwal, R. Agrawal, P. M. Deshpande, A. Gupta, J. F. Naughton, R. Ramakrishnan,

and S. Sarawagi. On the computation of multidimensional aggregates. In Proc. 1996 Int.
Conf. Very Large Data Bases, pages 506{521, Bombay, India, Sept. 1996.
[2] D. P. Ballou and G. K. Tayi. Enhancing data quality in data warehouse environments.
[3] J. Devore and R. Peck. Statistics: The Exploration and Analysis of Data. New York:
Duxbury Press, 1997.
[4] A. J. Dobson. An Introduction to Generalized Linear Models. Chapman and Hall, 1990.
[5] V. Harinarayan, A. Rajaraman, and J. D. Ullman. Implementing data cubes efficiently.
[6] www.wikipedia.com
[7] www.buzpages.com

Christ University Bangalore Page| 12