Descriptive Statistics

3.1 Introduction

Descriptive statistics helps to simplify large amounts of data in a sensible way.

In contrast to inferential statistics, which will be introduced in a later chapter, in
descriptive statistics we do not draw conclusions beyond the data we are analyzing;
neither do we reach any conclusions regarding hypotheses we may make. We do not
try to infer characteristics of the population (see below) of the data, but claim to
present quantitative descriptions of it in a manageable form. It is simply a way to
describe the data.
Statistics, and in particular descriptive statistics, is based on two main concepts:

a population is a collection of objects, items (units) about which information is

sought;
a sample is a part of the population that is observed.

Descriptive statistics applies the concepts, measures, and terms that are used to
describe the basic features of the samples in a study. These procedures are essential
to provide summaries about the samples as an approximation of the population.
Together with simple graphics, they form the basis of every quantitative analysis of
data. In order to describe the sample data and to be able to infer any conclusion, we
should go through several steps:

1. Data preparation: Given a specific example, we need to prepare the data for
generating statistically valid descriptions.
2. Descriptive statistics: This generates different statistics to describe and summa-
rize the data concisely and evaluate different ways to visualize them.

Springer International Publishing Switzerland 2017 29

L. Igual and S. Segu, Introduction to Data Science,
Undergraduate Topics in Computer Science, DOI 10.1007/978-3-319-50017-1_3
30 3 Descriptive Statistics

3.2 Data Preparation

One of the first tasks when analyzing data is to collect and prepare the data in a format
appropriate for analysis of the samples. The most common steps for data preparation
involve the following operations.

1. Obtaining the data: Data can be read directly from a file or they might be obtained
by scraping the web.
2. Parsing the data: The right parsing procedure depends on what format the data
are in: plain text, fixed columns, CSV, XML, HTML, etc.
3. Cleaning the data: Survey responses and other data files are almost always in-
complete. Sometimes, there are multiple codes for things such as, not asked, did
not know, and declined to answer. And there are almost always errors. A simple
strategy is to remove or ignore incomplete records.
4. Building data structures: Once you read the data, it is necessary to store them in
a data structure that lends itself to the analysis we are interested in. If the data fit
into the memory, building a data structure is usually the way to go. If not, usually
a database is built, which is an out-of-memory data structure. Most databases
provide a mapping from keys to values, so they serve as dictionaries.

3.2.1 The Adult Example

Let us consider a public database called the Adult dataset, hosted on the UCIs
Machine Learning Repository.1 It contains approximately 32,000 observations con-
cerning different financial parameters related to the US population: age, sex, marital
(marital status of the individual), country, income (Boolean variable: whether the per-
son makes more than $50,000 per annum), education (the highest level of education
achieved by the individual), occupation, capital gain, etc.
We will show that we can explore the data by asking questions like: Are men
more likely to become high-income professionals than women, i.e., to receive an
income of over $50,000 per annum?

1 https://archive.ics.uci.edu/ml/datasets/Adult.
 3.2 Data Preparation 31

First, let us read the data:

In [1]:
file = open ( files / ch03 / adult . data , r )
def c h r _ i n t ( a ) :
if a . i s d i g i t () : r e t u r n int ( a )
else : r e t u r n 0

data = []
for line in file :
data1 = line . split ( , )
if len ( d a t a 1 ) == 15:
d a t a . a p p e n d ([ c h r _ i n t ( d a t a 1 [0]) , data1 [1] ,
c h r _ i n t ( d a t a 1 [2]) , data1 [3] ,
c h r _ i n t ( d a t a 1 [4]) , data1 [5] ,
d a t a 1 [6] , d a t a 1 [7] , d a t a 1 [8] ,
d a t a 1 [9] , c h r _ i n t ( d a t a 1 [ 1 0 ] ) ,
chr_int ( data1 [11]) ,
chr_int ( data1 [12]) ,
d a t a 1 [13] , data1 [14]
])

Checking the data, we obtain:

In [2]:
p r i n t data [1:2]

Out[2]: [[50, Self-emp-not-inc, 83311, Bachelors, 13,

Married-civ-spouse, Exec-managerial, Husband, White,
Male, 0, 0, 13, United-States, <= 50K ]]
One of the easiest ways to manage data in Python is by using the DataFrame
structure, defined in the Pandas library, which is a two-dimensional, size-mutable,
potentially heterogeneous tabular data structure with labeled axes:
In [3]:
df = pd . D a t a F r a m e ( data )
df . c o l u m n s = [
age , t y p e _ e m p l o y e r , f n l w g t ,
education , education_num , marital ,
o c c u p a t i o n , r e l a t i o n s h i p , race ,
sex , c a p i t a l _ g a i n , c a p i t a l _ l o s s ,
hr_per_week , country , income
]

The command shape gives exactly the number of data samples (in rows, in this
case) and features (in columns):
In [4]:
df . s h a p e

Out[4]: (32561, 15)

 32 3 Descriptive Statistics

Thus, we can see that our dataset contains 32,561 data records with 15 features
each. Let us count the number of items per country:
In [5]:
c o u n t s = df . g r o u p b y ( c o u n t r y ) . size ()
p r i n t c o u n t s . h e a d ()

Out[5]: country
? 583
Cambodia 19
Vietnam 67
Yugoslavia 16
The first row shows the number of samples with unknown country, followed by
the number of samples corresponding to the first countries in the dataset.
Let us split people according to their gender into two groups: men and women.
In [6]:
ml = df [( df . sex == Male ) ]

If we focus on high-income professionals separated by sex, we can do:

In [7]:
ml1 = df [( df . sex == Male ) & ( df . i n c o m e == >50 K \ n )
]
fm = df [( df . sex == F e m a l e ) ]
fm1 = df [( df . sex == F e m a l e ) & ( df . i n c o m e == >50 K \ n
) ]

3.3 Exploratory Data Analysis

The data that come from performing a particular measurement on all the subjects
in a sample represent our observations for a single characteristic like country,
age, education, etc. These measurements and categories represent a sample
distribution of the variable, which in turn approximately represents the population
distribution of the variable. One of the main goals of exploratory data analysis is
to visualize and summarize the sample distribution, thereby allowing us to make
tentative assumptions about the population distribution.

3.3.1 Summarizing the Data

The data in general can be categorical or quantitative. For categorical data, a simple
tabulation of the frequency of each category is the best non-graphical exploration
for data analysis. For example, we can ask ourselves what is the proportion of high-
income professionals in our database:
 3.3 Exploratory Data Analysis 33

In [8]:
df1 = df [( df . i n c o m e == >50 K \ n ) ]
p r i n t The rate of people with high income is : ,
int ( len ( df1 ) / f l o a t ( len ( df ) ) * 1 0 0 ) , %.
p r i n t The rate of men with high i n c o m e is : ,
int ( len ( ml1 ) / f l o a t ( len ( ml ) ) * 1 0 0 ) , %.
p r i n t The rate of women with high i n c o m e is : ,
int ( len ( fm1 ) / f l o a t ( len ( fm ) ) * 1 0 0 ) , %.

Out[8]: The rate of people with high income is: 24 %.

The rate of men with high income is: 30 %.
The rate of women with high income is: 10 %.
Given a quantitative variable, exploratory data analysis is a way to make prelim-
inary assessments about the population distribution of the variable using the data of
the observed samples. The characteristics of the population distribution of a quanti-
tative variable are its mean, deviation, histograms, outliers, etc. Our observed data
represent just a finite set of samples of an often infinite number of possible samples.
The characteristics of our randomly observed samples are interesting only to the
degree that they represent the population of the data they came from.

3.3.1.1 Mean
One of the first measurements we use to have a look at the data is to obtain sample
statistics from the data, such as the sample mean [1]. Given a sample of n values,
{xi }, i = 1, . . . , n, the mean, , is the sum of the values divided by the number of
values,2 in other words:
n
1!
= xi . (3.1)
n
i=1
The terms mean and average are often used interchangeably. In fact, the main
distinction between them is that the mean of a sample is the summary statistic com-
puted by Eq. (3.1), while an average is not strictly defined and could be one of many
summary statistics that can be chosen to describe the central tendency of a sample.
In our case, we can consider what the average age of men and women samples in
our dataset would be in terms of their mean:

2 We will use the following notation: X is a random variable, x is a column vector, xT (the transpose
of x) is a row vector, X is a matrix, and xi is the i-th element of a dataset.
 34 3 Descriptive Statistics

In [9]:
p r i n t The a v e r a g e age of men is : ,
ml [ age ]. mean ()
p r i n t The a v e r a g e age of w o m e n is : ,
fm [ age ]. mean ()

p r i n t The a v e r a g e age of high - i n c o m e men is : ,

ml1 [ age ]. mean ()
p r i n t The a v e r a g e age of high - i n c o m e w o m e n is : ,
fm1 [ age ]. mean ()

Out[9]: The average age of men is: 39.4335474989

The average age of women is: 36.8582304336
The average age of high-income men is: 44.6257880516
The average age of high-income women is: 42.1255301103
This difference in the sample means can be considered initial evidence that there
are differences between men and women with high income!
Comment: Later, we will work with both concepts: the population mean and the
sample mean. We should not confuse them! The first is the mean of samples taken
from the population; the second, the mean of the whole population.

3.3.1.2 Sample Variance

The mean is not usually a sufficient descriptor of the data. We can go further by
knowing two numbers: mean and variance. The variance 2 describes the spread of
the data and it is defined as follows:
1!
2 = (xi )2 . (3.2)
n
i
The term (xi ) is called the deviation from the mean, so the variance is the mean
squared deviation. The square root of the variance, , is called the standard deviation.
We consider the standard deviation, because the variance is hard to interpret (e.g., if
the units are grams, the variance is in grams squared).
Let us compute the mean and the variance of hours per week men and women in
our dataset work:
In [10]:
m l _ m u = ml [ age ]. mean ()
f m _ m u = fm [ age ]. mean ()
m l _ v a r = ml [ age ]. var ()
f m _ v a r = fm [ age ]. var ()
m l _ s t d = ml [ age ]. std ()
f m _ s t d = fm [ age ]. std ()
p r i n t S t a t i s t i c s of age for men : mu : ,
ml_mu , var : , ml_var , std : , m l _ s t d
p r i n t S t a t i s t i c s of age for w o m e n : mu : ,
fm_mu , var : , fm_var , std : , f m _ s t d
 3.3 Exploratory Data Analysis 35

Out[10]: Statistics of age for men: mu: 39.4335474989 var: 178.773751745

std: 13.3706301925
Statistics of age for women: mu: 36.8582304336 var:
196.383706395 std: 14.0136970994
We can see that the mean number of hours worked per week by women is signif-
icantly lesser than that worked by men, but with much higher variance and standard
deviation.

3.3.1.3 Sample Median

The mean of the samples is a good descriptor, but it has an important drawback: what
will happen if in the sample set there is an error with a value very different from the
rest? For example, considering hours worked per week, it would normally be in a
range between 20 and 80; but what would happen if by mistake there was a value
of 1000? An item of data that is significantly different from the rest of the data is
called an outlier. In this case, the mean, , will be drastically changed towards the
outlier. One solution to this drawback is offered by the statistical median, 12 , which
is an order statistic giving the middle value of a sample. In this case, all the values
are ordered by their magnitude and the median is defined as the value that is in the
middle of the ordered list. Hence, it is a value that is much more robust in the face
of outliers.
Let us see, the median age of working men and women in our dataset and the
median age of high-income men and women:
In [11]:
m l _ m e d i a n = ml [ age ]. m e d i a n ()
f m _ m e d i a n = fm [ age ]. m e d i a n ()
p r i n t " M e d i a n age per men and w o m e n : " ,
ml_median , f m _ m e d i a n

m l _ m e d i a n _ a g e = ml1 [ age ]. m e d i a n ()
f m _ m e d i a n _ a g e = fm1 [ age ]. m e d i a n ()
p r i n t " M e d i a n age per men and w o m e n w i t h high -
income : " ,
ml_median_age , fm_median_age

Out[11]: Median age per men and women: 38.0 35.0

Median age per men and women with high-income: 44.0 41.0
As expected, the median age of high-income people is higher than the whole set
of working people, although the difference between men and women in both sets is
the same.

3.3.1.4 Quantiles and Percentiles

Sometimes we are interested in observing how sample data are distributed in general.
In this case, we can order the samples {xi }, then find the x p so that it divides the data
into two parts, where:
 36 3 Descriptive Statistics

Fig. 3.1 Histogram of the age of working men (left) and women (right)

a fraction p of the data values is less than or equal to x p and

the remaining fraction (1 p) is greater than x p .

That value, x p , is the p-th quantile, or the 100 p-th percentile. For example, a
5-number summary is defined by the values xmin , Q 1 , Q 2 , Q 3 , xmax , where Q 1 is
the 25 p-th percentile, Q 2 is the 50 p-th percentile and Q 3 is the 75 p-th
percentile.

3.3.2 Data Distributions

Summarizing data by just looking at their mean, median, and variance can be danger-
ous: very different data can be described by the same statistics. The best thing to do
is to validate the data by inspecting them. We can have a look at the data distribution,
which describes how often each value appears (i.e., what is its frequency).
The most common representation of a distribution is a histogram, which is a graph
that shows the frequency of each value. Let us show the age of working men and
women separately.
In [12]:
m l _ a g e = ml [ age ]
ml_age . hist ( normed = 0, histtype = stepfilled ,
bins = 20)

In [13]:
f m _ a g e = fm [ age ]
fm_age . hist ( normed = 0, histtype = stepfilled ,
bins = 10)

The output can be seen in Fig. 3.1. If we want to compare the histograms, we can
plot them overlapping in the same graphic as follows:
 3.3 Exploratory Data Analysis 37

Fig. 3.2 Histogram of the age of working men (in ochre) and women (in violet) (left). Histogram of
the age of working men (in ochre), women (in blue), and their intersection (in violet) after samples
normalization (right)

In [14]:
i m p o r t s e a b o r n as sns
fm_age . hist ( normed = 0, histtype = stepfilled ,
alpha = .5 , bins = 20)
ml_age . hist ( normed = 0, histtype = stepfilled ,
a l p h a = .5 ,
c o l o r = sns . d e s a t u r a t e ( " i n d i a n r e d " ,
.75) ,
bins = 10)

The output can be seen in Fig. 3.2 (left). Note that we are visualizing the absolute
values of the number of people in our dataset according to their age (the abscissa of
the histogram). As a side effect, we can see that there are many more men in these
conditions than women.
We can normalize the frequencies of the histogram by dividing/normalizing by
n, the number of samples. The normalized histogram is called the Probability Mass
Function (PMF).
In [15]:
fm_age . hist ( normed = 1, histtype = stepfilled ,
alpha = .5 , bins = 20)
ml_age . hist ( normed = 1, histtype = stepfilled ,
alpha = .5 , bins = 10 ,
c o l o r = sns . d e s a t u r a t e ( " i n d i a n r e d " ,
.75) )

This outputs Fig. 3.2 (right), where we can observe a comparable range of indi-
viduals (men and women).
The Cumulative Distribution Function (CDF), or just distribution function,
describes the probability that a real-valued random variable X with a given proba-
bility distribution will be found to have a value less than or equal to x. Let us show
the CDF of age distribution for both men and women.
 38 3 Descriptive Statistics

Fig. 3.3 The CDF of the age

of working male (in blue)
and female (in red) samples

In [16]:
m l _ a g e . h i s t ( n o r m e d = 1 , h i s t t y p e = step ,
c u m u l a t i v e = True , l i n e w i d t h = 3.5 ,
bins = 20)
f m _ a g e . h i s t ( n o r m e d = 1 , h i s t t y p e = step ,
c u m u l a t i v e = True , l i n e w i d t h = 3.5 ,
bins = 20 ,
c o l o r = sns . d e s a t u r a t e ( " i n d i a n r e d " ,
.75) )

The output can be seen in Fig. 3.3, which illustrates the CDF of the age distributions
for both men and women.

3.3.3 Outlier Treatment

As mentioned before, outliers are data samples with a value that is far from the central
tendency. Different rules can be defined to detect outliers, as follows:

Computing samples that are far from the median.

Computing samples whose values exceed the mean by 2 or 3 standard deviations.

For example, in our case, we are interested in the age statistics of men versus
women with high incomes and we can see that in our dataset, the minimum age is 17
years and the maximum is 90 years. We can consider that some of these samples are
due to errors or are not representable. Applying the domain knowledge, we focus on
the median age (37, in our case) up to 72 and down to 22 years old, and we consider
the rest as outliers.
 3.3 Exploratory Data Analysis 39

In [17]:
df2 = df . drop ( df . index [
( df . i n c o m e == >50 K \ n ) &
( df [ age ] > df [ age ]. m e d i a n () + 35) &
( df [ age ] > df [ age ]. m e d i a n () -15)
])
m l 1 _ a g e = ml1 [ age ]
f m 1 _ a g e = fm1 [ age ]

m l 2 _ a g e = m l 1 _ a g e . drop ( m l 1 _ a g e . i n d e x [
( m l 1 _ a g e > df [ age ]. m e d i a n () + 35) &
( m l 1 _ a g e > df [ age ]. m e d i a n () - 15)
])
f m 2 _ a g e = f m 1 _ a g e . drop ( f m 1 _ a g e . i n d e x [
( f m 1 _ a g e > df [ age ]. m e d i a n () + 35) &
( f m 1 _ a g e > df [ age ]. m e d i a n () - 15)
])

We can check how the mean and the median changed once the data were cleaned:
In [18]:
m u 2 m l = m l 2 _ a g e . mean ()
s t d 2 m l = m l 2 _ a g e . std ()
m d 2 m l = m l 2 _ a g e . m e d i a n ()
m u 2 f m = f m 2 _ a g e . mean ()
s t d 2 f m = f m 2 _ a g e . std ()
m d 2 f m = f m 2 _ a g e . m e d i a n ()

print " Men s t a t i s t i c s : "

print " Mean : " , mu2ml , " Std : " , s t d 2 m l
print " Median :" , md2ml
print " Min : " , m l 2 _ a g e . min () , " Max : " , m l 2 _ a g e . max ()

print " Women statistics :"

print " Mean : " , mu2fm , " Std : " , s t d 2 f m
print " Median :" , md2fm
print " Min : " , f m 2 _ a g e . min () , " Max : " , f m 2 _ a g e . max ()

Out[18]: Men statistics: Mean: 44.3179821239 Std: 10.0197498572 Median:

44.0 Min: 19 Max: 72
Women statistics: Mean: 41.877028181 Std: 10.0364418073 Median:
41.0 Min: 19 Max: 72
Let us visualize how many outliers are removed from the whole data by:
In [19]:
plt . f i g u r e ( f i g s i z e = (13.4 , 5) )
df . age [( df . i n c o m e == >50 K \ n ) ]
. plot ( alpha = .25 , color = blue )
df2 . age [( df2 . i n c o m e == >50 K \ n ) ]
. plot ( alpha = .45 , color = red )
 40 3 Descriptive Statistics

Fig. 3.4 The red shows the cleaned data without the considered outliers (in blue)

Figure 3.4 shows the outliers in blue and the rest of the data in red. Visually, we
can confirm that we removed mainly outliers from the dataset.
Next we can see that by removing the outliers, the difference between the popula-
tions (men and women) actually decreased. In our case, there were more outliers in
men than women. If the difference in the mean values before removing the outliers
is 2.5, after removing them it slightly decreased to 2.44:
In [20]:
p r i n t The mean d i f f e r e n c e with o u t l i e r s is : %4.2 f .

% ( ml_age . mean () - fm_age . mean () )
p r i n t The mean d i f f e r e n c e w i t h o u t o u t l i e r s is :
%4.2 f .
% ( m l 2 _ a g e . mean () - f m 2 _ a g e . mean () )

Out[20]: The mean difference with outliers is: 2.58.

The mean difference without outliers is: 2.44.
Let us observe the difference of men and women incomes in the cleaned subset
with some more details.
In [21]:
countx , d i v i s i o n x = np . h i s t o g r a m ( ml2_age , n o r m e d =
True )
county , d i v i s i o n y = np . h i s t o g r a m ( fm2_age , n o r m e d =
True )

val = [( d i v i s i o n x [ i ] + d i v i s i o n x [ i +1]) /2
for i in r a n g e ( len ( d i v i s i o n x ) - 1) ]
plt . p l o t ( val , c o u n t x - county , o - )

The results are shown in Fig. 3.5. One can see that the differences between male
and female values are slightly negative before age 42 and positive after it. Hence,
women tend to be promoted (receive more than 50 K) earlier than men.
 3.3 Exploratory Data Analysis 41

Fig. 3.5 Differences in high-income earner men versus women as a function of age

3.3.4 Measuring Asymmetry: Skewness and Pearsons Median

Skewness Coefficient

For univariate data, the formula for skewness is a statistic that measures the asym-
metry of the set of n data samples, xi :
"
1 i (xi 3 )
g1 = , (3.3)
n 3
where is the mean, is the standard deviation, and n is the number of data points.
Negative deviation indicates that the distribution skews left (it extends further
to the left than to the right). One can easily see that the skewness for a normal
distribution is zero, and any symmetric data must have a skewness of zero. Note
that skewness can be affected by outliers! A simpler alternative is to look at the
relationship between the mean and the median 12 .
In [22]:
def s k e w n e s s ( x ) :
res = 0
m = x . mean ()
s = x . std ()
for i in x :
res += (i - m ) * ( i - m ) * (i - m )
res /= ( len ( x ) * s * s * s )
r e t u r n res

p r i n t " S k e w n e s s of the male p o p u l a t i o n = " ,

skewness ( ml2_age )
p r i n t " S k e w n e s s of the f e m a l e p o p u l a t i o n is = " ,
skewness ( fm2_age )
 42 3 Descriptive Statistics

Out[22]: Skewness of the male population = 0.266444383843

Skewness of the female population = 0.386333524913
That is, the female population is more skewed than the male, probably since men
could be most prone to retire later than women.
The Pearsons median skewness coefficient is a more robust alternative to the
skewness coefficient and is defined as follows:
g p = 3( 12 ).
There are many other definitions for skewness that will not be discussed here. In
our case, if we check the Pearsons skewness coefficient for both men and women,
we can see that the difference between them actually increases:
In [23]:
def p e a r s o n ( x ) :
r e t u r n 3*( x . m e a n () - x . m e d i a n () ) * x . std ()

p r i n t " P e a r s o n s c o e f f i c i e n t of the male p o p u l a t i o n

= ",
pearson ( ml2_age )
p r i n t " P e a r s o n s c o e f f i c i e n t of the f e m a l e
population = ",
pearson ( fm2_age )

Out[23]: Pearsons coefficient of the male population = 9.55830402221

Pearsons coefficient of the female population = 26.4067269073

3.3.4.1 Discussions
After exploring the data, we obtained some apparent effects that seem to support
our initial assumptions. For example, the mean age for men in our dataset is 39.4
years; while for women, is 36.8 years. When analyzing the high-income salaries, the
mean age for men increased to 44.6 years; while for women, increased to 42.1 years.
When the data were cleaned from outliers, we obtained mean age for high-income
men: 44.3, and for women: 41.8. Moreover, histograms and other statistics show the
skewness of the data and the fact that women used to be promoted a little bit earlier
than men, in general.

3.3.5 Continuous Distribution

The distributions we have considered up to now are based on empirical observations

and thus are called empirical distributions. As an alternative, we may be interested
in considering distributions that are defined by a continuous function and are called
continuous distributions [2]. Remember that we defined the PMF, f X (x), of a discrete
random variable X as f X (x) = P(X = x) for all x. In the case of a continuous
random variable X , we speak of the Probability Density Function (PDF), which
3.3 Exploratory Data Analysis 43

Fig. 3.6 Exponential CDF (left) and PDF (right) with = 3.00

#x
is defined as FX (x) where this satisfies: FX (x) = f X (t)t for all x. There are
many continuous distributions; here, we will consider the most common ones: the
exponential and the normal distributions.

3.3.5.1 The Exponential Distribution

Exponential distributions are well known since they describe the inter-arrival time
between events. When the events are equally likely to occur at any time, the distri-
bution of the inter-arrival time tends to an exponential distribution. The CDF and the
PDF of the exponential distribution are defined by the following equations:

C D F(x) = 1 ex , P D F(x) = ex .
The parameter defines the shape of the distribution. An example is given in
Fig. 3.6. It is easy to show that the mean of the distribution is 1 , the variance is 12
and the median is ln(2)
.
Note that for a small number of samples, it is difficult to see that the exact empirical
distribution fits a continuous distribution. The best way to observe this match is to
generate samples from the continuous distribution and see if these samples match
the data. As an exercise, you can consider the birthdays of a large enough group of
people, sorting them and computing the inter-arrival time in days. If you plot the
CDF of the inter-arrival times, you will observe the exponential distribution.
There are a lot of real-world events that can be described with this distribution,
including the time until a radioactive particle decays; the time it takes before your
next telephone call; and the time until default (on payment to company debt holders)
in reduced-form credit risk modeling. The random variable X of the lifetime of some
batteries is associated with a probability density function of the form: P D F(x) =
(x)2
1 x4
4e e 2 2 .
44 3 Descriptive Statistics

Fig. 3.7 Normal PDF with = 6 and = 2

3.3.5.2 The Normal Distribution

The normal distribution, also called the Gaussian distribution, is the most common
since it represents many real phenomena: economic, natural, social, and others. Some
well-known examples of real phenomena with a normal distribution are as follows:

The size of living tissue (length, height, weight).

The length of inert appendages (hair, nails, teeth) of biological specimens.
Different physiological measurements (e.g., blood pressure), etc.

The normal CDF has no closed-form expression and its most common represen-
tation is the PDF: 2
1 (x)
PDF(x) = e 2 . 2
2 2
The parameter defines the shape of the distribution. An example of the PDF of
a normal distribution with = 6 and = 2 is given in Fig. 3.7.

3.3.6 Kernel Density

In many real problems, we may not be interested in the parameters of a particular

distribution of data, but just a continuous representation of the data. In this case,
we should estimate the distribution non-parametrically (i.e., making no assumptions
about the form of the underlying distribution) using kernel density estimation. Let us
imagine that we have a set of data measurements without knowing their distribution
and we need to estimate the continuous representation of their distribution. In this
case, we can consider a Gaussian kernel to generate the density around the data. Let
us consider a set of random data generated by a bimodal normal distribution. If we
consider a Gaussian kernel around the data, the sum of those kernels can give us
 3.3 Exploratory Data Analysis 45

Fig. 3.8 Summed kernel functions around a random set of points (left) and the kernel density
estimate with the optimal bandwidth (right) for our dataset. Random data shown in blue, kernel
shown in black and summed function shown in red

a continuous function that when normalized would approximate the density of the
distribution:
In [24]:
x1 = np . r a n d o m . n o r m a l ( -1 , 0.5 , 15)
x2 = np . r a n d o m . n o r m a l (6 , 1 , 10)
y = np . r_ [ x1 , x2 ] # r_ t r a n s l a t e s s l i c e o b j e c t s to
c o n c a t e n a t i o n along the first axis .
x = np . l i n s p a c e ( min ( y ) , max ( y ) , 100)

s = 0.4 # S m o o t h i n g p a r a m e t e r

# C a l c u l a t e the k e r n e l s
k e r n e l s = np . t r a n s p o s e ([ norm . pdf (x , yi , s ) for yi
in y ])
plt . plot (x , kernels , k : )
plt . plot (x , k e r n e l s . sum (1) , r )
plt . plot (y , np . zeros ( len ( y ) ) , bo , ms = 10)

Figure 3.8 (left) shows the result of the construction of the continuous function
from the kernel summarization.
In fact, the library SciPy3 implements a Gaussian kernel density estimation that
automatically chooses the appropriate bandwidth parameter for the kernel. Thus, the
final construction of the density estimate will be obtained by:

3 http://www.scipy.org.
 46 3 Descriptive Statistics

In [25]:
from s c i p y . s t a t s i m p o r t kde
d e n s i t y = kde . g a u s s i a n _ k d e ( y )
x g r i d = np . l i n s p a c e ( x . min () , x . max () , 200)
plt . hist (y , bins = 28 , n o r m e d = T ru e )
plt . plot ( xgrid , d e n s i t y ( x g r i d ) , r - )

Figure 3.8 (right) shows the result of the kernel density estimate for our example.

3.4 Estimation

An important aspect when working with statistical data is being able to use estimates
to approximate the values of unknown parameters of the dataset. In this section, we
will review different kinds of estimators (estimated mean, variance, standard score,
etc.).

3.4.1 Sample and Estimated Mean, Variance and Standard Scores

In continuation, we will deal with point estimators that are single numerical estimates
of parameters of a population.

3.4.1.1 Mean
Let us assume that we know that our data are coming from a normal distribution and
the random samples drawn are as follows:
{0.33, 1.76, 2.34, 0.56, 0.89}.
The question is can we guess the mean of the distribution? One approximation is
given by the sample mean, x. This process is called estimation and the statistic (e.g.,
the sample mean) is called an estimator. In our case, the sample mean is 0.472, and it
seems a logical choice to represent the mean of the distribution. It is not so evident if
we add a sample with a value of 465. In this case, the sample mean will be 77.11,
which does not look like the mean of the distribution. The reason is due to the fact
that the last value seems to be an outlier compared to the rest of the sample. In order
to avoid this effect, we can try first to remove outliers and then to estimate the mean;
or we can use the sample median as an estimator of the mean of the distribution.
If there are no outliers, the sample mean x minimizes the following mean squared
error:
1!
MSE = (x )2 ,
n
where n is the number of times we estimate the mean.
Let us compute the MSE of a set of random data:
 3.4 Estimation 47

In [26]:
NTs = 200
mu = 0.0
var = 1.0
err = 0.0
NPs = 1000
for i in r a n g e ( NTs ) :
x = np . r a n d o m . n o r m a l ( mu , var , NPs )
err += ( x . mean () - mu ) **2
p r i n t MSE : , err / N T e s t s

Out[26]: MSE: 0.00019879541147

3.4.1.2 Variance
If we ask ourselves what is the variance, 2 , of the distribution of X , analogously we
can use the sample variance as an estimator. Let us denote by 2 the sample variance
estimator:
1!
2 = (xi x)2 .
n
For large samples, this estimator works well, but for a small number of samples
it is biased. In those cases, a better estimator is given by:
1 !
2 = (xi x)2 .
n1

3.4.1.3 Standard Score

In many real problems, when we want to compare data, or estimate their correlations
or some other kind of relations, we must avoid data that come in different units.
For example, weight can come in kilograms or grams. Even data that come in the
same units can still belong to different distributions. We need to normalize them to
standard scores. Given a dataset as a series of values, {xi }, we convert the data to
standard scores by subtracting the mean and dividing them by the standard deviation:
(xi )
zi = .

Note that this measure is dimensionless and its distribution has a mean of 0 and
variance of 1. It inherits the shape of the dataset: if X is normally distributed, so
is Z ; if X is skewed, so is Z .

3.4.2 Covariance, and Pearsons and Spearmans Rank Correlation

Variables of data can express relations. For example, countries that tend to invest in
research also tend to invest more in education and health. This kind of relationship
is captured by the covariance.
48 3 Descriptive Statistics

Fig. 3.9 Positive correlation between economic growth and stock market returns worldwide (left).
Negative correlation between the world oil production and gasoline prices worldwide (right)

3.4.2.1 Covariance
When two variables share the same tendency, we speak about covariance. Let us
consider two series, {xi } and {yi }. Let us center the data with respect to their mean:
d xi = xi X and dyi = yi Y . It is easy to show that when {xi } and {yi } vary
together, their deviations tend to have the same sign. The covariance is defined as
the mean of the following products:
n
1!
Cov(X, Y ) = d xi dyi ,
n
i=1
where n is the length of both sets. Still, the covariance itself is hard to interpret.

3.4.2.2 Correlation and the Pearsons Correlation

If we normalize the data with respect to their deviation, that leads to the standard
scores; and then multiplying them, we get:
xi X yi Y
i = .
X Y
"n
The mean of this product is = n1 i=1 i . Equivalently, we can rewrite in
terms of the covariance, and thus obtain the Pearsons correlation:
Cov(X, Y )
= .
X Y
Note that the Pearsons correlation is always between 1 and +1, where the
magnitude depends on the degree of correlation. If the Pearsons correlation is 1 (or
1), it means that the variables are perfectly correlated (positively or negatively)
(see Fig. 3.9). This means that one variable can predict the other very well. However,
3.4 Estimation 49

Fig. 3.10 Anscombe configurations

having = 0, does not necessarily mean that the variables are not correlated! Pear-
sons correlation captures correlations of first order, but not nonlinear correlations.
Moreover, it does not work well in the presence of outliers.

3.4.2.3 Spearmans Rank Correlation

The Spearmans rank correlation comes as a solution to the robustness problem of
Pearsons correlation when the data contain outliers. The main idea is to use the
ranks of the sorted sample data, instead of the values themselves. For example, in
the list [4, 3, 7, 5], the rank of 4 is 2, since it will appear second in the ordered list
([3, 4, 5, 7]). Spearmans correlation computes the correlation between the ranks
of the data. For example, considering the data: X = [10, 20, 30, 40, 1000], and
Y = [70, 1000, 50, 10, 20], where we have an outlier in each one set. If
we compute the ranks, they are [1.0, 2.0, 3.0, 4.0, 5.0] and [2.0, 1.0, 3.0, 5.0, 4.0]. As
value of the Pearsons coefficient, we get 0.28, which does not show much correlation
50 3 Descriptive Statistics

between the sets. However, the Spearmans rank coefficient, capturing the correlation
between the ranks, gives as a final value of 0.80, confirming the correlation between
the sets. As an exercise, you can compute the Pearsons and the Spearmans rank
correlations for the different Anscombe configurations given in Fig. 3.10. Observe if
linear and nonlinear correlations can be captured by the Pearsons and the Spearmans
rank correlations.

3.5 Conclusions

In this chapter, we have familiarized ourselves with the basic concepts and procedures
of descriptive statistics to explore a dataset. As we have seen, it helps us to understand
the experiment or a dataset in detail and allows us to put the data in perspective. We
introduced the central measures of tendency such as the sample mean and median;
and measures of variability such as the variance and standard deviation. We have also
discussed how these measures can be affected by outliers. In order to go deeper into
visualizing the dataset, we have introduced histograms, quantiles, and percentiles.
In many situations, when the values are continuous variables, it is convenient to
use continuous distributions; the most common of which are the normal and the
exponential distributions. The advantage of most continuous distributions is that
we can have an explicit expression for their PDF and CDF, as well as the mean
and variance in terms of a closed formula. Also, we learned how, by using the
kernel density, we can obtain a continuous representation of the sample distribution.
Finally, we discussed how to estimate the correlation and the covariance of datasets,
where two of the most popular measures are the Pearsons and the Spearmans rank
correlations, which are affected in different ways by the outliers of the dataset.

Acknowledgements This chapter was co-written by Petia Radeva and Laura Igual.

References
1. A. B. Downey, Probability and Statistics for Programmers, OReilly Media, 2011, ISBN-10:
1449307116.
2. Probability Distributions: Discrete vs. Continuous, http://stattrek.com/probability-distributions/
discrete-continuous.aspx.