Pandas

1
Contents:
 Intro --- page 3
 Getting started --- page 4
 Pandas series --- page 7
 Pandas dataframes --- page 10
 Pandas read CSV --- page 14
 Pandas read JSON --- page 16
 Pandas analyzing data --- page 20
 Pandas data cleaning --- page 21
 Pandas empty cells --- page 24
 Pandas wrong format --- page 29
 Pandas wrong data --- page 31
 Pandas duplicates --- page 34
 Pandas corralations --- page 36
 Pandas plotting --- page 39
 W
 W
 W
 W
 W
 W
 W
 W
 W

2
Intro

What is Pandas?
Pandas is a Python library used for working with data sets.
It has functions for analyzing, cleaning, exploring, and
manipulating data.
The name "Pandas" has a reference to both "Panel Data", and
"Python Data Analysis" and was created by Wes McKinney in
2008.

Why Use Pandas?

Pandas allows us to analyze big data and make conclusions
based on statistical theories.
Pandas can clean messy data sets, and make them readable
and relevant.
Relevant data is very important in data science.

Data Science: is a branch of computer science where we study

how to store, use and analyze data for deriving information
from it.

3
 What Can Pandas Do?
Pandas gives you answers about the data. Like:

Is there a correlation between two or more columns?

 What is average value?
 Max value?
 Min value?

Pandas are also able to delete rows that are not relevant, or
contains wrong values, like empty or NULL values. This is called
cleaning the data.

Getting started

Import Pandas
Once Pandas is installed, import it in your applications by
adding the import keyword:

4
import pandas

Now Pandas is imported and ready to use.

import pandas

mydataset = {
'cars': ["BMW", "Volvo", "Ford"],
'passings': [3, 7, 2]
}

myvar = pandas.DataFrame(mydataset)
print(myvar)

Pandas as pd
Pandas is usually imported under the pd alias.
alias: In Python alias are an alternate name for referring to the
same thing.
Create an alias with the as keyword while importing:
5
import pandas as pd

mydataset = {
'cars': ["BMW", "Volvo", "Ford"],
'passings': [3, 7, 2]
}

myvar = pd.DataFrame(mydataset)
print(myvar)

Checking Pandas Version

The version string is stored under __version__ attribute.

import pandas as pd
print(pd.__version__)

6
Panda series

What is a Series?
A Pandas Series is like a column in a table.
It is a one-dimensional array holding data of any type.
Create a simple Pandas Series from a list:

import pandas as pd

a = [1, 7, 2]

myvar = pd.Series(a)
print(myvar)

Labels
If nothing else is specified, the values are labeled with their
index number. First value has index 0, second value has index 1
etc.
This label can be used to access a specified value.
7
Return the first value of the Series:

print(myvar[0])

Create Labels
With the index argument, you can name your own labels.
Create your own labels:

import pandas as pd

a = [1, 7, 2]

myvar = pd.Series(a, index = ["x", "y", "z"])

print(myvar)

When you have created labels, you can access an item by

referring to the label.
Return the value of "y":

8
print(myvar["y"])

Key/Value Objects as Series

You can also use a key/value object, like a dictionary, when
creating a Series.
Create a simple Pandas Series from a dictionary:

import pandas as pd

calories = {"day1": 420, "day2": 380, "day3": 390}

myvar = pd.Series(calories)
print(myvar)

DataFrames
Data sets in Pandas are usually multi-dimensional tables, called
DataFrames.
Series is like a column, a DataFrame is the whole table.
Create a DataFrame from two Series:
9
import pandas as pd

data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}

myvar = pd.DataFrame(data)
print(myvar)

Pandas dataframes

What is a DataFrame?
A Pandas DataFrame is a 2 dimensional data structure, like a 2
dimensional array, or a table with rows and columns.
Create a simple Pandas DataFrame:

10
import pandas as pd

data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}

#load data into a DataFrame object:

df = pd.DataFrame(data)
print(df)

Locate Row
As you can see from the result above, the DataFrame is like a
table with rows and columns.
Return row 0:

#refer to the row index:

print(df.loc[0])

11
Return row 0 and 1:

#use a list of indexes:

print(df.loc[[0, 1]])

Named Indexes
With the index argument, you can name your own indexes.
Add a list of names to give each row a name:

import pandas as pd

data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}

df = pd.DataFrame(data, index = ["day1", "day2", "day3"])

print(df)

12
 Locate Named Indexes
Use the named index in the loc attribute to return the specified
row(s).
Return "day2":

#refer to the named index:

print(df.loc["day2"])

Load Files Into a DataFrame

If your data sets are stored in a file, Pandas can load them into
a DataFrame.
Load a comma separated file (CSV file) into a DataFrame:

import pandas as pd

df = pd.read_csv('data.csv')
print(df)

13
Pandas read CSV

Read CSV Files

A simple way to store big data sets is to use CSV files (comma
separated files).
CSV files contains plain text and is a well know format that can
be read by everyone including Pandas.
In our examples we will be using a CSV file called 'data.csv'.
Load the CSV into a DataFrame:

import pandas as pd

df = pd.read_csv('data.csv')
print(df.to_string())

If you have a large DataFrame with many rows, Pandas will only
return the first 5 rows, and the last 5 rows:
Print the DataFrame without the to_string() method:

14
import pandas as pd

df = pd.read_csv('data.csv')
print(df)

max_rows
The number of rows returned is defined in Pandas option
settings.
You can check your system's maximum rows with the
pd.options.display.max_rows statement.
Check the number of maximum returned rows:

import pandas as pd
print(pd.options.display.max_rows)

In my system the number is 60, which means that if the

DataFrame contains more than 60 rows, the print(df) statement
will return only the headers and the first and last 5 rows.

15
You can change the maximum rows number with the same
statement.
Increase the maximum number of rows to display the entire
DataFrame:

import pandas as pd
pd.options.display.max_rows = 9999

df = pd.read_csv('data.csv')
print(df)

Pandas read JSON

Read JSON
Big data sets are often stored, or extracted as JSON.
JSON is plain text, but has the format of an object, and is well
known in the world of programming, including Pandas.
In our examples we will be using a JSON file called 'data.json'.

16
Load the JSON file into a DataFrame:

import pandas as pd

df = pd.read_json('data.json')
print(df.to_string())

Dictionary as JSON

JSON = Python Dictionary

JSON objects have the same format as Python dictionaries.
If your JSON code is not in a file, but in a Python Dictionary, you
can load it into a DataFrame directly:
Load a Python Dictionary into a DataFrame:

import pandas as pd

data = {
"Duration":{
17
 "0":60,
"1":60,
"2":60,
"3":45,
"4":45,
"5":60
},
"Pulse":{
"0":110,
"1":117,
"2":103,
"3":109,
"4":117,
"5":102
},
"Maxpulse":{
"0":130,
"1":145,

18
 "2":135,
"3":175,
"4":148,
"5":127
},
"Calories":{
"0":409,
"1":479,
"2":340,
"3":282,
"4":406,
"5":300
}
}

df = pd.DataFrame(data)
print(df)

19
Pandas analyzing data

Viewing the Data

One of the most used method for getting a quick overview of
the DataFrame, is the head() method.
The head() method returns the headers and a specified number
of rows, starting from the top.
Get a quick overview by printing the first 10 rows of the
DataFrame:

import pandas as pd

df = pd.read_csv('data.csv')
print(df.head(10))

There is also a tail() method for viewing the last rows of the
DataFrame.
The tail() method returns the headers and a specified number
of rows, starting from the bottom.
Print the last 5 rows of the DataFrame:
20
print(df.tail(5))

Info About the Data

The DataFrames object has a method called info(), that gives
you more information about the data set.
Print information about the data:

print(df.info())

Pandas cleaning

Data Cleaning
Data cleaning means fixing bad data in your data set.

Bad data could be:

 Empty cells
 Data in wrong format
21
  Wrong data
 Duplicates

In this tutorial you will learn how to deal with all of them.

Our Data Set

In the next chapters we will use this data set:

Duration Date Pulse Maxpulse Calories

0 60 '2020/12/01' 110 130 409.1

1 60 '2020/12/02' 117 145 479.0

2 60 '2020/12/03' 103 135 340.0

3 45 '2020/12/04' 109 175 282.4

4 45 '2020/12/05' 117 148 406.0

5 60 '2020/12/06' 102 127 300.0

6 60 '2020/12/07' 110 136 374.0

7 450 '2020/12/08' 104 134 253.3

8 30 '2020/12/09' 109 133 195.1

9 60 '2020/12/10' 98 124 269.0

22
10 60 '2020/12/11' 103 147 329.3

11 60 '2020/12/12' 100 120 250.7

12 60 '2020/12/12' 100 120 250.7

13 60 '2020/12/13' 106 128 345.3

14 60 '2020/12/14' 104 132 379.3

15 60 '2020/12/15' 98 123 275.0

16 60 '2020/12/16' 98 120 215.2

17 60 '2020/12/17' 100 120 300.0

18 45 '2020/12/18' 90 112 NaN

19 60 '2020/12/19' 103 123 323.0

20 45 '2020/12/20' 97 125 243.0

21 60 '2020/12/21' 108 131 364.2

22 45 NaN 100 119 282.0

23 60 '2020/12/23' 130 101 300.0

24 45 '2020/12/24' 105 132 246.0

25 60 '2020/12/25' 102 126 334.5

26 60 2020/12/26 100 120 250.0

27 60 '2020/12/27' 92 118 241.0

28 60 '2020/12/28' 103 132 NaN

29 60 '2020/12/29' 100 132 280.0

30 60 '2020/12/30' 102 129 380.3

31 60 '2020/12/31' 92 115 243.0

23
  The data set contains some empty cells ("Date" in row 22,
and "Calories" in row 18 and 28).
 The data set contains wrong format ("Date" in row 26).
 The data set contains wrong data ("Duration" in row 7).
 The data set contains duplicates (row 11 and 12).

Empty Cells

Empty cells can potentially give you a wrong result when you
analyze data.

Remove Rows
One way to deal with empty cells is to remove rows that
contain empty cells.

This is usually OK, since data sets can be very big, and removing
a few rows will not have a big impact on the result.
Return a new Data Frame with no empty cells:

24
import pandas as pd

df = pd.read_csv('data.csv')

new_df = df.dropna()
print(new_df.to_string())

Replace Empty Values

Another way of dealing with empty cells is to insert a new value
instead.
This way you do not have to delete entire rows just because of
some empty cells.

The fillna() method allows us to replace empty cells with a

value:
Replace NULL values with the number 130:

import pandas as pd

25
df = pd.read_csv('data.csv')
df.fillna(130, inplace = True)

Replace Only For Specified Columns

The example above replaces all empty cells in the whole Data
Frame.
To only replace empty values for one column, specify the
column name for the DataFrame:
Replace NULL values in the "Calories" columns with the number
130:

import pandas as pd

df = pd.read_csv('data.csv')
df["Calories"].fillna(130, inplace = True)

Replace Using Mean, Median, or Mode

A common way to replace empty cells, is to calculate the mean,
median or mode value of the column.

26
Pandas uses the mean() median() and mode() methods to
calculate the respective values for a specified column:

Calculate the MEAN, and replace any empty values with it:

import pandas as pd

df = pd.read_csv('data.csv')
x = df["Calories"].mean()

df["Calories"].fillna(x, inplace = True)

Mean = the average value (the sum of all values divided by

number of values).

Calculate the MEDIAN, and replace any empty values with it:

import pandas as pd

27
df = pd.read_csv('data.csv')
x = df["Calories"].median()

df["Calories"].fillna(x, inplace = True)

Median = the value in the middle, after you have sorted all
values ascending.

Calculate the MODE, and replace any empty values with it:

import pandas as pd

df = pd.read_csv('data.csv')
x = df["Calories"].mode()[0]

df["Calories"].fillna(x, inplace = True)

Mode = the value that appears most frequently.

28
Pandas wrong format

Data of Wrong Format

Cells with data of wrong format can make it difficult, or even
impossible, to analyze data.
To fix it, you have two options: remove the rows, or convert all
cells in the columns into the same format.

Convert Into a Correct Format

In our Data Frame, we have two cells with the wrong format.
Check out row 22 and 26, the 'Date' column should be a string
that represents a date:

Duration Date Pulse Maxpulse Calories

21 60 '2020/12/21' 108 131 364.2

22 45 NaN 100 119 282.0

23 60 '2020/12/23' 130 101 300.0

24 45 '2020/12/24' 105 132 246.0

25 60 '2020/12/25' 102 126 334.5

26 60 20201226 100 120 250.0

29
Let's try to convert all cells in the 'Date' column into dates.
Pandas has a to_datetime() method for this:
Convert to date:

import pandas as pd

df = pd.read_csv('data.csv')
df['Date'] = pd.to_datetime(df['Date'])

print(df.to_string())

Removing Rows
The result from the converting in the example above gave us a
NaT value, which can be handled as a NULL value, and we can
remove the row by using the dropna() method.
Remove rows with a NULL value in the "Date" column:

df.dropna(subset=['Date'], inplace = True)

30
Pandas wrong data

Wrong Data
"Wrong data" does not have to be "empty cells" or "wrong
format", it can just be wrong, like if someone registered "199"
instead of "1.99".

Sometimes you can spot wrong data by looking at the data set,
because you have an expectation of what it should be.

If you take a look at our data set, you can see that in row 7, the
duration is 450, but for all the other rows the duration is
between 30 and 60.

It doesn't have to be wrong, but taking in consideration that

this is the data set of someone's workout sessions, we conclude
with the fact that this person did not work out in 450 minutes.

Duration Date Pulse Maxpulse Calories

0 60 '2020/12/01' 110 130 409.1

1 60 '2020/12/02' 117 145 479.0

31
2 60 '2020/12/03' 103 135 340.0

3 45 '2020/12/04' 109 175 282.4

4 45 '2020/12/05' 117 148 406.0

5 60 '2020/12/06' 102 127 300.0

6 60 '2020/12/07' 110 136 374.0

7 450 '2020/12/08' 104 134 253.3

8 30 '2020/12/09' 109 133 195.1

How can we fix wrong values, like the one for "Duration" in row
7?

Replacing Values
One way to fix wrong values is to replace them with something
else.

In our example, it is most likely a typo, and the value should be

"45" instead of "450", and we could just insert "45" in row 7.
Set "Duration" = 45 in row 7:

df.loc[7, 'Duration'] = 45

32
For small data sets you might be able to replace the wrong data
one by one, but not for big data sets.

To replace wrong data for larger data sets you can create some
rules, e.g. set some boundaries for legal values, and replace any
values that are outside of the boundaries.
Loop through all values in the "Duration" column.

If the value is higher than 120, set it to 120:

for x in df.index:
if df.loc[x, "Duration"] > 120:
df.loc[x, "Duration"] = 120

Removing Rows
Another way of handling wrong data is to remove the rows that
contains wrong data.
This way you do not have to find out what to replace them
with, and there is a good chance you do not need them to do
your analyses.

33
Delete rows where "Duration" is higher than 120:

for x in df.index:
if df.loc[x, "Duration"] > 120:
df.drop(x, inplace = True)

Pandas duplicates

Discovering Duplicates
Duplicate rows are rows that have been registered more than
one time.

Duration Date Pulse Maxpulse Calories

0 60 '2020/12/01' 110 130 409.1

1 60 '2020/12/02' 117 145 479.0

2 60 '2020/12/03' 103 135 340.0

3 45 '2020/12/04' 109 175 282.4

4 45 '2020/12/05' 117 148 406.0

34
5 60 '2020/12/06' 102 127 300.0

6 60 '2020/12/07' 110 136 374.0

7 450 '2020/12/08' 104 134 253.3

8 30 '2020/12/09' 109 133 195.1

9 60 '2020/12/10' 98 124 269.0

10 60 '2020/12/11' 103 147 329.3

11 60 '2020/12/12' 100 120 250.7

12 60 '2020/12/12' 100 120 250.7

By taking a look at our test data set, we can assume that row 11
and 12 are duplicates.
To discover duplicates, we can use the duplicated() method.
The duplicated() method returns a Boolean values for each row:
Returns True for every row that is a duplicate, othwerwise
False:

print(df.duplicated())

Removing Duplicates
To remove duplicates, use the drop_duplicates() method.

35
Remove all duplicates:

df.drop_duplicates(inplace = True)

Pandas corralations

Finding Relationships
A great aspect of the Pandas module is the corr() method.

The corr() method calculates the relationship between each

column in your data set.
The examples in this page uses a CSV file called: 'data.csv'.

Show the relationship between the columns:

df.corr()

36
Result

Duration Pulse Maxpulse Calories

Duration 1.000000 -0.155408 0.009403 0.922721

Pulse -0.155408 1.000000 0.786535 0.025120

Maxpulse 0.009403 0.786535 1.000000 0.203814

Calories 0.922721 0.025120 0.203814 1.000000

Result Explained
The Result of the corr() method is a table with a lot of numbers
that represents how well the relationship is between two
columns.
The number varies from -1 to 1.

1 means that there is a 1 to 1 relationship (a perfect

correlation), and for this data set, each time a value went up in
the first column, the other one went up as well.

0.9 is also a good relationship, and if you increase one value,

the other will probably increase as well.

37
-0.9 would be just as good relationship as 0.9, but if you
increase one value, the other will probably go down.

0.2 means NOT a good relationship, meaning that if one value

goes up does not mean that the other will.

What is a good correlation? It depends on the use, but I think it

is safe to say you have to have at least 0.6 (or -0.6) to call it a
good correlation.

Perfect Correlation:
We can see that "Duration" and "Duration" got the number
1.000000, which makes sense, each column always has a
perfect relationship with itself.

Good Correlation:
"Duration" and "Calories" got a 0.922721 correlation, which is a
very good correlation, and we can predict that the longer you
work out, the more calories you burn, and the other way
around: if you burned a lot of calories, you probably had a long
work out.

38
 Bad Correlation:
"Duration" and "Maxpulse" got a 0.009403 correlation, which is
a very bad correlation, meaning that we can not predict the
max pulse by just looking at the duration of the work out, and
vice versa.

Pandas plotting

Plotting
Pandas uses the plot() method to create diagrams.
We can use Pyplot, a submodule of the Matplotlib library to
visualize the diagram on the screen.
Import pyplot from Matplotlib and visualize our DataFrame:

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('data.csv')

39
df.plot()
plt.show()

Scatter Plot
Specify that you want a scatter plot with the kind argument:

kind = 'scatter'

A scatter plot needs an x- and a y-axis.

In the example below we will use "Duration" for the x-axis and
"Calories" for the y-axis.
Include the x and y arguments like this:
x = 'Duration', y = 'Calories'

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('data.csv')
40
df.plot(kind = 'scatter', x = 'Duration', y = 'Calories')
plt.show()

Histogram
Use the kind argument to specify that you want a histogram:

kind = 'hist'

A histogram needs only one column.

A histogram shows us the frequency of each interval, e.g. how
many workouts lasted between 50 and 60 minutes?
In the example below we will use the "Duration" column to
create the histogram:

df["Duration"].plot(kind = 'hist')

41