Chapter 8

Data Analysis with Pandas

About Pandas
• Pandas is a Python library used for working with data
sets.
• It has functions for analyzing, cleaning, exploring, and
manipulating data.
• The name "Pandas" has a reference to both "Panel
Data", and "Python Data Analysis" and was created by
Wes McKinney in 2008.
Why use Pandas?
• Pandas allows us to analyze big data and make
conclusions based on statistical theories.
• Pandas can clean messy data sets, and make them
readable and relevant.
• Relevant data is very important in data science.
What Can Pandas Do?
• Pandas gives you answers about the data. Like:
• Is there a correlation between two or more columns?
• What is average value?
• Max value?
• Min value?
• Pandas are also able to delete rows that are not
relevant, or contains wrong values, like empty or NULL
values. This is called cleaning the data.
Example 1:

• import pandas

mydataset = {
'cars':
["BMW", "Volvo", "Ford"],
'passings': [3, 7, 2]
}

myvar =
pandas.DataFrame(mydataset)

print(myvar)
Pandas as pd

• import pandas as pd

mydataset = {
'cars':
["BMW", "Volvo", "Ford"],
'passings': [3, 7, 2]
}

myvar =
pd.DataFrame(mydataset)

print(myvar)
Pandas Series
• A Pandas Series is like a column in a table.
• It is a one-dimensional array holding data of any type.
Example: Create a simple Pandas Series from a list:
import pandas as pd

a = [1, 7, 2]

myvar = pd.Series(a)

print(myvar)
Labels
• If nothing else is specified, the values are labeled with
their index number. First value has index 0, second
value has index 1 etc.
• This label can be used to access a specified value.
• Example
• Return the first value of the Series:
print(myvar[0])
Create Labels
• With the index argument, you can name your own labels.
Example: Create your own labels:
import pandas as pd
a = [1, 7, 2]
myvar = pd.Series(a, index = ["x", "y", "z"])
print(myvar)
• When you have created labels, you can access an item by referring to
the label.
Example
Return the value of "y":
print(myvar["y"])
Key/Value Objects as Series
• You can also use a key/value object, like a dictionary, when creating a
Series.

Example: Create a simple Pandas Series from a dictionary:

import pandas as pd
calories = {"day1": 420, "day2": 380, "day3": 390}
myvar = pd.Series(calories)
print(myvar)
Key/Value Objects as Series
• To select only some of the items in the dictionary, use the index argument
and specify only the items you want to include in the Series.
Example: Create a Series using only data from "day1" and "day2":

import pandas as pd

calories = {"day1": 420, "day2": 380, "day3": 390}

myvar = pd.Series(calories, index = ["day1", "day2"])

print(myvar)
Data Frames

• Data sets in Pandas are usually multi-

dimensional tables, called DataFrames.
• Series is like a column, a DataFrame is the
whole table.
• Example: Create a DataFrame from two Series:
import pandas as pd
data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}
myvar = pd.DataFrame(data)
print(myvar)
Data Frames
• A Pandas DataFrame is a 2-dimensional data structure, like a
2-dimensional array, or a table with rows and columns.
• Example: Create a simple Pandas DataFrame:
import pandas as pd

data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}

#load data into a DataFrame object:

df = pd.DataFrame(data)

print(df)
Locate Row
• As you can see from the result above, the DataFrame is like a table with
rows and columns.
• Pandas use the loc attribute to return one or more specified row(s)

• Example: Return row 0:

• #refer to the row index:
• print(df.loc[0])
• Example: Return row 0 and 1:
#use a list of indexes:
print(df.loc[[0, 1]])
Named Indexes
• With the index argument, you can name your own indexes.
• Example: Add a list of names to give each row a name:

import pandas as pd
data = {
"calories": [420, 380, 390],
"duration": [50, 40, 45]
}
df = pd.DataFrame(data, index = ["day1", "day2", "day3"])
print(df)
Locate Named Indexes
• Use the named index in the loc attribute to return the specified
row(s).
• Example: Return "day2"
#refer to the named index:
print(df.loc["day2"])
Load Files Into a DataFrame
• If your data sets are stored in a file, Pandas can load
them into a DataFrame.
• Example: Load a comma separated file (CSV file) into a
DataFrame:
import pandas as pd

df = pd.read_csv('data.csv')

print(df)
Pandas Read CSV
• A simple way to store big data sets is to use CSV files (comma
separated files).
• CSV files contains plain text and is a well know format that can be
read by everyone including Pandas.
• In our examples we will be using a CSV file called 'data.csv’.
• Load the CSV into a DataFrame:
import pandas as pd

df = pd.read_csv('data.csv')

print(df.to_string())
Pandas Read CSV
• If you have a large DataFrame with many rows, Pandas will only
return the first 5 rows, and the last 5 rows:
• Example: Print the DataFrame without the to_string() method:

import pandas as pd
df = pd.read_csv('data.csv')
print(df)
max_rows
• The number of rows returned is defined in Pandas option settings.
• You can check your system's maximum rows with the
pd.options.display.max_rows statement.
• Example
Check the number of maximum returned rows:

import pandas as pd

print(pd.options.display.max_rows)

• In my system the number is 60, which means that if the DataFrame contains
more than 60 rows, the print(df) statement will return only the headers and the
first and last 5 rows.
• You can change the maximum rows number with the same statement.
• Example
• Increase the maximum number of rows to display the entire DataFrame:
import pandas as pd

pd.options.display.max_rows = 9999

df = pd.read_csv('data.csv')

print(df)
Pandas Read JSON
• Big data sets are often stored, or extracted as JSON.
• JSON is plain text, but has the format of an object, and is well known in
the world of programming, including Pandas.
• In our examples we will be using a JSON file called 'data.json’.
• Example
• Load the JSON file into a DataFrame:
import pandas as pd

df = pd.read_json('data.json')

print(df.to_string())
Dictionary as JSON
• JSON objects have the same format as Python dictionaries.
• If your JSON code is not in a file, but in a Python Dictionary, you can
load it into a DataFrame directly:
• Example
• Load a Python Dictionary into a DataFrame:
• import pandas as pd

data = {
"Duration":{
"0":60,
"1":60,
"2":60,
"3":45,
"4":45,
"5":60
},
"Pulse":{
"0":110,
"1":117,
"2":103,
"3":109,
"4":117,
"5":102
},
"Maxpulse":{
"0":130,
"1":145,
"2":135,
"3":175,
"4":148,
"5":127
},
"Calories":{
"0":409,
"1":479,
"2":340,
"3":282,
"4":406,
"5":300
}
}

df = pd.DataFrame(data)

print(df)
Pandas - Analyzing DataFrames: Viewing
the Data
• One of the most used method for getting a quick overview of the DataFrame,
is the head() method.
• The head() method returns the headers and a specified number of rows,
starting from the top.
• Example
Get a quick overview by printing the first 10 rows of the DataFrame:

import pandas as pd

df = pd.read_csv('data.csv')

print(df.head(10))
• There is also a tail() method for viewing the last rows of the
DataFrame.
• The tail() method returns the headers and a specified number of
rows, starting from the bottom.
• Example
Print the last 5 rows of the DataFrame:
print(df.tail())
Info About the Data
• The DataFrames object has a method called info(), that gives you
more information about the data set.
• Example
Print information about the data:

print(df.info())
Pandas - Cleaning Data
• Data cleaning means fixing bad data in your data set.
• Bad data could be:
• Empty cells
• Data in wrong format
• Wrong data
• Duplicates
Pandas - Cleaning Empty Cells
• Empty cells can potentially give you a wrong result when you analyze data.
• One way to deal with empty cells is to remove rows that contain empty cells.
• This is usually OK, since data sets can be very big, and removing a few rows will not have
a big impact on the result.
• Example: Return a new Data Frame with no empty cells:

import pandas as pd

df = pd.read_csv('data2.csv')

new_df = df.dropna()

print(new_df.to_string())
• If you want to change the original DataFrame, use the inplace = True
argument:
• Example: Remove all rows with NULL values:

import pandas as pd

df = pd.read_csv('data2.csv')

df.dropna(inplace = True)

print(df.to_string())
Replace Empty Values
• Another way of dealing with empty cells is to insert a new value instead.
• This way you do not have to delete entire rows just because of some empty cells.
• The fillna() method allows us to replace empty cells with a value:
• Example
#Replace NULL values with the number 130:

import pandas as pd

df = pd.read_csv('data2.csv')

df.fillna(130, inplace = True)

print(df.to_string())
• Replace Only For Specified Columns
• The example above replaces all empty cells in the whole Data Frame.
• To only replace empty values for one column, specify the column name for the
DataFrame
• Example
Replace NULL values in the "Calories" columns with the number 130:

import pandas as pd

df = pd.read_csv('data.csv')

df["Calories"].fillna(130, inplace = True)

Replace Using Mean, Median, or Mode
• A common way to replace empty cells, is to calculate the mean, median or mode value of
the column.
• Pandas uses the mean() median() and mode() methods to calculate the respective values
for a specified column:
• Example
#Calculate the MEAN, and replace any empty values with it:

import pandas as pd

df = pd.read_csv('data.csv')

x = df["Calories"].mean()

df["Calories"].fillna(x, inplace = True)

• Example
#Calculate the MEDIAN, and replace any empty values with it:

import pandas as pd

df = pd.read_csv('data.csv')

x = df["Calories"].median()

df["Calories"].fillna(x, inplace = True)

Calculate the MODE, and replace any empty values with it:

import pandas as pd

df = pd.read_csv('data.csv')

x = df["Calories"].mode()[0]

df["Calories"].fillna(x, inplace = True)

Pandas - Cleaning Data of Wrong Format
• Cells with data of wrong format can make it difficult, or
even impossible, to analyze data.
• To fix it, you have two options: remove the rows, or
convert all cells in the columns into the same format.
Convert Into a Correct Format
• In data3.csv, check out row 22 and 26, the 'Date' column should be a string that
represents a date:
• Let's try to convert all cells in the 'Date' column into dates.
• Pandas has a to_datetime() method for this:
• Example: Convert to date
import pandas as pd

df = pd.read_csv('data3.csv')

df['Date'] = pd.to_datetime(df['Date'])

print(df.to_string())
Removing Rows
• The result from the converting in the example above gave us a NaT
value, which can be handled as a NULL value, and we can remove the
row by using the dropna() method.

• Example: Remove rows with a NULL value in the "Date" column:

df.dropna(subset=['Date'], inplace = True)

Pandas - Fixing Wrong Data
• "Wrong data" does not have to be "empty cells" or "wrong format", it can just be wrong, like
if someone registered "199" instead of "1.99".

• Sometimes you can spot wrong data by looking at the data set, because you have an
expectation of what it should be.

• If you take a look at our data set, you can see that in row 7, the duration is 450, but for all the
other rows the duration is between 30 and 60.

• It doesn't have to be wrong, but taking in consideration that this is the data set of someone's
workout sessions, we conclude with the fact that this person did not work out in 450
minutes.
• How can we fix wrong values, like the one for "Duration" in row 7?
Replacing Values
• One way to fix wrong values is to replace them with something else.
• In our example, it is most likely a typo, and the value should be "45"
instead of "450", and we could just insert "45" in row 7:
• Example: Set "Duration" = 45 in row 7:
df.loc[7, 'Duration'] = 45
• For small data sets you might be able to replace the wrong data one
by one, but not for big data sets.
• To replace wrong data for larger data sets you can create some rules,
e.g. set some boundaries for legal values, and replace any values that
are outside of the boundaries.
Replacing Values
Example: Loop through all values in the "Duration" column.
• If the value is higher than 100, set it to 60:
import pandas as pd
df = pd.read_csv('data2.csv')
for x in df.index:
if df.loc[x, "Duration"] > 100:
df.loc[x, "Duration"] = 60
print(df.to_string())
Removing Rows
• Another way of handling wrong data is to remove the rows that
contains wrong data.
• This way you do not have to find out what to replace them with, and
there is a good chance you do not need them to do your analyses.
• Example: Delete rows where "Duration" is higher than 120:
for x in df.index:
if df.loc[x, "Duration"] > 120:
df.drop(x, inplace = True)
Pandas - Removing Duplicates
• Duplicate rows are rows that have been registered more than one time.
• By taking a look at our test data set, we can assume that row 11 and 12
are duplicates.

• To discover duplicates, we can use the duplicated() method.

• The duplicated() method returns a Boolean values for each row:
• Example: Returns True for every row that is a duplicate, othwerwise False:
print(df.duplicated())
• To remove duplicates, use the drop_duplicates() method.

• Example: Remove all duplicates:

df.drop_duplicates(inplace = True)
Pandas - Data Correlations (Finding Relationship)

• A great aspect of the Pandas

module is the corr() method.
• The corr() method calculates the
relationship between each column
in your data set.
import pandas as pd
df = pd.read_csv('data.csv')
df.corr()
print(df.corr())
Pandas - Plotting

• Pandas uses the plot() method to create

diagrams.
• We can use Pyplot, a submodule of the
Matplotlib library to visualize the
diagram on the screen.
• Example: Import pyplot from Matplotlib
and visualize our DataFrame:

import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv('data.csv')
df.plot()
plt.show()
Scatter Plot

• Specify that you want a scatter plot with the kind

argument:
kind = 'scatter'
• A scatter plot needs an x- and a y-axis.
• In the example below we will use "Duration" for
the x-axis and "Calories" for the y-axis.
• Include the x and y arguments like this:
x = 'Duration', y = 'Calories'

Example:
import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv('data.csv')
df.plot(kind = 'scatter', x = 'Duration', y =
'Calories')
plt.show()
• Example: A scatterplot where
there are no relationship between
the columns:
import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_csv('data.csv')
df.plot(kind = 'scatter', x =
'Duration', y = 'Maxpulse')
plt.show()
Histogram

• Use the kind argument to specify

that you want a histogram:
kind = 'hist'
• A histogram needs only one column.
• A histogram shows us the frequency
of each interval, e.g. how many
workouts lasted between 50 and 60
minutes?
• In the example below we will use
the "Duration" column to create the
histogram:
• Example
df["Duration"].plot(kind = 'hist')