Phython Example
Phython Example
Libraries
For the simplicity of the article, we will use a single dataset. We will use the
employee data for this. It contains 8 columns namely – First Name, Gender, Start
Date, Last Login, Salary, Bonus%, Senior Management, and Team. We can get the
dataset here Employees.csv
Let’s read the dataset using the Pandas read_csv() function and print the 1st five
rows. To print the first five rows we will use the head() function.
Python3
import pandas as pd
import numpy as np
# read datasdet using pandas
df = pd.read_csv('employees.csv')
df.head()
Output:
Python3
df.shape
Output:
(1000, 8)
This means that this dataset has 1000 rows and 8 columns.
Let’s get a quick summary of the dataset using the pandas describe() method. The
describe() function applies basic statistical computations on the dataset like extreme
values, count of data points standard deviation, etc. Any missing value or NaN value
is automatically skipped. describe() function gives a good picture of the distribution
of data.
Example:
Python3
df.describe()
Output:
Note we can also get the description of categorical columns of the dataset if we
specify include =’all’ in the describe function.
Now, let’s also see the columns and their data types. For this, we will use
the info() method.
Python3
Output:
We can see the number of unique elements in our dataset. This will help us in
deciding which type of encoding to choose for converting categorical columns into
numerical columns.
Python3
df.nunique()
Output:
First Name 200
Gender 2
Start Date 972
Last Login Time 720
Salary 995
Bonus % 971
Senior Management 2
Team 10
dtype: int64
Till now we have got an idea about the dataset used. Now Let’s see if our dataset
contains any missing values or not.
You all must be wondering why a dataset will contain any missing values. It can
occur when no information is provided for one or more items or for a whole unit. For
Example, Suppose different users being surveyed may choose not to share their
income, and some users may choose not to share their address in this way many
datasets went missing. Missing Data is a very big problem in real-life scenarios.
Missing Data can also refer to as NA(Not Available) values in pandas. There are
several useful functions for detecting, removing, and replacing null values in Pandas
DataFrame :
isnull()
notnull()
dropna()
fillna()
replace()
interpolate()
Now let’s check if there are any missing values in our dataset or not.
Example:
Python3
df.isnull().sum()
Output:
We can see that every column has a different amount of missing values. Like Gender
has 145 missing values and salary has 0. Now for handling these missing values
there can be several cases like dropping the rows containing NaN or replacing NaN
with either mean, median, mode, or some other value.
Now, let’s try to fill in the missing values of gender with the string “No Gender”.
Example:
Python3
df.isnull().sum()
Output:
Null values in dataframe after filling Gender column
We can see that now there is no null value for the gender column. Now, Let’s fill the
senior management with the mode value.
Example:
Python3
df.isnull().sum()
Output:
Now for the first name and team, we cannot fill the missing values with arbitrary
data, so, let’s drop all the rows containing these missing values.
Example:
Python3
print(df.isnull().sum())
df.shape
Output:
Null values in dataframe after dropping all null values
We can see that our dataset is now free of all the missing values and after dropping
the data the number of rows also reduced from 1000 to 899.
Note: For more information, refer to Working with Missing Data in Pandas .
After removing the missing data let’s visualize our data.
Data Encoding
There are some models like Linear Regression which does not work with categorical
dataset in that case we should try to encode categorical dataset into the numerical
column. we can use different methods for encoding like Label encoding or One-hot
encoding. pandas and sklearn provide different functions for encoding in our case we
will use the LabelEncoding function from sklearn to encode the Gender column.
Python3
Noe
Data visualization
Data Visualization is the process of analyzing data in the form of graphs or maps,
making it a lot easier to understand the trends or patterns in the data.
Let’s see some commonly used graphs –
Note: We will use Matplotlib and Seaborn library for the data visualization. If you
want to know about these modules refer to the articles –
Matplotlib Tutorial
Python Seaborn Tutorial
Histogram
sns.histplot(x='Salary', data=df, )
plt.show()
Output:
Boxplot
# importing packages
import seaborn as sns
import matplotlib.pyplot as plt
Output:
Boxplot of Salary and team column
# importing packages
import seaborn as sns
import matplotlib.pyplot as plt
plt.show()
Output:
Scatter plot of salary and Team column
# importing packages
import seaborn as sns
import matplotlib.pyplot as plt
Output:
Pairplot of columns of dataframe
Handling Outliers
An Outlier is a data item/object that deviates significantly from the rest of the (so-
called normal)objects. They can be caused by measurement or execution errors. The
analysis for outlier detection is referred to as outlier mining. There are many ways to
detect outliers, and the removal process of these outliers from the dataframe is the
same as removing a data item from the panda’s dataframe.
Let’s consider the iris dataset and let’s plot the boxplot for the SepalWidthCm
column.
Example:
Python3
# importing packages
import seaborn as sns
import matplotlib.pyplot as plt
sns.boxplot(x='SepalWidthCm', data=df)
Output:
Boxplot of sample width column before outliers removal
In the above graph, the values above 4 and below 2 are acting as outliers.
Removing Outliers
For removing the outlier, one must follow the same process of removing an entry
from the dataset using its exact position in the dataset because in all the above
methods of detecting the outliers end result is the list of all those data items that
satisfy the outlier definition according to the method used.
Example: We will detect the outliers using IQR and then we will remove them. We
will also draw the boxplot to see if the outliers are removed or not.
Python3
# Importing
import sklearn
from sklearn.datasets import load_boston
import pandas as pd
import seaborn as sns
# IQR
Q1 = np.percentile(df['SepalWidthCm'], 25,
interpolation = 'midpoint')
Q3 = np.percentile(df['SepalWidthCm'], 75,
interpolation = 'midpoint')
IQR = Q3 - Q1
print("Old Shape: ", df.shape)
# Upper bound
upper = np.where(df['SepalWidthCm'] >= (Q3+1.5*IQR))
# Lower bound
lower = np.where(df['SepalWidthCm'] <= (Q1-1.5*IQR))
sns.boxplot(x='SepalWidthCm', data=df)
Output:
Note: for more information, refer Detect and Remove the Outliers using Python
These are some of the EDA we do during our data science project however it
depends upon your requirement and how much data analysis we do.