The Unique Computers

Neelmatha Lucknow
Python Pandas
What is Pandas?
Pandas is a powerful Python library that is specifically designed to work on
data frames that have "relational" or "labeled" data. Its aim aligns with doing
real-world data analysis using Python. Its flexibility and functionality make it
indispensable for various data-related tasks. Hence, this Python package
works well for data manipulation, operating a dataset, exploring a data
frame, data analysis, and machine learning-related tasks.

Generally, Pandas operates a data frame using Series and DataFrame; where
Series works on a one-dimensional labeled array holding data of any type
like integers, strings, and objects, while a DataFrame is a two-dimensional
data structure that manages and operates data in tabular form (using rows
and columns).

Why Pandas?
The beauty of Pandas is that it simplifies the task related to data frames and
makes it simple to do many of the time-consuming, repetitive tasks involved
in working with data frames, such as:

• Import datasets - available in the form of spreadsheets, comma-

separated values (CSV) files, and more.

• Data cleansing - dealing with missing values and representing them

as NaN, NA, or NaT.

• Size mutability - columns can be added and removed from

DataFrame and higher-dimensional objects.

• Data normalization – normalize the data into a suitable format for

analysis.

• Data alignment - objects can be explicitly aligned to a set of labels.

Intuitive merging and joining data sets – we can merge and join
datasets.
 • Reshaping and pivoting of datasets – datasets can be reshaped
and pivoted as per the need.

• Efficient manipulation and extraction - manipulation and

extraction of specific parts of extensive datasets using intelligent label-
based slicing, indexing, and subsetting techniques.

• Statistical analysis - to perform statistical operations on datasets.

• Data visualization - Visualize datasets and uncover insights.

Applications of Pandas
The most common applications of Pandas are as follows:

• Data Cleaning: Pandas provides functionalities to clean messy data,

deal with incomplete or inconsistent data, handle missing values,
remove duplicates, and standardize formats to do effective data
analysis.

• Data Exploration: Pandas easily summarize statistics, find trends,

and visualize data using built-in plotting functions, Matplotlib, or
Seaborn integration.

• Data Preparation: Pandas may pivot, melt, convert variables, and

merge datasets based on common columns to prepare data for
analysis.

• Data Analysis: Pandas supports descriptive statistics, time series

analysis, group-by operations, and custom functions.

• Data Visualisation: Pandas itself has basic plotting capabilities; it

integrates and supports data visualization libraries like Matplotlib,
Seaborn, and Plotly to create innovative visualizations.

• Time Series Analysis: Pandas supports date/time indexing,

resampling, frequency conversion, and rolling statistics for time series
data.

• Data Aggregation and Grouping: Pandas HYPERLINK

"https://www.tutorialspoint.com/python_pandas/python_pandas_groupb
y.htm"groupby HYPERLINK
"https://www.tutorialspoint.com/python_pandas/python_pandas_groupb
y.htm"() function lets you aggregate data and compute group-wise
summary statistics or apply functions to groups.
 • Data Input/Output: Pandas makes data input and export easy by
reading and writing CSV, Excel, JSON, SQL databases, and more.

• Machine Learning: Pandas works well with Scikit-learn for data

preparation, feature engineering, and model input data.

• Financial Analysis: Pandas is commonly used in finance for stock

market data analysis, financial indicator calculation, and portfolio
optimization.

• Text Data Analysis: Pandas' string manipulation, regular expressions,

and text mining functions help analyse textual data.

• Experimental Data Analysis: Pandas makes manipulating and

analysing large datasets, performing statistical tests, and visualizing
results easy.

Python Pandas Data Structures

Data structures in Pandas are designed to handle data efficiently. They allow
for the organization, storage, and modification of data in a way that
optimizes memory usage and computational performance. Python Pandas

−
library provides two primary data structures for handling and analyzing data

• Series

• DataFrame

Dimension and Description of Pandas Data Structures

Data Dimensio Description

Structure ns
Series 1 A one-dimensional labeled homogeneous array, sizeimmutable.
Data Frames 2 A two-dimensional labeled, size-mutable tabular structure with
potentially heterogeneously typed columns.

Series

A Series is a one-dimensional labeled array that can hold any data type. It
can store integers, strings, floating-point numbers, etc. Each value in a
Series is associated with a label (index), which can be an integer or a string.

Name Steve
 Age 35

Gender Male

Rating 3.5

Example

Consider the following Series which is a collection of different data types

import pandas as pd
data = ['Steve', '35', 'Male', '3.5']
series = pd.Series(data, index=['Name', 'Age', 'Gender', 'Rating'])
print(series)

On executing the above program, you will get the following output −

Name Steve

Age 35

Gender Male

Rating 3.5

dtype: object

Key Points

Following are the key points related to the Pandas Series.

• Homogeneous data

• Size Immutable

• Values of Data Mutable

DataFrame

A DataFrame is a two-dimensional labeled data structure with columns that

can hold different data types. It is similar to a table in a database or a

rating of a sales team −

spreadsheet. Consider the following data representing the performance

Name Age Gender Rating

 Steve 32 Male 3.45

Lia 28 Female 4.6

Vin 45 Male 3.9

Katie 38 Female 2.78

Example

The above tabular data can be represented in a DataFrame as follows −

Open Compiler

import pandas as pd

# Data represented as a dictionary

data = {

'Name': ['Steve', 'Lia', 'Vin', 'Katie'],

'Age': [32, 28, 45, 38],

'Gender': ['Male', 'Female', 'Male', 'Female'],

'Rating': [3.45, 4.6, 3.9, 2.78]

# Creating the DataFrame

df = pd.DataFrame(data)
print(df)

Output

On executing the above code you will get the following output −

Name Age Gender Rating

0 Steve 32 Male 3.45

1 Lia 28 Female 4.60

2 Vin 45 Male 3.90

3 Katie 38 Female 2.78

Key Points

Following are the key points related the Pandas DataFrame −

• Heterogeneous data

• Size Mutable

• Data Mutable

Creation of Data Frames

Creation New dataFrames

import pandas as pd
data={"name":["rahul","neha","amit"],
"age":[12,15,27],
"Salary":[1200,1500,1200]
}
df=pd.DataFrame(data)
print(df)

Reading CSV File

import pandas as pd
data=pd.read_csv("book.csv")
print(data)

reading Excel File

import pandas as pd
data=pd.read_excel("book1.xlsx")
print(data)

Exploring Data in Pandas

There are some Function in Pandas

Head()

Tail()

Info()

Describe()

Isnull()
Isnull().sum()

Dealing With Duplicate Values

Data.duplicated()

import pandas as pd

data=pd.read_excel("salary.xlsx")

print(data.duplicated())

Data[“emp_id”].duplicated()

import pandas as pd

data=pd.read_excel("salary.xlsx")

print(data["Emp_ID"].duplicated())

Data[“emp_id”].duplicated().sum()

import pandas as pd

data=pd.read_excel("salary.xlsx")

print(data["Emp_ID"].duplicated().sum())

Data.drop_duplicates(“emp_id”)

import pandas as pd

data=pd.read_excel("salary.xlsx")

print(data.drop_duplicates("Emp_ID"))

Working with missing values

Data.isnull
To print null values
import pandas as pd
data=pd.read_excel("salary.xlsx")
print(data.isnull())
data.isnull().sum())
to count null values
import pandas as pd
data=pd.read_excel("salary.xlsx")
print(data.isnull().sum())
data.dropna()
To delete null values
import pandas as pd
data=pd.read_excel("salary.xlsx")
print(data)
print("\n\n\n")
print(data.dropna())
data.replace(np.nan,"hii")
to replace nan
import numpy as np
import pandas as pd
data=pd.read_excel("salary.xlsx")
print(data)
data.replace(np.nan,"hii")
data["Salary"]=data["Salary"].replace(np.nan,30000)
to replace any special char
import pandas as pd
import numpy as np
data=pd.read_excel("salary.xlsx")
data["Salary"]=data["Salary"].replace(np.nan,30000)
print(data)
data["Salary"].mean()
import pandas as pd
import numpy as np
data=pd.read_excel("salary.xlsx")
print(data["Salary"].mean())
data.fillna(method="bfill")
import pandas as pd
import numpy as np
data=pd.read_excel("salary.xlsx")
print(data)
print("\n\n\n")
print(data.fillna(method="bfill"))
data.fillna(method="ffill")
import pandas as pd
import numpy as np
data=pd.read_excel("salary.xlsx")
print(data)
print("\n\n\n")
print(data.fillna(method="ffill"))
 Column transformation in Pandas

To create new column

import pandas as pd

data=pd.read_excel("salary.xlsx")

print(data,"\n\n")

data.loc[(data["Bonus"] == 0),"GetBonus"]="No Bonus"

data.loc[(data["Bonus"] > 0,"GetBonus")]="Bonus"

print(data)

To marge two column

import pandas as pd

data=pd.read_excel("salary.xlsx")

print(data,"\n\n")

data["Full name"]=data["Name"]+" "+data["Last Name"]

print(data)

To Add Calculation in column

import pandas as pd

data=pd.read_excel("salary.xlsx")

print(data,"\n\n")

data["Bonus"]=(data["Salary"]/100)*20

print(data)

To extract some latter from dataFrame

import pandas as pd

data={"Month":["January","Fabruary","March","April"]}

a=pd.DataFrame(data)
print(a)

def extract(value):

return value[0:3]

a["Short_Months"]=a["Month"].map(extract)

print(a)

GroupBy In Pandas
Count gender by Deparment

import pandas as pd
data=pd.read_excel("Salary.xlsx")
print(data)
gp=data.groupby("Department").agg({"Gender":"count"})
print(gp)
By Job Title count Emp_id

import pandas as pd
data=pd.read_excel("Salary.xlsx")
print(data)
gp=data.groupby("Job Title").agg({"Emp_ID":"count"})
print(gp)

By Gender

import pandas as pd
data=pd.read_excel("Salary.xlsx")
print(data)
gp=data.groupby(["Department","Gender"]).agg({"Emp_ID":"count"})
print(gp)
By Age
import pandas as pd
data=pd.read_excel("Salary.xlsx")
print(data)
print("\n\n\n")
a=data.groupby("Countries").agg({"Age":"max"})
print(a)
By Age and Gender
import pandas as pd
data=pd.read_excel("Salary.xlsx")
print(data)
print("\n\n\n")
a=data.groupby(["Countries","Gender"]).agg({"Age":"max"})
print(a)

Merge Join and Concatenate in Pandas

Merge
On the basis of EEID
import pandas as pd
data1={"EEID":["A01","A02","A03","A04","A05","A06"],
"Name":["Amit","priya","Neha","Lovely","Karab","Mohit"],
"Age":[34,56,24,27,28,26]}
print(data1)
data2={"EEID":["A01","A02","A03","A04","A05","A06"],
"Salary":[45000,47000,30000,14200,42300,456600]}
print(data2)
print("\n\n\n")
df1=pd.DataFrame(data1)
df2=pd.DataFrame(data2)
print(df1)
print()
print(df2)
print()
print(pd.merge(df1,df2,on="EEID"))
Use of how
P1
import pandas as pd
data1={"EEID":["A01","A02","A03","A04","A05","A06"],
"Name":["Amit","priya","Neha","Lovely","Karab","Mohit"],
"Age":[34,56,24,27,28,26]}
print(data1)
data2={"EEID":["A01","A02","A03","A04","A05","A06"],
"Salary":[45000,47000,30000,14200,42300,456600]}
print(data2)
print("\n\n\n")
df1=pd.DataFrame(data1)
df2=pd.DataFrame(data2)
print(df1)
print()
print(df2)
print()
print(pd.merge(df1,df2,on="EEID", how="inner"))
P2
print(pd.merge(df1,df2,on="EEID", how="left"))
P3
print(pd.merge(df1,df2,on="EEID", how="right"))

Concatenate
import pandas as pd
data1={"EEID":["A01","A02","A03","A04","A05","A06"],
"Name":["Amit","priya","Neha","Lovely","Karan","Mohit"]}
data2={"EEID":["A07","A08","A09","A010","A11","A12"],
"Name":["Atin","Pankaj","Alia","Suman","Sanjay","Karan"]}
df1=pd.DataFrame(data1)
df2=pd.DataFrame(data2)
print(df1)
print(df2)
print()
ndf=pd.concat([df1,df2])
print(ndf)

Join
import pandas as pd
data1={"EEI":["A01","A02","A03","A04","A05","A06"],
"Name":["Amit","priya","Neha","Lovely","Karab","Mohit"]}
print(data1)
data2={"EEID":["A09","A02","A03","A010","A05","A06"],
"Salary":[45000,47000,30000,14200,42300,456600]}
print(data2)
print("\n\n\n")
df1=pd.DataFrame(data1)
df2=pd.DataFrame(data2)
print(df1)
print()
print(df2)
print()
print(df1.join(df2))