Big Data Hadoop and Spark Developer

Data Manipulation with Pandas

 Learning Objectives

By the end of this lesson, you will be able to:

Explain Pandas and its features

List different data structures of Pandas

Outline the process to create series and DataFrame with data inputs

Explain how to view, select, and access elements in a data structure

Describe the procedure to handle vectorized operations

Illustrate how to handle missing values

Analyze data with different data operation methods

Introduction to Pandas
 Why Pandas

NumPy is great for mathematical computing, but why do we need Pandas?

Pandas with several

functionalities

NumPy
 Why Pandas

Intrinsic data
alignment

Data Structures
Data operation
handle major
functions
use cases
Pandas

Data standardization ? Data handling

functions functions
 Why Pandas

Data structures handling major use cases

Pandas
 Features of Pandas

The various features of Pandas make it an efficient library for Data Scientists.

Powerful data
structure

Fast and
High performance
efficient
merging and joining
data wrangling
of data sets

Pandas
Intelligent and Easy data
automated aggregation and
data alignment transformation

Tools for reading

and writing data
Data Structures
 Data Structures

The four main libraries of Pandas data structure are:

• One-dimensional labeled array

Series
• Supports multiple data types
• Two-dimensional labeled array
• Supports multiple data types
Data Frame • Input can be a series
• Input can be another DataFrame
• Three-dimensional labeled array
• Supports multiple data types Panel
• Items 🡪 axis 0 • Four-dimensional labeled array
• Major axis 🡪 rows Panel 4D • Supports multiple data types
• Minor axis🡪 columns (Experimental) • Labels 🡪 axis 0
• Items 🡪 axis 1
• Major axis 🡪 rows
• Minor axis🡪 columns
 Understanding Series

Series is a one-dimensional array-like object containing data and labels (or index).

Data 4 11 21 36
0 1 2 3

Label(index)

Data alignment is intrinsic and will not be broken until changed explicitly by program.
 Series

Series can be created with different data inputs:

Data Input

• Integer
• ndarray 2 3 8 4
• String
• dict
• Python 0 1 2 3
• scalar
Object
• list Label(index)
• Floating Point

Data Types
Series
 How to Create Series?

Key points to note while creating a series are:

•Import Pandas as it is the main library (Import pandas as pd)

•Import NumPy while working with ndarrays (Import numpy as np)
•Apply the syntax and pass the data elements as arguments

Basic Method

4 11 21 36
S = pd.Series(data, index = [index])
Series
 Creating Series from a List

Import libraries

Pass list as an argument

Data value

Index

Data type

We have not created index for data but notice that data alignment is done automatically.
Creating Series from an ndarray

ndarray for countries

Pass ndarray as an
argument

countrie
s

Data type
 Creating Series from dict

A series can also be created with dict data input for faster operations.

dict for countries and their

gdp

Countries have been passed as an

index and GDP as the actual data
value
GDP

Country

Data type
 Creating Series from Scalar

Scalar input

Index

Data

index

Data type
 Accessing Elements in Series
Data can be accessed through different functions like loc, iloc by passing data element position or index range.
 Vectorizing Operations in Series

Vectorized operations are performed by the data element’s position.

Add the series

Vectorizing Operations in Series
DataFrames
 DataFrame

DataFrame is a two-dimensional labeled data structure with columns of potentially different types.

Data Input

• Integer
• ndarray 2 3 8 4
• String
• dict 5 8 10 1
• Python
• List
Object 0 1 2 3
• Series
• Floating Point Label(index)
• DataFrame

Data Types
DataFrame
Creating DataFrame from Lists

Pass the list to the DataFrame

 Creating DataFrame from dict

This example shows you how to create a DataFrame from a series of dicts.

dict one dict two

Entire dict
 Viewing DataFrame

You can view a DataFrame by referring to the column name or with the describe function.
Creating DataFrame from dict of Series
Creating DataFrame from ndarray

Create a ndarray with years

Create a dict with the ndarray

Pass this dict to a new DataFrame

Creating DataFrame from DataFrame Object

Create a DataFrame from a

DataFrame object
View and Select Data

Problem Statement: Demonstrate how to view and select data in a DataFrame

Access: Click on the Practice Labs tab on the left side panel of the LMS. Copy or note the
username and password that is generated. Click on the Launch Lab button. On the page that
appears, enter the username and password in the respective fields, and click Login.
Missing Values
 Missing Values

Various factors may lead to missing data values:

Data not provided by the

source Software issue Data integration issue Network issue
 Handling Missing Values
It’s difficult to operate a dataset when it has missing values or uncommon indices.
 Handling Missing Values with Functions
The dropna function drops all the values with uncommon indices.
 Handling Missing Values with Functions

The fillna function fills all the uncommon indices with a number instead of dropping them.

Fill the missing values with zero

Handling Missing Values with Functions: Example
Data Operation
 Data Operation

Data operation can be performed through various built-in methods for faster data processing.
 Data Operation with Functions
While performing data operation, custom functions can be applied using the applymap method.

Declare a custom function

Test the function

Apply the function to the DataFrame

Data Operation with Statistical Functions

Create a DataFrame with two test

Apply the max function to find

the maximum score

Apply the mean function to find

the average score

Apply the std function to find the standard

deviation for both the tests
Data Operation Using Groupby

Create a DataFrame with first

and last name as former
presidents

Group the DataFrame with the first name

Group the DataFrame with the first name

Data Operation Using Sorting

Sort values by first name

Data Operations

Problem Statement: Demonstrate how to perform data operations

Access: Click on the Practice Labs tab on the left side panel of the LMS. Copy or note the
username and password that is generated. Click on the Launch Lab button. On the page that
appears, enter the username and password in the respective fields, and click Login.
Data Standardization
Data Standardization

Create a function to return the standardize value

Apply the function to the entire dataset

Standardized test data is applied for the entire

DataFrame
 File Read and Write Support

read_hdf
read_excel to_hdf read_clipboard
to_excel to_clipboard

read_csv read_html
to csv to_html

read_json read_pickle
to_json to_pickle

read_sql read_stata
read_sas
to_sql to_stata
to sas
 Activity: Sequence it Right!
The code here is buggy. You have to correct its sequence to debug it. To do that, click any two code snippets,
which you feel are out of place, to swap their places.

Click any two code snippets to swap them.

 Activity: Sequence it Right!
The code here is buggy. You must correct its sequence to debug it. To do that, click any two code snippets,
which you feel are out of place, to swap their places.

Click any two code snippets to swap them.

Pandas SQL Operations
Pandas SQL Operation
Pandas SQL Operation
Pandas SQL Operation
Analyze the Federal Aviation Authority (FAA) Dataset using
Pandas

Problem Statement:
Analyze the Federal Aviation Authority (FAA) dataset using Pandas to do the following:
1.View
a. Aircraft manufacturer name
b. State name
c. Aircraft model name
d. Text information
e. Flight phase
f. Event description type
g. Fatal flag
2. Clean the dataset and replace the fatal flag NaN with “No”
3. Find the aircraft types and their occurrences in the dataset
4. Remove all the observations where aircraft names are not available
5. Display the observations where fatal flag is “Yes”
Analyze the Federal Aviation Authority (FAA) Dataset using
Pandas

Instructions to perform the assignment:

•Download the FAA dataset from the “Resource” tab. Upload the dataset to your Jupyter
notebook to view and evaluate it.

Common instructions:
•If you are new to Python, download the “Anaconda Installation Instructions” document
from the “Resources” tab to view the steps for installing Anaconda and the Jupyter
notebook.
•Download the “Assignment 01” notebook and upload it on the Jupyter notebook to access
it.
•Follow the cues provided to complete the assignment.
Analyzing the Dataset

Problem Statement:
A dataset in CSV format is given for the Fire Department of the New York City. Analyze the
dataset to determine:
1. The total number of fire department facilities in the New York city
2. The number of fire department facilities in each borough
3. The facility names in Manhattan
Analyzing the Dataset

Instructions to perform the assignment:

•Download the FDNY dataset from the “Resource” tab. You can upload the dataset to your
Jupyter notebook to use it.

Common instructions:
•If you are new to Python, download the “Anaconda Installation Instructions” document
from the “Resources” tab to view the steps for installing Anaconda and the Jupyter
notebook.
•Download the “Assignment 02” notebook and upload it on the Jupyter notebook to access
it.
•Follow the cues provided to complete the assignment.
 Key Takeaways

You are now able to:

Explain Pandas and its features

List different data structures of Pandas

Outline the process to create series and DataFrame with data inputs

Explain how to view, select, and access elements in a data structure

Describe the procedure to handle vectorized operations

Illustrate how to handle missing values

Analyze data with different data operation methods

Knowledge Check
Knowledge
Check How is an index for data elements assigned while creating a Pandas series ? Select all
that apply?
1

a. Created automatically

b. Needs to be assigned

c. Once created can not be changed or altered

d. Index is not applicable as series is one-dimensional

 Knowledge
Check How is an index for data elements assigned while creating a Pandas series ? Select all
that apply?
1

a. Created automatically

b. Needs to be assigned

c. Once created can not be changed or altered

d. Index is not applicable as series is one-dimensional

The correct answer is a, b

Data alignment is intrinsic in Pandas data structure and happens automatically. One can also assign index to
data elements.
Knowledge
Check
What will the result be in vector addition if label is not found in a series?
2

a. Marked as zeros for missing labels

b. Labels will be skipped

c. Marked as NaN for missing labels

d. Will prompt an exception, index not found

 Knowledge
Check
What will the result be in vector addition if label is not found in a series?
2

a. Marked as zeros for missing labels

b. Labels will be skipped

c. Marked as NaN for missing labels

d. Will prompt an exception, index not found

The correct answer is c

The result will be marked as NaN (Not a Number) for missing labels.
Knowledge
Check
What is the result of DataFrame[3:9]?
3

a. Series with sliced index from 3 to 9

b. dict of index positions 3 and 9

c. DataFrame of sliced rows index from 3 to 9

d. DataFrame with data elements at index 3 to 9

 Knowledge
Check
What is the result of DataFrame[3:9]?
3

a. Series with sliced index from 3 to 9

b. dict of index positions 3 and 9

c. DataFrame of sliced rows index from 3 to 9

d. DataFrame with data elements at index 3 to 9

The correct answer is c

This is DataFrame slicing technique with indexing or selection on data elements. When a user passes the
range 3:9, the entire range from 3 to 9 gets sliced and displayed as output.
Knowledge
Check
What does the fillna() method do?
4

a. Fills all NaN values with zeros

b. Fills all NaN values with one

c. Fills all NaN values with values mentioned in the parenthesis

d. Drops NaN values from the dataset

 Knowledge
Check
What does the fillna() method do?
4

a. Fills all NaN values with zeros

b. Fills all NaN values with one

c. Fills all NaN values with values mentioned in the parenthesis

d. Drops NaN values from the dataset

The correct answer is c

fillna is one of the basic methods to fill NaN values in a dataset with a desired value by passing that in
parenthesis.
Knowledge
Check
Which of the following data structures is used to store three-dimensional data?
5

a. Series

b. DataFrame

c. Panel

d. PanelND
 Knowledge
Check
Which of the following data structures is used to store three-dimensional data?
5

a. Series

b. DataFrame

c. Panel

d. PanelND

The correct answer is c

Panel is a data structure used to store three-dimensional data.

Knowledge
Check
Which method is used for label-location indexing by label?
6

a. iat

b. iloc

c. loc

d. std
 Knowledge
Check
Which method is used for label-location indexing by label?
6

a. iat

b. iloc

c. loc

d. std

The correct answer is c

The loc method is used for label-location indexing by label; iat is strictly integer location and iloc is integer-
location-based indexing by position.
Knowledge
Check
While viewing a dataframe, head() method will _____.
7

a. return only the first row

b. return only headers or column name of the DataFrame

c. return the first five rows of the DataFrame

d. throw an exception as it expects parameter(number) in parenthesis

 Knowledge
Check
While viewing a dataframe, head() method will _____.
7

a. return only the first row

b. return only headers or column name of the DataFrame

c. return the first five rows of the DataFrame

d. throw an exception as it expects parameter(number) in parenthesis

The correct answer is c

The default value is 5 if nothing is passed in head method. So, it will return the first five rows of the
DataFrame.
Thank You