Python Revisited

PYTHON DATA

Overview of Python Libraries for Data Science

Pandas: Reading Data; Selecting and Filtering the Data; Data

manipulation, sorting, grouping, rearranging
SCIENCE

Scikit-Learn Machine Learning

2
Python
Lets find what we know.....
•
Print the string “Hello Python”
•
Declare 2 variable x and y with any value. Write an If condition to print which is
bigger.
•
Print numbers from 1 to 10 using a while loop
•
Create a list with 5 elements clue: [], append, remove, pop
– Print items in list one by one
– Add one more item
– Add a list to the end of the list
– print the number of items in the list
– print the third item
– print the 2nd item to 5th item
– print 3rd item to the last item
– Remove the last item from the list
– Remove an item from list : by value and by index
Lets find what we know.....
•
Create a dictionary with your name, semester, place and phone number clue: {}
 Print your name
 Print the keys
 Print the values
 Print the items as key:value using for loop
•
Write a function to add 2 numbers
•
Print todays data by importing the datetime library clue: now()
Python Libraries for Data Science

6
Python Libraries for Data Science

Pandas:
 adds data structures and tools designed to work with table-like data (similar
to Series and Data Frames in R)

 provides tools for data manipulation: reshaping, merging, sorting, slicing,

aggregation etc.

 allows handling missing data

Link: http://pandas.pydata.org/

7
Python Libraries for Data Science

SciKit-Learn:
 provides machine learning algorithms: classification, regression, clustering,
model validation etc.

 built on NumPy, SciPy and matplotlib

Link: http://scikit-learn.org/

8
What will we learn ?

Importing Pandas Library

How to Open a Dataset / Table / CSV

Exploring the Dataset

Renaming Column Headers

Grouping Rows

Filtering Rows

Slicing the Dataset

Sorting the Rows

Join 2 Dataset by Rows

Join two related Datasets
10
Loading Pandas Libraries

In [ ]: #Import Python Libraries

import pandas as pd

Press Shift+Enter to execute the jupyter cell

11
 Reading data using pandas

In [ ]: #Read csv file

df = pd.read_csv("files/itsalary.csv")

Note: The above command has many optional arguments to fine-tune the data import process.

There is a number of pandas commands to read other data formats:

pd.read_excel('myfile.xlsx',sheet_name='Sheet1', index_col=None,
na_values=['NA'])

12
 Exploring data frames

In [3]: #List first 5 records

df.head()

Out[3]:

13
 Hands-on exercises


Try to read the first 20 records;


Can you guess how to view the last few records; clue:

14
Rename Column Names

15
Data Frames attributes
Python objects have attributes and methods.

df.attribute description
dtypes list the types of the columns
columns list the column names
axes list the row labels and column names
ndim number of dimensions

size number of elements

shape return a tuple representing the dimensionality

16
 Hands-on exercises


Find how many records this data frame has;


What are the column names?


What types of columns we have in this data frame?

17
Data Frames methods
Unlike attributes, python methods have parenthesis.

df.method() description
head( [n] ), tail( [n] ) first/last n rows

describe() generate descriptive statistics (for numeric columns only)

max(), min() return max/min values for all numeric columns

mean(), median() return mean/median values for all numeric columns

std() standard deviation

sample(n) returns a random sample of the data frame

dropna() drop all the records with missing values

18
 Hands-on exercises


Give the summary for the numeric columns in the dataset


Calculate standard deviation for all numeric columns;


What are the average salary of the first 50 records in the dataset? Hint: head()

19
Selecting a column in a Data Frame

Method 1: Subset the data frame using column name:

df['gender']

Method 2: Use the column name as an attribute:

df.gender

Note: there is an attribute rank for pandas data frames, so to select a column with a name "rank" we
should use method 1.

20
Drop a column in a Data Frame

df.drop(['col1','col2'],axis = 1)

21
 Hands-on exercises


Calculate the basic statistics for the salary column;


Find how many values in the salary column (use count method);


Calculate the average salary;

22
 Data Frames groupby method

Using "group by" method we can:

•
Split the data into groups based on some criteria

In [ ]: #Group data using rank

df_gender = df.groupby(['gender'])
•
Calculate statistics (or apply a function) to each group

In [ ]: #Calculate mean value for each numeric column per each group
df_rank.mean()
df_rank.describe()

23
 Data Frames groupby method

Once groupby object is create we can calculate various statistics for each group:

In [ ]: #Calculate mean salary for each professor rank:

group1 = df.groupby('gender')
group1[['salary']].mean()

Note: If single brackets are used to specify the column (e.g. salary), then the output is Pandas Series object.
When double brackets are used the output is a Data Frame
24
 Data Frame: filtering

To subset the data we can apply Boolean indexing. This indexing is commonly
known as a filter. For example if we want to subset the rows in which the salary
value is greater than $120K:
In [ ]: #Calculate mean salary for each professor rank:
df_sub = df[ df['salary'] > 120000 ]

Any Boolean operator can be used to subset the data:

> greater; >= greater or equal;
< less; <= less or equal;
== equal; != not equal;
In [ ]: #Select only those rows that contain female professors:
df_f = df[ df['sex'] == 'Female' ]
25
Data Frames: Slicing

There are a number of ways to subset the Data Frame:

•
one or more columns
•
one or more rows
•
a subset of rows and columns

Rows and columns can be selected by their position or label

26
 Data Frames: Slicing

When selecting one column, it is possible to use single set of brackets, but the
resulting object will be a Series (not a DataFrame):
In [ ]: #Select column salary:
df['salary']

When we need to select more than one column and/or make the output to be a
DataFrame, we should use double brackets:
In [ ]: #Select column salary:
df[['name','salary']]

27
 Data Frames: Selecting rows

If we need to select a range of rows, we can specify the range using ":"

In [ ]: #Select rows by their position:

df[10:20]

Notice that the first row has a position 0, and the last value in the range is
omitted:
So for 0:10 range the first 10 rows are returned with the positions starting with 0
and ending with 9

28
 Data Frames: method loc

If we need to select a range of rows, using their labels we can use method loc:

In [ ]: #Select rows by their labels:

df_sub.loc[10:20,['name','salary']]

29
 Data Frames: method iloc

If we need to select a range of rows and/or columns, using their positions we can
use method iloc:
In [ ]: #Select rows by their labels:
df_sub.iloc[10:20,[0, 3, 4, 5]]

Out[ ]:

30
Data Frames: method iloc
(summary)
df.iloc[0] # First row of a data frame
df.iloc[i] #(i+1)th row
df.iloc[-1] # Last row

df.iloc[:, 0] # First column

df.iloc[:, -1] # Last column

df.iloc[0:7] #First 7 rows

df.iloc[:, 0:2] #First 2 columns
df.iloc[1:3, 0:2] #Second through third rows and first 2 columns
df.iloc[[0,5], [1,3]] #1st and 6th rows and 2nd and 4th columns

31
 Data Frames: Sorting

We can sort the data by a value in the column. By default the sorting will occur in
ascending order and a new data frame is return.

In [ ]: # Create a new data frame from the original sorted by the column Salary
df_sorted = df.sort_values( by ='salary')
df_sorted.head()

Out[ ]:

32
 Data Frames: Sorting

We can sort the data using 2 or more columns:

In [ ]: df_sorted = df.sort_values( by =['gender', 'name'], ascending = [True, False])
df_sorted.head(10)

33
Missing Values

There are a number of methods to deal with missing values in the data frame:

df.method() description
dropna() Drop missing observations

dropna(how='all') Drop observations where all cells is NA

dropna(axis=1, how='all') Drop column if all the values are missing

dropna(thresh = 5) Drop rows that contain less than 5 non-missing values

fillna(0) Replace missing values with zeros

34
Missing Values

•
When summing the data, missing values will be treated as zero
•
If all values are missing, the sum will be equal to NaN
•
cumsum() and cumprod() methods ignore missing values but preserve them in
the resulting arrays
•
Missing values in GroupBy method are excluded
•
Many descriptive statistics methods have skipna option to control if missing
data should be excluded . This value is set to True by default

35
Drop Duplicate Rows

Read file duplicates.csv

rem_dup=df.drop_duplicates(['Product'])

Hands on: Drop same data repeated

36
Create new column

df['new column'] = 10

Hands on : Add new Column salesprice which is 30% more of price

37
Change existing column

df['Price'] = 10
df.Price=df.Price * 1.3

How to change values conditionally ?

df.loc[df['condition_column'] == 0, 'Column_To_Change'] = 0

38
 Concatenate two Data Frames
•
Load csv Sales.csv and Sales2.csv from item_sales folder

•
pd.concat([df1,df2])
 Join two Data Frames
•
Load csv Sales.csv and Sales2.csv from item_sales folder – We did this
•
Load csv items.csv from item_sales folder

•
pd.merge(df1,df2,on='column1',how='inner')
•
pd.merge(df1,df2,on='column1',how='left')
•
pd.merge(df1,df2,on='column1',how='right')
•
pd.merge(df1,df2,on='column1',how='outer')