NAAC Accredited A+ Grade

Tulsiramji Gaikwad-
Patil
College
(An of Engineering
Autonomous Institute) &
Technology

MBA Internship Progress

Presentation
(SESSION 2023-24)
Presentation
By
------------------------
Saurabh Arvind Naik
On the Topic of
DATA SCIENCE USING PYTHON
Date:- 12/11/2024 GAIKWAD-PATIL
GROUP OF INSTITUTIONS
 Overview of Python Libraries for Data
Tutorial Content Scientists

Reading Data; Selecting and Filtering the Data; Data manipulation,

sorting, grouping, rearranging

Plotting the data

Descriptive statistics

Inferential statistics
GAIKWAD-PATIL
GROUP OF INSTITUTIONS
2
Python Libraries for Data Science
Many popular Python toolboxes/libraries:
• NumPy
• SciPy All these libraries are
• Pandas installed on the SCC
• SciKit-Learn

Visualization libraries
• matplotlib
• Seaborn

and many more … GAIKWAD-PATIL

GROUP OF INSTITUTIONS
3
Python Libraries for Data Science
NumPy:
▪ introduces objects for multidimensional arrays and matrices, as well as
functions that allow to easily perform advanced mathematical and statistical
operations on those objects

▪ provides vectorization of mathematical operations on arrays and matrices

which significantly improves the performance

▪ many other python libraries are built on NumPy

SciPy:
▪ collection of algorithms for linear algebra, differential equations, numerical
Link: http://www.numpy.org/
integration, optimization, statistics and more GAIKWAD-PATIL
GROUP OF INSTITUTIONS
4
Python Libraries for Data Science
Pandas:
▪ adds data structures and tools designed to work with table-like data (similar
to Series and Data Frames in R)

▪ provides tools for data manipulation: reshaping, merging, sorting, slicing,

aggregation etc.

▪ allows handling missing data

SciKit-Learn:
▪ provides machine learning algorithms: classification, regression, clustering,
model validation etc.
Link: http://pandas.pydata.org/ GAIKWAD-PATIL
▪ built on NumPy, SciPy and matplotlib GROUP OF INSTITUTIONS
5
Python Libraries for Data Science
matplotlib:
▪ python 2D plotting library which produces publication quality figures in a variety of hardcopy
formats

▪ a set of functionalities similar to those of MATLAB

▪ line plots, scatter plots, barcharts, histograms, pie charts etc.

▪ relatively low-level; some effort needed to create advanced visualization

Seaborn:
▪ based on matplotlib

Link: ▪ provides high level interface for drawing attractive statistical graphics
https://matplotlib.org/ GAIKWAD-PATIL
GROUP OF INSTITUTIONS
6
▪ Similar (in style) to the popular ggplot2 library in R
Start Google Colab
# On the Shared Computing Cluster
[scc1 ~] Google Colab

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
7
Loading Python Libraries
In [ ]: #Import Python Libraries
import numpy as np
import scipy as sp
import pandas as pd
import matplotlib as mpl
import seaborn as sns

Press Shift+Enter to execute the jupyter cell

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
8
 Reading data using pandas
In [ ]: #Read csv file
df = pd.read_csv("http://rcs.bu.edu/examples/python/data_analysis/Salaries.csv")

Note: The above command has many optional arguments to fine-tune the data import process.

There is a number of pandas commands to read other data formats:

pd.read_excel('myfile.xlsx',sheet_name='Sheet1', index_col=None,
na_values=['NA'])
pd.read_stata('myfile.dta')
pd.read_sas('myfile.sas7bdat')
pd.read_hdf('myfile.h5','df') GAIKWAD-PATIL
GROUP OF INSTITUTIONS
9
 Exploring data frames
In [3]: #List first 5 records
df.head()

Out[3]:

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
10
Data Frame data types
Pandas Type Native Python Type Description
object string The most general dtype. Will be
assigned to your column if
column has mixed types
(numbers and strings).
int64 int Numeric characters. 64 refers to
the memory allocated to hold
this character.
float64 float Numeric characters with
decimals. If a column contains
numbers and NaNs(see below),
pandas will default to float64, in
case your missing value has a
decimal.
datetime64, timedelta[ns] N/A (but see the datetime Values meant to hold time data.
module in Python’s standard Look into these for time series
library) experiments. GAIKWAD-PATIL
GROUP OF INSTITUTIONS
11
Data Frames attributes
Python objects have attributes and methods.

df.attribute description
dtypes list the types of the columns
columns list the column names
axes list the row labels and column names
ndim number of dimensions

size number of elements

shape return a tuple representing the dimensionality
values numpy representation of the data
GAIKWAD-PATIL
GROUP OF INSTITUTIONS
12
Data Frames methods
Unlike attributes, python methods have parenthesis.
All attributes and methods can be listed with a dir() function: dir(df)

df.method() description
head( [n] ), tail( [n] ) first/last n rows

describe() generate descriptive statistics (for numeric columns

only)
max(), min() return max/min values for all numeric columns

mean(), median() return mean/median values for all numeric columns

std() standard deviation

sample([n]) returns a random sample of the data frame

GAIKWAD-PATIL
dropna() drop all the records with missing values GROUP OF INSTITUTIONS
13
 Data Frames groupby method
Using "group by" method we can:
• Split the data into groups based on some criteria
• Calculate statistics (or apply a function) to each group
• Similar to dplyr() function in R
In [ ]: #Group data using rank
df_rank = df.groupby(['rank'])

In [ ]: #Calculate mean value for each numeric column per each group
df_rank.mean()

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
14
 Data Frames groupby method

Once groupby object is create we can calculate various statistics for each group:
In [ ]: #Calculate mean salary for each professor rank:
df.groupby('rank')[['salary']].mean()

Note: If single brackets are used to specify the column (e.g. salary), then the output is Pandas Series object.
When double brackets are used the output is a Data Frame
15
 Data Frames groupby method

groupby performance notes:

- no grouping/splitting occurs until it's needed. Creating the groupby object
only verifies that you have passed a valid mapping
- by default the group keys are sorted during the groupby operation. You may
want to pass sort=False for potential speedup:

In [ ]: #Calculate mean salary for each professor rank:

df.groupby(['rank'], sort=False)[['salary']].mean()

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
16
 Data Frame: filtering

To subset the data we can apply Boolean indexing. This indexing is commonly
known as a filter. For example if we want to subset the rows in which the salary
value is greater than $120K:
In [ ]: #Calculate mean salary for each professor rank:
df_sub = df[ df['salary'] > 120000 ]

Any Boolean operator can be used to subset the data:

> greater; >= greater or equal;
< less; <= less or equal;
== equal; != not equal;
In [ ]: #Select only those rows that contain female professors: GAIKWAD-PATIL
df_f = df[ df['sex'] == 'Female' ] GROUP OF INSTITUTIONS
17
Data Frames: Slicing

There are a number of ways to subset the Data Frame:

• one or more columns
• one or more rows
• a subset of rows and columns

Rows and columns can be selected by their position or label

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
18
 Data Frames: Slicing

When selecting one column, it is possible to use single set of brackets, but the
resulting object will be a Series (not a DataFrame):
In [ ]: #Select column salary:
df['salary']

When we need to select more than one column and/or make the output to be a
DataFrame, we should use double brackets:
In [ ]: #Select column salary:
df[['rank','salary']]

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
19
 Data Frames: Sorting

We can sort the data by a value in the column. By default the sorting will occur in
ascending order and a new data frame is return.

In [ ]: # Create a new data frame from the original sorted by the column Salary
df_sorted = df.sort_values( by ='service')
df_sorted.head()

Out[ ]:

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
20
 Data Frames: Sorting

We can sort the data using 2 or more columns:

In [ ]: df_sorted = df.sort_values( by =['service', 'salary'], ascending = [True, False])
df_sorted.head(10)

Out[ ]:

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
21
 Missing Values
Missing values are marked as NaN
In [ ]: # Read a dataset with missing values
flights = pd.read_csv("http://rcs.bu.edu/examples/python/data_analysis/flights.csv")

In [ ]: # Select the rows that have at least one missing value

flights[flights.isnull().any(axis=1)].head()

Out[ ]:

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
22
Missing Values
There are a number of methods to deal with missing values in the data frame:

df.method() description
dropna() Drop missing observations

dropna(how='all') Drop observations where all cells is NA

dropna(axis=1, Drop column if all the values are missing

how='all')
dropna(thresh = 5) Drop rows that contain less than 5 non-missing values

fillna(0) Replace missing values with zeros

isnull() returns True if the value is missing

GAIKWAD-PATIL
notnull() Returns True for non-missing values GROUP OF INSTITUTIONS
23
Missing Values
• When summing the data, missing values will be treated as zero
• If all values are missing, the sum will be equal to NaN
• cumsum() and cumprod() methods ignore missing values but preserve them in
the resulting arrays
• Missing values in GroupBy method are excluded (just like in R)
• Many descriptive statistics methods have skipna option to control if missing
data should be excluded . This value is set to True by default (unlike R)

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
24
Aggregation Functions in Pandas
Aggregation - computing a summary statistic about each group, i.e.
• compute group sums or means
• compute group sizes/counts

Common aggregation functions:

min, max
count, sum, prod
mean, median, mode, mad
std, var

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
25
 Aggregation Functions in Pandas
agg() method are useful when multiple statistics are computed per column:
In [ ]: flights[['dep_delay','arr_delay']].agg(['min','mean','max'])

Out[ ]:

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
26
Basic Descriptive Statistics
df.method() description
describe Basic statistics (count, mean, std, min, quantiles, max)

min, max Minimum and maximum values

mean, median, mode Arithmetic average, median and mode

var, std Variance and standard deviation

sem Standard error of mean

skew Sample skewness

kurt kurtosis

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
27
 Graphics to explore the data
Seaborn package is built on matplotlib but provides high level
interface for drawing attractive statistical graphics, similar to ggplot2
library in R. It specifically targets statistical data visualization

To show graphs within Python notebook include inline directive:

In [ ]: %matplotlib inline

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
28
Graphics
description
distplot histogram
barplot estimate of central tendency for a numeric variable
violinplot similar to boxplot, also shows the probability density of the
data
jointplot Scatterplot
regplot Regression plot
pairplot Pairplot
boxplot boxplot
swarmplot categorical scatterplot
factorplot General categorical plot
GAIKWAD-PATIL
GROUP OF INSTITUTIONS
29
30
31
 Objectives of the Internship
● Primary Objective: Apply data science techniques to analyze and interpret
data.
● Secondary Goals:
○ Enhance data quality and insights for better decision-making.
○ Support SAVVYSCRIBE CPA's data initiatives using Python tools and
techniques.

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
32
 Responsibilities & Tasks
● Data Collection and Cleaning: Managed data preparation, addressing
missing values, and normalizing datasets.
● Data Analysis: Performed descriptive and predictive analyses using Python
libraries.
● Modeling: Developed simple models to extract insights (e.g., regression,
clustering).
● Data Visualization: Created charts and graphs for presenting data insights
using Matplotlib and Seaborn.

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
33
 Skill Development
● Technical Skills:
a. Proficiency in Python and its data science libraries (Pandas, NumPy,
Scikit-Learn).
b. Enhanced skills in data manipulation, cleaning, and visualization.
● Soft Skills:
● Improved teamwork, adaptability, and time management.
● Gained experience in presenting technical findings to non-technical
stakeholders.

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
34
 Lessons Learned
● Data Quality Challenges: Learned to handle data inconsistencies effectively.
● Analytical Thinking: Gained skills in approaching complex data problems
logically.
● Effective Communication: Developed the ability to convey data insights in
an understandable manner.

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
35
 Conclusion
● Summary of Experience: Recap of the key contributions and learning
outcomes.
● Future Impact: How the experience aligns with and supports career
aspirations in data science.
● Acknowledgment: Thank the team at SAVVYSCRIBE CPA and college
faculty for support.

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
36
Thank You

GAIKWAD-PATIL
GROUP OF INSTITUTIONS
37