Data Science With Python

Data Science with Python
Lesson 1—Data Science Overview

What You’ll Learn
Know what Data Science is
Discuss the roles and responsibilities of a Data Scientist
List various applications of Data Science
Understand how Data Science and Big Data work together
Explore Data Science as a discipline
Understand how and why Data Science is gaining importance
Understand what Python is and what problems it resolves

The Components of Data Science
When we combine domain expertise and scientific methods with technology, we get Data Science.
Domain Expertise and Scientific Methods Technology
Operating Systems
Analysis
Mathematical and Scientific Python Application

Statistical Models Tools and Methods Language Design
Data Science
Data Processing Library

Tools
Domain Expertise and Scientific Methods
Data Scientists collect data and explore, analyze, and visualize it. They apply mathematical and statistical
models to find patterns and solutions in the data.
Data analysis can be:
Analysis
• Descriptive: Study a dataset to decipher the
details
• Predictive: Create a model based on existing
Mathematical Scientific
and Statistical Tools and information to predict outcome and behavior
Models Methods
• Prescriptive: Suggest actions for a given
situation using the collected information
Data Processing and Analytics
Modern tools and technologies have made data processing and analytics faster and efficient.
These technologies help Data Scientists to Technology
• Build and train machine learning models Operating System

• Manipulate data with technology
Python
• Build data tools, applications, and services Language Application
Design
• Extract information from data
Library
Data Processing
Tools
Data analysis that uses only technology and domain knowledge without mathematical and statistical knowledge often leads to
! incorrect patterns and wrong interpretations. This can cause serious damage to businesses.
A Day in a Data Scientist’s Life
Basic Skills of a Data Scientist
A Data Scientist should be able to

• Ask the right questions
• Understand data structure
• Interpret and wrangle data
• Apply statistical and mathematical methods
• Visualize data and communicate with stakeholders
• Work as a team player
Sources of Big Data
Data Scientists work with different types of datasets for various purposes. Now that Big Data is generated every
second through different media, the role of Data Science has become more important.
The 3 Vs of Big Data
Big Data is characterized by the following:
Volume Enormous amount of data generated from various sources
Large amount of data streaming in at great speeds, which requires quick

Velocity
data processing
Variety Different formats of data: Structured, Semi-structured, and Unstructured
Big Data is a huge collection of data stored on distributed systems/machines popularly referred to as Hadoop
clusters. Data Science helps extract information from the Data and build information-driven enterprises.
Different Sectors Using Data Science
Various sectors use Data Science to extract the information they need to create different services and
products.
Using Data Science—Social Network Platforms
LinkedIn uses data points from its users to provide them with relevant digital services and data products.
Profile
Groups
Location
Digital
Information
Data Points
Services
Connections
Data
Products
Post
Likes
Using Data Science—Search Engines
Google uses Data Science to provide relevant search recommendations as the user types a query.
Search keyword
Autocomplete feature is fed by data models (Machine Learning)
Fast and real-time analytics is made possible Influencing Factors

by modern and advanced infrastructure, 1. Query Volume – Unique and verifiable users
tools, and technologies 2. Geographical locations
3. Keyword/phrase matches on the web
4. Some scrubbing for inappropriate content
Using Data Science—Healthcare
Wearable devices use Data Science to analyze data gathered by their biometric sensors.
Biometric data IOT Gateway Data transfer to Enterprise

transfer Servers Infrastructure
Wearable device
Make informed
decisions
Data Analytics
Engagement
Dashboard
Using Data Science—Finance
A Loan Manager can easily access and sift through a loan applicant’s financial details using Data
Science.
Data transfer to Enterprise

Loan Application Servers Infrastructure
Portal
Loan Applicant
Credit report, Credit history,

Approved amount, Risk etc.
Make informed Engagement

decisions Dashboard Data Analytics
Using Data Science—Public Sector
The governments in different countries share large datasets from various domains with the public.
Data.gov is a website hosted and maintained by the U.S. government.
Large collection of datasets
Sectors/Domains
The Real Challenge
Some of the challenges Data Scientists face in the real

world are listed here.
• Data quality doesn’t conform to the set standards.
• Data integration is a complex task.
• Data is distributed into large clusters in HDFS, which
is difficult to integrate and analyze.
• Unstructured and semi-structured data are harder
to analyze.
Data Analytics and Python
Python deals with each stage of data analytics efficiently by applying different libraries and packages.
Acquire
Wrangle
Explore
Model
Data Analytics
Visualize
Bokeh
Python Tools and Technologies
Python is a general purpose, open source, programming language that lets you work quickly and
integrate systems more effectively.
Benefits of Python
Easy to learn
Open source
Efficient and multi platform support
Huge collection of libraries,

functions and modules
Big open source community
Integrates well with enterprise apps

and systems
Great vendor and product support

Big Data Platforms and Processing Frameworks for Python
Python is supported by well-established data platforms and processing frameworks that help analyze
data in a simple and an efficient way.
Data Scientist
Python Tool for data analysis and processing
Big Data Processing Framework
Enterprise Big Data Platform
Big Data
Key Takeaways
Data Science is a discipline that combines aspects of statistics, mathematics,

programming, and domain expertise.
Data Scientists solve big problems in public and private sectors.
A lot of datasets are freely available to apply Data Science and turn them
into data services and data products.
Data Scientists are more in demand with the evolution of Big Data and
real-time analytics.
Python is a powerful language and a preferred tool for Data

Science.
Quiz
QUIZ
A Data Scientist _____.
1
a. asks the right questions
b. acquires data
c. performs data wrangling and data visualization
d. All of the above

QUIZ
A Data Scientist:
1
a. Asks the right questions
b. Acquires data
c. Performs data wrangling and data visualization
d. All of the above
The correct answer is d.
Explanation: A Data Scientist asks the right questions to the stakeholders, acquires data from various sources and data points,
performs data wrangling that makes the data available for analysis, and creates reports and plots for data visualization.
QUIZ
The Search Engine’s Autocomplete feature identifies unique and verifiable users who
search for a particular keyword or phrase _____. Select all that apply.
2
a. to scrub inappropriate content.
b. to build a Query Volume.
c. to tag the location to a query.
d. to find similar instances on the web.

QUIZ
The Search Engine’s Autocomplete feature identifies unique and verifiable users who
search for a particular keyword or phrase _____. Select all that apply.
2
a. to scrub inappropriate content
b. to build a Query Volume
c. to tag the location to a query
d. to find similar instances on the web
The correct answer is b, c .
Explanation: The Search Engine’s Autocomplete feature identifies unique and verifiable users who search for a particular
keyword or phrase to build a Query Volume. It also helps identify the users’ locations and tag them to the query, enabling it to be
location-specific.
QUIZ
What is the sequential flow of Data Analytics?
3
a. Data wrangling, exploration, modeling, acquisition, and visualization
b. Data exploration, acquisition, modeling, wrangling, and visualization
c. Data acquisition, wrangling, exploration, modeling, and visualization
d. Data modeling, acquisition, exploration, wrangling, and visualization

QUIZ
What is the sequential flow of Data Analytics?
3
a. Data wrangling, exploration, modeling, acquisition, and visualization
b. Data exploration, acquisition, modeling, wrangling, and visualization
c. Data acquisition, wrangling, exploration, modeling, and visualization
d. Data modeling, acquisition, exploration, wrangling, and visualization
The correct answer is c.
Explanation: In Data Analytics, the data is acquired from various sources and is then wrangled to ease its analysis. This is
followed by data exploration and data modeling. The final stage is data visualization, where the data is presented and the
patterns are identified.
This concludes “Data Science Overview.”
The next lesson is “Data Analytics Overview.”
Lesson 2 – Data Analytics Overview
What’s In It For Me
Data Analytics process and its steps
Skills and tools required for Data Analysis
Challenges of the Data Analytics Process
Exploratory Data Analysis technique
Data visualization techniques
Hypothesis testing to analyze data

Why Data Analytics
Data by itself is just an information source. But unless you can understand it, you will not be able to use
it effectively.
Date Description Deposit Withdrawal Balance
Apr 1 ATM Post Debit 100 $200,000
Apr 2 Paypal Tranfer 231054 200 $202,000
Apr 3 Simplilearn course fee 150 $200,500

Information source;
Apr 4 Starluck Café 210 $198,400 overall patterns not
clearly visible
Apr 5 Walcart TX 230 $196,100
Apr 6 ebuy swiss watch 239 250 $193,600

Apr 7 Caterpallor black boots men 270 $190,900
Apr 8 Halo blue shirt 831 160 $189,300

Why Data Analytics (contd.)
When the transaction details are presented as a line chart, the deposit and withdrawal patterns
become apparent.
Overall pattern
Amount in $
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Bank Transaction Details for April 2016
Why Data Analytics (contd.)
When the transaction details are presented as a line chart, the deposit and withdrawal patterns become
apparent. It helps view and analyze general trends and discrepancies.
Mar-16
Apr-16
May-16 Discrepancy
Amount in $
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Bank Transaction Details
Introduction to Data Analytics
Data Analytics is a combination of processes to extract information from datasets.

Introduction to Data Analytics
Data Analytics is a combination of processes to extract information from datasets.

Business Problem
The process of analytics begins with questions or business problems of stakeholders.
Sales Inventory
Customers Traffic volume
Business problems trigger the need to

analyze data and find answers.
Data Acquisition
Collect data from various sources for analysis to answer the question raised in step 1.
Twitter, Facebook,
LinkedIn, and other
social media and
information sites
provide streaming APIs.
Data Scientist Expertise:
File handling
• Database Skills Data
• File formats
• Web scraping Server logs can be
extracted from
enterprise system
servers to analyze and
optimize application
performance.
Data Wrangling and Exploration
Data wrangling is the most important phase of the data analytic process.
Data cleansing Data manipulation Data discovery Data pattern
Data Wrangling Data Exploration

Data Wrangling—Challenges
This phase includes data cleansing, data manipulation, data aggregation, data split, and reshaping of data.
Causes of challenges in the data wrangling phase:
• Unexpected data format

• Erroneous data
• Voluminous data to be manipulated
• Classifying data into linear or clustered
• Determining relationship between observation, feature, and response
Data wrangling is the most challenging phase and takes up 70% of the data scientist’s time.
Data Exploration—Model Selection
This phase includes data cleansing, data manipulation, data aggregation, data split, and reshaping of data.
Model selection
• Based on the overall data analysis process
• Should be accurate to avoid iterations
• Depends on pattern identification and algorithms
• Depends on hypothesis building and testing
• Leads to building mathematical statistical functions
Exploratory Data Analysis (EDA)
Let’s take a look at the exploratory data analysis phase.
APPROACH FOCUS ASSUMPTIONS EDA TECHNIQUES
EDA techniques Quantitative:

EDA approach The focus is on make minimal or Provides numeric
studies the data to data; its structure, no assumptions. outputs for the
recommend outliers, and They present and inputted data
suitable models models suggested show all the Graphical: Uses
that best fit the by the data. underlying data statistical
data. without any data functions for
loss. graphical output
EDA— Quantitative Technique
EDA – Quantitative technique has two goals, measurement of central tendency and
spread of data.
Measurement of Central Tendency
Mean Mean is the point which indicates how centralized the data points
are.
• Suitable for symmetric distributions
Median Median is the exact middle value.

• Suitable for skewed distributions and for catching outliers
in the dataset
Mode Mode is the most common value in the data (frequency).
Measurement of Spread
Variance Variance is approximately the mean of the squares of the
deviations.
Standard Deviation Standard deviation is the square root of the variance.
Inter-quartile Range Inter-quartile range is the distance between the 75th and 25th
percentile. It’s essentially the middle 50% of the data.
EDA – Graphical Technique
Histograms and Scatter Plots are two popular graphical techniques to depict data.
25
Histogram graphically summarizes the distribution of

a univariate dataset.
Frequency
20
It shows:
• the center or location of data (mean, median, or mode)
• the spread of data
15
• the skewness of data

• the presence of outliers
10
• the presence of multiple modes in the data

10 15 20 25
Miles Per Gallon
EDA – Graphical Technique
Histograms and Scatter Plots are two popular graphical techniques to depict data.
A Scatter plot represents relationships between two variables.

It can answer these questions visually:
• Are variables X and Y related?
• Are variables X and Y linearly related?
• Are variables X and Y non-linearly related?
• Does change in variation of Y depend on X?
• Are there outliers?
Quiz Knowledge Check
KNOWLEDGE What is the goal of data acquisition?
CHECK
Select all that apply.
a. Collect data from various data sources
b. Answer business questions through graphics
c. Collect web server logs
d. Scrape the web through web APIs

KNOWLEDGE What is the goal of data acquisition?
CHECK
a.
Collect data from various data sources
b. Answer business questions through graphics
c. Collect web server logs
d.
Scrape web through web APIs
The correct answer is a, c ,d.
Explanation: Data acquisition is a process to collect data from various data sources such as RDBMS, No
SQL databases, web server logs and also scrape the web through web APIs.
KNOWLEDGE What is the Exploratory Data Analysis technique?
CHECK
a. Analysis of data using quantitative techniques
b. Conducted only on a small subset of data
c. Analysis of data using graphical techniques
d. Suggests admissible models that best fit the data

What is the Exploratory Data Analysis technique?
KNOWLEDGE
CHECK
a.
Analysis of data using quantitative techniques
b. Conducted only on small subset of data
c. Analysis of data using graphical techniques
d.
Suggests models that best fit the data
The correct answer is a, c, d .
Explanation: Most EDA techniques are graphical in nature with a few quantitative techniques and also
suggest models that best fit the data. They use almost the entire data with minimum and no assumptions.
Conclusion or Predictions
This step involves reaching a conclusion and making predictions based on the data
analysis.
This phase:
• Involves heavy use of mathematical and statistical functions
• Requires model selection, training, and testing to help in forecasting
• Is called “machine learning” as data analysis is fully or semi-automated
with minimal or no human intervention

Meaning of Hypothesis
Hypothesis is used to establish the relationship between dependent and independent

variables.
Hypothesis building begins in the data exploration stage but

becomes more mature in the conclusion or prediction phase.
Data Exploration Stage Conclusion and Prediction
Key Considerations of Hypothesis Building

Testable explanations of a problem or observation
Used in quantitative and qualitative analyses to provide research solutions
Involves two variables, one dependent on another
Independent variable manipulated by the researcher
Dependent variable changes when the independent variable changes
Hypothesis Building Using Feature Engineering
Domain knowledge leads to hypothesis building using feature engineering.
Feature engineering involves domain expertise to:

• Make sense of data
• Construct new features from raw data
automatically
• Construct new features from raw data manually
Hypothesis Building Using a Model
There are three phases to hypothesis building which are model building, model evaluation,
and model deployment.
Phase 1: Model Building

• Identify best input variables
• Evaluate the model’s capacity to forecast with
these variables
Phase 2: Model Evaluation

• Train and test the model for accuracy
• Optimize model accuracy, performance, and
comparisons with other models
Phase 3: Model Deployment

• Use the model for prediction
• Use the model to compare actual outcome with
expectations
Hypothesis Testing
Draw two samples from the population and calculate the difference between their means.
μ1 Calculating the
difference
S1
between the two
means is
“hypothesis
μ2 testing”.
S2
Hypothesis Testing
Draw two samples from the population and calculate the difference between their means.
Alternative Hypothesis
• Proposed model outcome is
accurate and matches the
data.
• There is a difference between
the means of S1 and S2.
Null Hypothesis
• Opposite of the alternative
hypothesis.
• There is no difference
between the means of S1 and
S2.
Hypothesis Testing Process
Choosing the training and test dataset and evaluating them with the null and alternative
hypothesis.
Usually the training dataset is between 60% and 80% of the big dataset and the test dataset is
between 20% and 40% of the big dataset.
Communication
Data analysis process and results are presented to stakeholders.
Forms of Data analysis presentation:

• Visual graphs
• Plotting maps
• Reports
• Whitepaper reports
• PowerPoint presentations
Data Visualization
Data visualization techniques are used for effective communication of data.
Benefits of data visualization: Examples of data visualization:

• Simplifies quantitative information through • Presenting information about new and
visuals existing customers on the website and their
• Shows the relationship between data points behavior when they access the website
and variables • Representing web traffic pattern for the
• Identifies patterns website, for example, more activity on the
• Establishes trends website in the morning than in the evening
Plotting
Plotting is a data visualization technique used to represent underlying data through

graphics.
Features of plotting:
• Plotting is like telling a story about
data using different colors, shapes,
and sizes.
• Plotting shows the relationship
between variables.
• Example:
o Change in value of Y results in
change in value of X.
o X is independent of y.
Data Types for Plotting
There are various data types used for plotting.

There are two types of numerical data:
Discrete Data – Distinct or counted values
Example: Number of employees in a company or number of students in a class
Continuous Data – Values within a range that can be measured
Example: Height can be measured in feet or inches and weight can be measured in pounds or kilograms
Numerical Data
There are two types of categorical data:

Cluster or group – Grouped values
Example: Students can be divided into different groups based on height – Tall, Medium, and Short
Ordinal data – Grouped values as per ranks
Categorical Data Example: A ranking system; a five-point scale with ranks like “Agree,” “Strongly agree,” and “Disagree”
Data measured in time blocks such date, month, year, and time (hours, minutes, and seconds
Time Series
Types of Plot
Different data types can be visualized using various plotting techniques.

Data Analytics – An Iterative Process
Data analytics is an iterative process involving tracing back the steps, often to ensure that
you are on the right track.
Process Result: Question is answered or business problem is solved.

Data Analytics – Skills and Tools
Skills and tools required for each step of the data analysis process.
Question or Data Data Data Conclusion or Communication or

Business Problem Acquisition Wrangling Exploration Predictions Data Visualization
• Ability to ask • BeautifulSoup • CSV or other • NumPy • Scikit-Learn – • Pandas

appropriate for web file knowledge • SciPy the main • Database
questions and scraping • NumPy • Pandas machine • Matplotlib
know the • CSV or other • Pandas • Matplotlib learning library • PPT
business file knowledge • Database • CSV or other • CSV or other
• Domain • NumPy • SciPy file knowledge file knowledge
knowledge • Pandas • NumPy
• Passion for • Database • Pandas
data • Database
• Analytical • SciPy
approach
Knowledge Check
KNOWLEDGE Which plotting technique is used for continuous data?
CHECK
a. Regression plot
b. Line chart
c. Histogram
d. Heat map
Which plotting technique is used for continuous data?
KNOWLEDGE
CHECK
a.
Regression plot
b. Line chart
c. Histogram
d.
Heat map
The correct answer is b , c .
Explanation: Line charts and histograms are used to plot continuous data.
Quiz
QUIZ
Which Python library is the main machine learning library?
1
a. Pandas
b. Matplotlib
c. Scikit-learn
d. NumPy
QUIZ
Which Python library is the main machine learning library?
1
a. Pandas
b. Matplotlib
c. Scikit-learn
d. NumPy

Explanation: SciKit-learn is the main machine library in Python.
QUIZ Which of the following includes data transformation, merging, aggregation, group by
operation, and reshaping?
2
a. Data Acquisition
b. Data Visualization
c. Data Wrangling
d. Machine learning
QUIZ Which of the following includes data transformation, merging, aggregation, group by
operation, and reshaping?
2
a. Data Acquisition
b. Data Visualization
c. Data Wrangling
d. Machine learning
The correct answer is c .

Explanation: Data wrangling includes data transformation, merging, aggregation, group by operation, and
reshaping.
QUIZ
Which measure of central tendency is used to catch outliers in the data?
3
a. Mean
b. Median
c. Mode
d. Variance
QUIZ
Which measure of central tendency is used to catch outliers in the data?
3
a. Mean
b. Median
c. Mode
d. Variance
The correct answer is b.

Explanation: Median is the exact middle value and most suitable to catch outliers.
QUIZ
In hypothesis testing, the proposed model is built on:
4
a. the entire dataset.
b. the test dataset.
c. a small subset.
d. the training dataset.

QUIZ
In hypothesis testing, the proposed model is built on:
4
a. the entire dataset.
b. the test dataset.
c. a small subset.
d. the training dataset.
The correct answer is d .

Explanation: The proposed model is built on the training dataset in hypothesis testing.
QUIZ
Beautiful soup library is used for _____.
5
a. data wrangling
b. web scraping
c. plotting
d. machine learning
QUIZ
Beautiful soup library is used for _____.
5
a. data wrangling
b. web scraping
c. plotting
d. machine learning.

Explanation: BeautifulSoup is used for web scraping and mainly used in the data acquisition phase.
Key Takeaways
Data analytics is used to solve business problems.

Data analysis requires a number of skills and tools.
Data wrangling, data exploration, and model selection
processes are challenging.
EDA includes quantitative and graphical techniques.
Data visualization helps show data characteristics and patterns
effectively.
Hypothesis testing establishes the relationship between
dependent and independent variables in data analytics.
This concludes “Data Analytics”
The next lesson is “Statistical Analysis and Business
Applications”
Lesson 3 – Statistical Analysis and Business Applications
What You’ll Learn
The difference between statistical and non-statistical

analysis
The two major categories of statistical analysis and their
differences
The statistical analysis process
Mean, median, mode, and percentile
Data distribution and the various methods of

representing it
Hypothesis testing and the Chi square test
Types of frequencies
Correlation matrix and its uses

Introduction to Statistics
Statistics is the study of the collection, analysis, interpretation, presentation, and organization of data.
Well-informed decision
PROBLEMS SOLVED
DATA
COMPLEX PROBLEMS
Introduction to Statistics
Statistics is the study of the collection, analysis, interpretation, presentation, and organization of data.
Tools available to analyze

data:
• Statistical principles
• Functions
• Algorithms
What you can do using statistical tools:

• Analyze the primary data
• Build a statistical model
• Predict the future outcome
Statistical and Non-statistical Analysis
Statistical Analysis Non-statistical Analysis
Statistical Analysis is: Non-statistical Analysis is:

• scientific • based on very generic information
• based on numbers or statistical values • exclusive of statistical or quantitative
• useful in providing complete insight to the data
Although both forms of analysis provide results, quantitative analysis provides more insight
and a clearer picture. This is why statistical analysis is important for businesses.
Major Categories of Statistics
There are two major categories of statistics: Descriptive analytics and inferential analytics
Descriptive analysis organizes the data and focuses on the main characteristics of the data.
HIGH
NUMBER OF STUDENT
MEDIUM
MATH SCORES
LOW
1847369250
0 1 2 3 4 5 6 7 8 9
LOW MEDIUM HIGH

Major Categories of Statistics
Inferential analytics uses the probability theory to arrive at a conclusion.
• Random sample is drawn

from the population
• Used to describe and
make inferences about
the population
Inferential analytics is valuable when it is not possible to examine each member of the
population.
Major Categories of Statistics – An Example
Study of the height of the population
Tall
Inferential Method
Medium Categorize height as “Tall,” “Medium,” and “Short”
Take a sample to study from the population.
Short
Descriptive Method
Record the height of each and every person.
Provide the tallest, shortest, and average height of
the population.
Statistical Analysis Considerations
Purpose Document Questions Define Population of Determine Sample

Clear and well-defined Prepare a Interest Based on the
questionnaire in Select population based purpose of study
advance on the purpose of
analysis
Population and Sample
A population consists of various samples. The samples together represent the population.
A sample is:
• The part/piece drawn from the population
• The subset of the population
• A random selection to represent the characteristics of
the population
• Representative analysis of the entire population
Statistics and Parameters
“Statistics” are quantitative values calculated from the sample.
“Parameters” are the characteristics of the population.
Sample  Xo, X1,X2……….Xn
Population
Sample Statistics Formula
Parameters
1
x   xi
Mean µ x n

1

2 2
 
2 2
Variance S S n 1
( x i
x )
Standard
Deviation
 S S
1
 ( 
xi )x 2
n 1
Terms Used to Describe Data
Typical terms used in data analysis are:
SEARCH INSPECT CHARACTERIZE CONCLUSION
“Inspect” refers to “Characterize” refers “Conclusion” refers to

“Search” is used to find
studying the shape and to determining the preliminary or high-
unusual data. Data that
spread of data. central tendency of level conclusions about
does not match the
the data. the data.
parameters.
Statistical Analysis Process
There are four steps in the statistical analysis process.
Step 1: Find the population of interest that suits the purpose of

statistical analysis.
Step 2: Draw a random sample that represents the population.
Step 3: Compute sample statistics to describe the spread and shape of
the dataset.
Step 4: Make inferences using the sample and calculations. Apply it back
to the population.
Data Distribution
The collection of data values arranged in a sequence according to their relative frequency and
occurrences.
1.5
Range of the data refers to
minimum and maximum values.
1
Frequency indicates the number

0
19.1% 19.1%
of occurrences of a data value.
-0.5
15.0% 15.0% Central tendency indicates data
-1
9.2% 9.2% accumulation toward the middle
-1.5 4.4% 4.4% of the distribution or toward the
1.7% 1.7%
end.
-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3
Measures of Central Tendency
The measures of central tendency are Mean, Median, and Mode.
Mean is the average.

Determine the mean score of these Math scores. Median is the 50th percentile.
1. 80 55 60 70 75 78 80 80 80 90
2. 70 Median = 78
3. 75
4. 90
5. 80 Mode is the most frequent value.
6. 78 55 60 70 75 78 80 80 80 90
7. 55 Mode = 80
8. 60
9. 80
Ʃ [80+70+75+90+80+78+55+60+80]/9
Mean = 74.22
Percentiles in Data Distribution
A percentile (or a centile) indicates the value below which a given percentage of observations fall.
Observations
98
95
92 75th percentile =91 Third Quartile
90
85
81 50th percentile =80 Second Quartile or Median
79
70
63 25th percentile =59 First Quartile
55
47
42
Dispersion
Dispersion denotes how stretched or squeezed a distribution is.
Observations
98
95
92 75th percentile = 91 Range: The difference between the maximum and
90 minimum values
85
Inter-quartile Range: Difference between the 25th
81 50th percentile = 80 and 75th percentiles
79
70 Variance: Data values around the Mean. (74.75)
63 25th percentile = 59
Standard Deviation: Square root of the variance
55 measured in small units
47
42
Knowledge Check
KNOWLEDGE
What does frequency indicate?
CHECK
a. Range of the values present in the dataset
b. Number of occurrences of a particular value in a dataset
c. How spread out the data is
d. Size of the sample drawn from a population

KNOWLEDGE
CHECK
What does frequency indicate?
a. Range of the values present in the dataset
b. Number of occurrences of a particular value in a dataset
c. How spread out the data is
d. Size of the sample drawn from a population
Explanation: Frequency indicates the number of occurrences of a particular value in a dataset.

Histogram
Graphical representation of data distribution
Features of a Histogram:
• It was first introduced by Karl Pearson. 5.0
• To construct a Histogram, “bin” the range of values. 4.0

• Bins are consecutive, non-overlapping intervals of a
Frequency
3.0
variable.
• Bins are of equal size. 2.0
• The bars represent the bins. 1.0

• The height of the bar represents the frequency of
0
the values in the bin.
20 30 40 50 60 70 80 90 100
• It helps assess the probability distribution of a Age
variable.
bin
Bell Curve – Normal Distribution
The bell curve is characterized by its bell shape and two parameters, mean and standard deviation.
Normal Curve
Bell curve is: Standard Deviation
• Symmetric around the mean, Mean
• Symmetric on both sides of the center,

Dense
• Having equal mean, median, and mode values,
• Denser in the center and less dense in the tails or sides, 19.1% 19.1%
• Defined by mean and standard deviation, and tails 15.0% 15.0% tails
• Known as the “Gaussian” curve. 9.2% 9.2%
0.1% 0.5% 0.5% 0.1%
4.4% 4.4%
1.7% 1.7%
-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3
The Bell curve is fully characterized by the mean (μ) and standard deviation (σ).
The Bell Curve
The Bell curve is divided into three parts to understand data distribution better.
Peak Peak = Within one standard

deviation from the mean
Flanks
Flanks = Between one and two
standard deviations from the mean
Tail
Tail = Beyond two standard
deviations from the mean
Bell Curve – Left Skewed
Skewed data distribution indicates the tendency of the data distribution to be more spread out on one side.
Left Skewed
• The data is left skewed. 80
• Mean < Median 70
• The distribution is negatively skewed. 60

50
Frequency
• Left tail contains large distributions.
40
30
20
10
0
1 2 3 4 5 6 7 8 9 10 11 12 13
Measurement
Bell Curve – Right Skewed
Skewed data distribution indicates the tendency of the data distribution to be more spread out on one
side.
Right
• The data is right skewed. Skewed
80
• The distribution is positively skewed. 70
• Mean > Median 60

50
Frequency
• Right tail contains large distributions.
40
30
20
10
0
1 2 3 4 5 6 7 8 9 10 11 12 13
Measurement
Kurtosis
Kurtosis describes the shape of a probability distribution.
Kurtosis
There are:
measures the tendency of the data toward
the center or toward the tail. (+) Leptokurtic
• Different ways of quantifying a theoretical
Platykurtic
distribution.
is negative kurtosis.
(0) Mesokurtic (Normal)
Mesokurtic represents
• Corresponding a normal
ways of distribution
estimating it from curve.
a sample ofispopulation.
Leptokurtic positive kurtosis. (-) Platykurtic
Knowledge Check
KNOWLEDGE
Which of the following is true for a normal distribution?
CHECK
a. Mean and median are equal
b. Mean and mode are equal
c. Mean, median, and mode are equal
d. Mode and median are equal

KNOWLEDGE
CHECK
Which of the following is true for a normal distribution?
a. Mean and median are equal
b. Mean and mode are equal
c. Mean, median and mode are equal
d. Mode and median are equal
The correct answer is . c
Explanation: for Bell curve mean, median, and mode are equal.
Hypothesis Testing
Hypothesis testing is an inferential statistical technique that determines if a certain condition is true for the
population.
Alternative Hypothesis (H1) Null Hypothesis (H0)
A statement that has to be A statement of “no effect” or “no

concluded as true. difference”.
It’s the logical opposite of the

It’s a research hypothesis.
alternative hypothesis.
It needs significant evidence to It indicates that the alternative

support the initial hypothesis. hypothesis is incorrect.
If the alternative hypothesis Weak evidence of alternative

garners strong evidence, reject hypothesis indicates that the null
the null hypothesis. hypothesis has to be accepted.
Hypothesis Testing – Error Types
Representation of decision parameters using null hypothesis
• Rejects the null hypothesis when it is true
Type I Error (α) • The probability of making Type I error is represented by
α
• Fails to Reject the null hypothesis when it false
Type II Error (β) • The probability of making Type II error is represented
by β
• The probability of observing extreme values
p-value • Calculated from collected data
Decision Ho is True Ho is False
Fail to Reject Null Correct Type II Error
Reject Null Type I Error Correct

Hypothesis Testing - Process
There are four steps to the hypothesis testing process.
Step 4
Step 3
Make decision
Step 2 p-value < α
Collect Data p-value ≥ α
Step 1 Collect sample from
Set α Significant population
Level
Set Hypothesis Set α or choose the
H0 (μ1 = μ2) : Equality significant level for
H1(μ1 ≠ μ2) : Difference the population.
Reject the null hypothesis if p-value < α

Fail to reject the null hypothesis if p-value ≥ α
Perform Hypothesis Testing
An example of clinical trials data analysis.
Company A Company B
Null Hypothesis:
Both medicines are
equally effective. Alternative Hypothesis:
Both medicines are
NOT equally effective.
Data for Hypothesis Testing
There are three types of data on which you can perform hypothesis testing.
Continuous Data Binomial Data Poisson Data
Evaluate the mean, Evaluate the Evaluate rate of

median, standard percentage, general occurrence or
deviation, or variance. classification of data. frequency.
Types of Variables
There are three types of variables in categorical data.
Nominal Variables Ordinal Variables

• Values with no logical ordering • Values are in logical order
• Variables are independent of • Relative distance between
each other two data values is not clear
• Sequence does not matter
Association
Two variables are associated 85% 15% 68% 32%
or independent of each other.
85% 15% 95% 55%
Chi-Square Test
It is a hypothesis test that compares the observed distribution of your data to an expected distribution of data.
Test of Association:
To determine whether one variable is associated with a different variable. For
example, determine whether the sales for different cellphones depends on the city
or country where they are sold.
Test of Independence:
To determine whether the observed value of one variable depends on the
observed value of a different variable. For example, determine whether the color
of the car that a person chooses is independent of the person’s gender.
Test is usually applied when there are two categorical variables from a single population.
Chi Square Test - Example
An example of Chi-Square test.
Null Hypothesis
• There is no association between gender and

purchase.
• The probability of purchase does not change
for 500 dollars or more whether female or <$500 >$500
male.
fo .55 .45
Alternative Hypothesis
fo .75 .25
• There is association between gender and
purchase.
• The probability of purchase over 500 dollars
is different for female and male.
Types of Frequencies
Expected and observed frequencies are the two types of frequencies.
Expected Frequencies (fe) Purchases
The cell frequencies that are expected in a <$500 >$500

bivariate table if the two tables are
statistically independent.
fo .55 .45
fo .75 .25
Observed Frequencies (fo)
• There is association between gender and No Association

purchase. Observed Frequency = Expected Frequency
• The probability of purchase over 500 dollars Association
is different for female and male. Observed Frequency ≠ Expected Frequency
Features of Frequencies
The formula for calculating expected and observed frequencies using Chi Square:
( fe  fo ) 2
 2
fe
Features of Expected and Observed frequencies:

• Requires no assumption of the underlying population
• Requires random sampling
Knowledge Check
KNOWLEDGE
CHECK
In Chi-Square test, there is no association of variables if _____.
a. Observed Frequency ≠ Expected Frequency
b. Observed Frequency = Expected Frequency
c. Independent of observed frequencies
d. Independent of expected frequencies

KNOWLEDGE
CHECK
In Chi-Square test, there is no association of variables if:
a. Observed Frequency ≠ Expected Frequency
b. Observed Frequency = Expected Frequency
c. Independent of observed frequencies
d. Independent of expected frequencies
The correct answer is b .
Explanation: Observed Frequency = Expected Frequency indicates no association.

Correlation Matrix
A Correlation matrix is a square matrix that compares a large number of variables.
Correlation matrix – a square matrix

n × n Matrix
(very large number of rows and columns)
(0,0) (0,1) (0,2) Correlation coefficient measures the extent to which two
variables tend to change together.
(1,0) (1,1) (1,2)
The coefficient describes both the strength and direction of
(2,0) (2,1) (2,2) the relationship.
3 × 3 matrix (simple square matrix)
Correlation Matrix
A Correlation matrix is a square matrix that compares a large number of variables.
It evaluates the linear relationship between two continuous variables.

Pearson product moment
correlation Linear relationship means that a change in one variable results in a
proportional change in the other.
It evaluates the monotonic relationship between two continuous or ordinal

variables.
Spearman rank order • Monotonic relationship means that the variables tend to change together
correlation though not necessarily at a constant rate.
• The correlation coefficient is based on the ranked values for each variable
rather than the raw data.
Correlation Matrix - Example
An example of a correlation matrix calculated for a stock market.
A correlation matrix that is calculated for the stock market will probably show the short-term,
medium-term, and long-term relationship between data variables.
Inferential Statistics
Inferential statistics uses a random sample from the data to make inferences about the population.
Inferential statistics can be used only under the following conditions:

• A complete list of the members of the population is available.
• A random sample has been drawn from the population.
• Using a pre-established formula, you determine that the sample size is
large enough.
Inferential statistics can be used even if the data does not meet the
criteria.
• It can help determine the strength of the relationships within the
sample.
• If it is very difficult to obtain a population list and draw a random
sample, do the best you can with what you have.
Applications of Inferential Statistics
Inferential Statistics has its uses in almost every field such as business, medicine, data science, and so on.
Inferential Statistics 120
• Is an effective tool for forecasting. 100
• Is used to predict future patterns. 80
60
40
20
0
Quiz
QUIZ
If a sample of five boxes weigh 90, 135, 160, 115, and 110 pounds, what will be the
median weight of this sample?
1
a. 160
b. 115
c. 90
d. 135
QUIZ
If a sample of five boxes weigh 90, 135, 160, 115, and 110 pounds, what will be the
median weight of this sample?
1
a. 160
b. 115
c. 90
d. 135

Explanation: Arrange in a sequential order and the middle number will be the median. If the set of numbers is even then take
the average or mean of the two numbers in the middle.
QUIZ
Identify the parameters that characterize a bell curve. Select all that apply.
2
a. Variance
b. Mean
c. Standard deviation
d. Range
QUIZ
Identify the parameters that characterize a bell curve. Select all that apply.
2
a. Variance
b. Mean
c. Standard deviation
d. Range
The correct answer is b,c .

Explanation: Bell Curve is completely characterized by mean and standard deviation.
QUIZ
Identify the accurate statement about the relationship between standard deviation
and variance.
3
a. Standard deviation is the square root of variance.
b. Variance is the square root of standard deviation.
c. Both are inversely proportional.
d. Both are directly proportional.

QUIZ
Identify the accurate statement about the relationship between standard deviation
and variance
3
a. Standard deviation is the square root of variance.
b. Variance is the square root of standard deviation.
c. Both are inversely proportional.
d. Both are directly proportional.
The correct answer is a.
Explanation: Standard deviation is the square root of variance.

QUIZ
Identify the hypothesis decision rules. Select all that apply.
4
a. Reject the null hypothesis if p-value < α
b. Is independent of p-value
c. Fail to reject the null hypothesis if p-value ≥ α
d. Is independent of α
QUIZ
Identify the hypothesis decision rules. Select all that apply.
4
a. Reject the null hypothesis if p-value < α
b. Is independent of p-value
c. Fail to reject the null hypothesis if p-value ≥ α
d. Is independent of α
The correct answer is a, c .
Explanation: A hypothesis decision rule :

•Reject the null hypothesis if p-value < α
•Fail to reject the null hypothesis if p-value ≥ α
Key Takeaways
Statistics is the study of the collection, analysis, interpretation,

presentation, and organization of data.
Statistical analysis is more reliable when compared to non-statistical
analysis.
Descriptive and inferential are the two major categories of statistics.
Mean, median, and mode are measures of central tendency, while variance
and standard deviation measure the spread of data.
The spread of distribution is called dispersion and is graphically
represented by a histogram and a bell curve.
Hypothesis testing is an inferential statistical technique that is useful for
forecasting future patterns.
Chi-Square test is a hypothesis test that compares observed distribution to
an expected distribution.
The correlation coefficient or covariance is measured with the help of
correlation matrix.
This concludes “Statistical Analysis and Business
The next lessonApplications”
is “Data Analytics Overview”
Lesson 04—Python: Environment Setup and Essentials
What You’ll Learn
How to install Anaconda and Jupyter notebook
Some of the important data types supported by Python
Data structures such as lists, tuples, sets, and dicts
Slicing and accessing the four data structures
Few basic operators and functions
Some important control flow statements

Python: Environment Setup and Essentials
Python Environment Setup
Quick Recap: Python for Data Science
You have seen how Python and its different libraries are useful in various aspects of Data Science.
Acquire
Wrangle
Explore
Model
Data Science
Visualize
Bokeh
Why Anaconda
To use Python, we recommend that you download Anaconda. Following are some of the reasons why
Anaconda is one of the best Data Science platforms:
Open source Python

distribution
400+ popular Python packages
Enterprise-ready data analytics

platform
Modern Data Science analytics

architecture
Multi-workload data analytics
Interactive visualizations,
governance, security, and
operational support
Supports Big Data environments

Installation of Anaconda Python Distribution
Currently, there are two versions of Python. You can download and use either of them although the 2.7
version is preferable.
Installation of Anaconda Python Distribution (contd.)
You can install and run the Anaconda Python distribution on different platforms.
Windows Mac OS Linux
Website URL:
https://www.continuum.io/downloads
Graphical Installer
• Download the graphical installer.
• Double-click the .exe file to install Anaconda and
follow the instructions on the screen.
Click each tab to know how to install Python on those operating systems.
Website URL:
Graphical Installer
• Download the graphical installer.
• Double-click the downloaded .pkg file and follow the instructions.
Command Line Installer

• Download the command line installer.
• In your terminal window, type the command listed below and
follow the given instructions:
Python 2.7:
bash Anaconda2-4.0.0-MacOSX-x86_64.sh
Website URL:
Command Line Installer

• Download the installer.
• In your terminal window, type the command line shown below
and follow the instructions:
Python 2.7:
bash Anaconda2-4.0.0-Linux-x86_64.sh
Jupyter Notebook
Jupyter is an open source and interactive web-based Python interface for Data Science and scientific
computing. Some of its advantages are:
Python language Content sharing and

support contribution
Big Data platform Built-in interactive

integration widgets
Jupyter Notebook: Installation
To install Jupyter notebook on your system, type the command shown here on Anaconda prompt and
press Enter to execute it.
Python: Environment Setup and Essentials
Python Primer
Getting Started
Import sys module
Print sys version
Import platform library

View python version
Comment line
Test string
Test number operation

Variables and Assignment
A variable can be assigned or bound to any value. Some of the characteristics of binding a variable in
Python are listed here:
The variable refers to the memory

location of the assigned value.
The variable appears on the left,

while the value appears on the right.
The data type of the assigned

value and the variable is the same.
Example—Variables and Assignment
Let us look at an example of how you can assign a value to a variable, and print it and its data type.
Assignment
Variable data value
Data type of the object

Multiple Assignments
You can access a variable only if it is defined. You can define multiple variables simultaneously.
Access variable
without assignment
Access variable after

assignment
Multiple assignments
Assignment and Reference
When a variable is assigned a value, it refers to the value’s memory location or address. It does not
equal the value itself.
Ref: <address 1> Ref: <address 12>

Garbage collected
7 7
8
Memory location Memory location

Basic Data Types: Integer and Float
Python supports various data types. There are two main numeric data types:
Numeric
Integer value
Integer Float
Float value
32-bit 64-bit
Basic Data Types: String
Python has extremely powerful and flexible built-in string processing capabilities.
With single quote
With double quote

String Three double quotes
Print string values

Basic Data Types: None and Boolean
Python also supports the Null and Boolean data types.
Null value type
Boolean type
Boolean type
Type Casting
You can change the data type of a number using type casting.
Float number
Type cast to integer
Type cast to string value

Data Structure: Tuple
A tuple is a one-dimensional, immutable ordered sequence of items which can be of mixed data types.
Create a tuple
View tuple
Access the data at

index value 1
Try to modify
the tuple
Error: A tuple is immutable

and can’t be modified
Data Structure: Accessing Tuples
You can access a tuple using indices.
Tuple
Access with positive index
Access with negative index

Data Structure: Slicing Tuples
You can also slice a range of elements by specifying the start and end indices of the desired range.
Tuple
Count starts with the first index

but stops before the second index
Even for negative indices, the count

stops before the second index
Data Structure: List
A list is a one-dimensional, mutable ordered sequence of items which can be of mixed data types.
Create a list
View a list
Modify a list: Add new items
Modify a list: Remove items
Access and remove list data using

element indices
Modify a list: Insert a new item at a

certain index
Data Structure: Accessing Lists
Just like tuples, you can access elements in a list through indices.
New modified list
Access with positive index
Access with negative index

Data Structure: Slicing Lists
Just like tuples, you can access elements in a list through indices.
New modified list
Count starts with the first index

but stops before the second index
Even for negative indices, the count

stops before the second index
Data Structure: Dictionary (dict)
Dictionaries store a mapping between a set of keys and a set of values.
Key Value
Any
Any data
Dictionary immutable
type
type
Define Modify View Lookup

Data Structure: View Dictionaries
You can view the keys and values in a dict, either separately or together, using the syntax shown here.
Create a
dictionary
View entire
dictionary
View only
keys
View only
values
Data Structure: Access and Modify dict Elements
You can also access and modify individual elements in a dict.
Access with key
Modify dictionary:
update
Modify dictionary:
delete
Data Structure: Set
A set is an unordered collection of unique elements.
Create a set
View the set
Create a set
View the
object type
View the set

Data Structure: Set Operations
Let us look at some basic set operations.
Create sets
OR – Union
set operation
View the output of the OR

operation
AND – Intersection set operation
View the output of the

NOT operation
Basic Operator: “in”
The “in” operator is used to generate a Boolean value to indicate whether a given value is present in the
container or not.
Create a list
Test presence of string

with ‘in’ operator
Create a string
Test presence of substrings

with ‘in’ operator
Basic Operator: “+”
The “plus” operator produces a new tuple, list, or string whose value is the concatenation of its arguments.
Create tuples
Add tuples
Create lists
Add lists
Create strings
Concatenate
strings
Basic Operator: “*”
The “multiplication” operator produces a new tuple, list, or string that “repeats” the original content.
* operator with tuple
* operator with list
* operator with string
The ‘*” operator does not actually multiply the values; it only repeats the values for the specified
number of times.
Functions
Functions are the primary method of code organization and reuse in Python.
Syntax Properties
def <name>(arg1, arg2, ..., argN): • Outcome of the function is communicated by

<statements> return statement
return <value> • Arguments in parenthesis are basically
assignments
Use def to create a function and assign it a name.

Functions: Considerations
Some important points to consider while defining functions:
• A function should always have a “return” value.

• If “return” is not defined, then it returns “None.”
• Function overloading is not permitted.
Function name
Create function Argument
Return type
Call function
Functions: Returning Values
You can use a function to return a single value or multiple values.
Create function
Call function
Create function
Multiple return
Call function
Built-in Sequence Functions
The built-in sequence functions of Python are as follows:
enumerate
Indexes data to keep track of indices and corresponding data mapping
sorted
Returns the new sorted list for the given sequence
reversed
Iterates the data in reverse order
Zip
Creates lists of tuples by pairing up elements of lists, tuples, or other sequence
Built-in Sequence Functions: enumerate
List of food
stores
Print data element and

index using enumerate
method
Create a data
element and index
map using dict
View the store map in the

form of key-value pair
Built-in Sequence Functions: sorted
This screen explains the sorted function
Sort numbers
Sort a string
value
Built-in Sequence Functions: reversed and zip
Let us see how to use reversed and zip functions
Create a list of
numbers for range 15
Use reversed function

to reverse the order
Define list of subjects

and count
Zip function to pair

the data elements of
lists
Returns list of tuples
View type
Control Flow: if, elif, else
The “if”, “elif,” and “else” statements are the most commonly used control flow statements.
If condition
Else block
Nested if, elif and else

Control Flow: “for” Loops
A “for” loop is used to iterate over a collection (like a list or tuple) or an iterator.
For loop iterator
The ‘continue’ statement
The ‘break’ statement

Control Flow: “while” Loops
A while loop specifies a condition and a block of code that is to be executed until the condition evaluates to
False or the loop is explicitly ended with break.
While condition
Control Flow: Exception Handling
Handling Python errors or exceptions gracefully is an important part of building robust programs and
algorithms.
Create function
Pass wrong argument type
Error
Exception handling with try –except block

Quiz
QUIZ
What is the data type of the object x = 3 * 7.5?
1
a. Int
b. Float
c. String
d. None of the above

QUIZ
What is the data type of the object x = 3 * 7.5?
1
a. Int
b. Float
c. String
d. None of the above

Explanation: Since one of the operands is float, the x variable will also be of the float data type.
QUIZ
Which of the data structures can be modified? Select all that apply.
2
a. tuple
b. list
c. dict
d. set
QUIZ
Which of the data structures can be modified? Select all that apply.
2
a. tuple
b. list
c. dict
d. set
The correct answer is b, c,. d
Explanation: Only a tuple is immutable and cannot be modified. All the other data structures can be modified.
What will be the output of the following code?
QUIZ
a. [‘NYC', 'Madrid']
b. [‘London', 'Madrid']
c. [‘Miami', 'Madrid']
d. [‘Miami', ‘Paris']
What will be the output of the following code?
QUIZ
a. [‘NYC', 'Madrid']
b. [‘London', 'Madrid']
c. [‘Miami', 'Madrid']
d. [‘Miami', ‘Paris']

Explanation: Slicing starts at the first index and stops before the second index. Here, the element at index 3 is “London” and the
element before index -1 is “Madrid.”
QUIZ
Which of the following data structures is preferred to contain a unique collection of
values?
4
a. dict
b. list
c. set
d. tuple
QUIZ
Which of the following data structures is preferred to contain a unique collection of
values?
4
a. dict
b. list
c. set
d. tuple

Explanation: A set is used when a unique collection of values is desired.
Key Takeaways
Download Python 2.7 version from Anaconda and install Jupyter notebook.
When you assign values to variables, you create references and not duplicates.
Integers, floats, strings, None, and Boolean are some of the data types supported by
Python.
Tuples, lists, dicts, and sets are some of the data structures of Python.
You can use indices to access individual or a range of elements in a data

structure.
The “in”, “+”, and “*” are some of the basic operators.
Functions are the primary and the most important methods of code
organization and reuse in Python.
The conditional “if”, “elif” statements, “while” and “for” loops,

and exception handling are some important control flow
statements.
This concludes “Python: Environment Setup and Essentials.”
The next lesson is “Mathematical Computing with Python (NumPy).”
Lesson 5—Mathematical Computing with Python (NumPy)
What You’ll Learn
What NumPy is and why it is important
Basics of NumPy, including its fundamental objects
Create and print a NumPy array
Carry out basic operations in NumPy
Use shape manipulation and copying methods
Execute linear algebraic functions
Build basic programs using NumPy

Quick Recap: Lists
A list is a collection of values. You can individually add, remove, or update these values. A single list can
contain multiple data types.
List
Collection of values
Multiple types (heterogeneous)

Add, remove, update
Limitations of Lists
Though you can change individual values in a list, you cannot apply a mathematical operation over the entire list.
Mathematical operation over the entire “distance” and “time” lists
Error
Why NumPy
Numerical Python (NumPy) supports multidimensional arrays over which you can easily apply mathematical
operations.
Import NumPy
Create “distance” and “time” NumPy arrays

Mathematical function applied over the entire “distance” and “time” arrays
Output
NumPy Overview
NumPy is the foundational package for mathematical computing in Python.
It has the following properties:
Supports fast and efficient

multidimensional arrays (ndarray)
Executes element-wise
computations and mathematical
calculations
Performs linear algebraic
operations, Fourier transforms,
and random number generation
Tools for reading/writing array

based datasets to disk
Efficient way of storing and

manipulating data
Tools for integrating language

codes (C, C++)
Properties of ndarray
An array in NumPy has the following properties:
Collection of values Multidimensional
Add, remove, change ndarray Mathematical function support
Single type (homogeneous) Fast and efficient

Purpose of ndarray
The ndarray in Python is used as the primary container to exchange data between algorithms.
Algorithm
[1, 2, 1]
Question/Problem
[[ 1, 0, 0],
[ 0, 1, 2]]
Sharing
Data
([[ 2, 8, 0, 6],
Algorithm [ 4, 5, 1, 1],
[ 8, 9, 3, 6]])
ndarray
Write Program
Algorithm
Knowledge Check—Sequence it Right!
The code here is buggy. You have to correct its sequence to debug it.
4
Knowledge Check—Sequence it Right!
The code here is buggy. You have to correct its sequence to debug it.
4
Types of Arrays
Arrays can be one-dimensional, two dimensional, three-dimensional, or multi-dimensional.
One-Dimensional Array Two-Dimensional Array Three-Dimensional Array
Printed as rows Printed as matrices (2x3) Printed as list of matrices (3x3x3)
array([[[ 0, 1, 2],
1 axis [ 3, 4, 5],
array([5, 7,9]) array([[ 0, 1, 2], 2 axes
rank 1 [ 6, 7, 8]],
[ 5, 6, 7]]) rank 2
Length = 3 [[ 9, 10, 11],
3 axes
[12, 13, 14],
rank 3
Length = 3 [15, 16, 17]],
[[18, 19, 20],

5 7 9 0 1 2 [21, 22, 23],
y axis (0,0) (0,1) (0,2) [24, 25, 26]]])
0 1 2 5 6 7
x axis (1,0) (1,1) (1,2) y axis
Length = 3
x axis
x axis
z axis
Demo 01—Creating and Printing an ndarray
Demonstrate how to create and print an ndarray.
Knowledge Check
KNOWLEDGE How many elements will the following code print?
CHECK
print(np.linspace(4,13,7))
a. 4
b. 7
c. 11
d. 13
KNOWLEDGE How many elements will the following code print?
CHECK
print(np.linspace(4,13,7))
a. 4
b. 7
c. 11
d. 13
Explanation: In the “linspace” function, “4” is the starting element and “13” is the end element. The last number “7” specifies that
a total of seven equally spaced elements should be created between “4” and “13,” both numbers inclusive. In this case, the
“linspace” function creates the following array: [ 4. 5.5 7. 8.5 10. 11.5 13. ]
Class and Attributes of ndarray—.ndim
Numpy’s array class is “ndarray,” also referred to as “numpy.ndarray.” The attributes of ndarray are:
This refers to the number of axes (dimensions) of the array. It is also called the rank of the
ndarray.ndim array.
ndarray.shape
ndarray.size
Two axes or 2D array Three axes or 3D array
ndarray.dtype
Concept Example
Class and Attributes of ndarray—.ndim
The array “np_city” is one-dimensional, while the array “np_city_with_state” is two-

ndarray.ndim dimensional.
ndarray.shape
ndarray.size
ndarray.dtype
Concept Example
Class and Attributes of ndarray—.shape
This consists of a tuple of integers showing the size of the array in each dimension. The
ndarray.ndim length of the “shape tuple” is the rank or ndim.
ndarray.shape
2 rows, 3 columns 2 rows, 3 columns, 2 ranks

ndarray.size
Shape: (2, 3) Shape: (2, 3, 2)
ndarray.dtype
Concept Example
Class and Attributes of ndarray—.shape
The shape tuple of both the arrays indicate their size along each dimension.
ndarray.ndim
ndarray.shape
ndarray.size
ndarray.dtype
Concept Example
Class and Attributes of ndarray—.size
It gives the total number of elements in the array. It is equal to the product of the elements
ndarray.ndim of the shape tuple.
ndarray.shape
Array contains 6 elements Array contains 12 elements

ndarray.size
Array a = (2, 3) Array b = (2, 3, 2)

Size = 6 Size = 12
ndarray.dtype
Concept Example
Class and Attributes of ndarray—.size
Look at the examples to see how the shape tuples of the arrays are used to calculate their
ndarray.ndim size.
ndarray.shape
ndarray.size
ndarray.dtype
Concept Example
Class and Attributes of ndarray—.dtype
It’s an object that describes the type of the elements in the array. It can be created or
ndarray.ndim specified using Python.
ndarray.shape
Array contains integers Array contains floats
ndarray.size Array a = [3, 7, 4] Array b = [1.3, 5.2, 6.7]

[2, 1, 0] [0.2, 8.1, 9.4]
[2.6, 4.2, 3.9]

[7.8, 3.4, 0.8]
ndarray.dtype
Concept Example
Class and Attributes of ndarray—.dtype
Both the arrays are of “string” data type (dtype) and the longest string is of length 7, which is
ndarray.ndim “Houston.”
ndarray.shape
ndarray.size
ndarray.dtype
Concept Example
Basic Operations
Using the following operands, you can easily apply various mathematical, logical, and comparison operations
on an array.
Mathematical Operations Logical Operations Comparison Operations
Addition + And & Greater >

Subtraction - Or | Greater or equal >=
Multiplication * Not ~ Less <
Division / Less or equal <=
Exponentiation ** Equal ==
Not equal !=
Demo 03—Executing Basic Operations
Demonstrate how to apply some basic operations on an array.
Basic Operations—Example
NumPy uses the indices of the elements in each array to carry out basic operations. In this case, where we are
looking at a dataset of four cyclists during two trials, vector addition of the arrays gives the required output.
First trial
Second trial
Total distance
Array (First trial) Array (Second trial) Array (Total distance)
10 15 17 26 + 12 11 21 24 = 22 26 38 50
0 1 2 3 0 1 2 3
Index Index
Vector addition
Accessing Array Elements: Indexing
You can access an entire row of an array by referencing its axis index.
1st set data 2nd set data
Create 2D array using cyclist trial data shown earlier
First trial data
Second trial data
2D array containing cyclists’ data
10 15 17 26 First trial (axis 0)
12 11 21 24 Second trial (axis 1)

Accessing Array Elements: Indexing (contd.)
You can refer the indices of the elements in an array to access them. You can also select a particular index of more
than one axis at a time.
First cyclist: first trial data
First cyclist: all trial data

(Use “:” to select all the rows of an array)
(0, 0) (0, 1) (0, 2) (0, 3) (0, 0) (0, 1) (0, 2) (0, 3)

Cyclist 1, first 10 15 17 26 Cyclist 1, all 10 15 17 26
trial data trials data
12 11 21 24 12 11 21 24
(1, 0) (1, 1) (1, 2) (1, 3) (1, 0) (1, 1) (1, 2) (1, 3)
Accessing Array Elements: Slicing
Use the slicing method to access a range of values within an array.
Shape of the array
10 15 17 26
2 rows
Shape of the array 12 11 21 24
4 columns
Slicing the array
Use ‘:’ to select

10 15 17 26
all rows 12 11 21 24
Slicing the array data [ : , 1 : 3] 0 1 2 3

where 1 is inclusive but 3 is not
Starting Ending
index (1) index (2)
Activity—Slice It!
Select any two elements from the array to see how the statement required to slice the range changes.
Rules of the Game

• Choose the first element of the range. Then, choose the element that ends the range.
• See how the values in the statement change according to your choices.
• Refresh to try again.
5 8 10 21
Select any two elements from the array.

Accessing Array Elements: Iteration
Use the iteration method to go through each data element present in the dataset.
Iterate with “for loop”

through entire dataset
Iterate with “for loop” through

the “two cyclist” datasets
Indexing with Boolean Arrays
Boolean arrays are useful when you need to select a dataset according to set criteria. Here, the original dataset
contains test scores of two students. You can use a Boolean array to choose only the scores that are above a given
value.
Test scores
Student 1 83 71 57 63 83 71 57 63
Student 2 54 68 81 45 54 68 81 45
Test score > 60
Test 1 Test 2 Test 3 Test 4
Setting the passing score
Shows data elements which fit the

criteria (Boolean array)
Send “passing score” as an argument to “test scores” object
Copy and Views
When working with arrays, data is copied into new arrays only in some cases. Following are the three possible scenarios:
In this method, a variable is directly assigned the value of another variable. No new copy is made.
Simple Assignments
Original dataset
View/Shallow Copy
Assigned dataset
Shows both objects are the same

Deep Copy
Copy and Views
When working with arrays, data is copied into new arrays only in some cases. There are three possible scenarios:
A view, also referred to as a shallow copy, creates a new array object.
Simple Assignments
Original dataset
View/Shallow Copy Change value in “view” object
Original dataset changed

Deep Copy
Copy and Views
When working with arrays, data is copied into new arrays only in some cases. There are three possible scenarios:
Copy is also called “deep copy” because it entirely copies the original dataset. Any change in the copy
will not affect the original dataset.
Simple Assignments
Shows “copy” and original
object are different
Shows “copy” object data is not

owned by the original dataset
View/Shallow Copy Change value in “copy”
“Copy” object changed
Original dataset retained

Deep Copy
Universal Functions (ufunc)
NumPy provides useful mathematical functions called Universal Functions. These functions operate element-wise on
an array, producing another array as output. Some of these functions are listed here:
sqrt function provides the square root sqrt cos function gives cosine values for all
of every element in the array. elements in the array.
cos
floor
floor function returns the largest exp exp function performs exponentiation
integer value of every element in the
on each element.
array.
ufunc—Examples
Let’s look at some common ufunc examples:
Numbers for which square root will be calculated
Square root values
Import pi*
Trigonometric functions
Return the floor of the input element wise
Exponential functions for complex

mathematical calculations
Shape Manipulation
You can use certain functions to manipulate the shape of an array to do the following:
Flatten
Split Resize
Shape
Manipulation
Stack Reshape
Shape Manipulation—Example
You can use certain functions to manipulate the shape of an array to do the following:
Flattens the dataset
Changes or reshapes the dataset to 3 rows and 4 columns
Resizes again to 2 rows and 6 columns
Splits the array into two
Stacks the arrays together

Broadcasting
NumPy uses broadcasting to carry out arithmetic operations between arrays of different shapes. In this method,
NumPy automatically broadcasts the smaller array over the larger array.
array_a
2 3 5 8
Element-wise
multiplication
0.3 0.3 0.3 0.3
array_b
array_a
2 3 5 8
Broadcasting
0.3
scalar_c
Broadcasting—Constraints
Though broadcasting can help carry out mathematical operations between different-shaped arrays, they are subject
to certain constraints as listed below:
• When NumPy operates on two arrays, it compares their

shapes element-wise. It finds these shapes compatible
only if:
• Their dimensions are the same or
• One of them has a dimension of size 1.
• If these conditions are not met, a "ValueError” is thrown,
indicating that the arrays have incompatible shapes.
Broadcasting—Example
Let’s look at an example to see how broadcasting works to calculate the number of working hours of a worker per
day in a certain week.
Week one earnings

Week two earnings
Element-wise operation
Total earning for 2 weeks
Calculate week one hours
Hourly wage
Number of working hours

per day in week one
Linear Algebra—Transpose
NumPy can carry out linear algebraic functions as well. The “transpose()” function can help you interchange rows as
columns, and vice-versa.
transpose()
Axis 1 Axis 1
83 71 57 63
54 68 81 45
Axis 0
Axis 0
Linear Algebra—Inverse and Trace Functions
Using NumPy, you can also find the inverse of an array and add its diagonal data elements.
np.linalg.inv()
Inverse of the given array
* Can be applied only on a square matrix
np.trace()
Sum of diagonal elements “10” and “31”

Assignment
© Copyright 2015, Simplilearn. All rights reserved.

Assignment 01
Problem Instructions
Evaluate the dataset containing the GDPs of different countries to:

• Find and print the name of the country with the highest GDP,
• Find and print the name of the country with the lowest GDP,
• Print out text and input values iteratively,
• Print out the entire list of the countries with their GDPs, and
• Print the highest GDP value, lowest GDP value, mean GDP value, standardized GDP value, and the
sum of all the GDPs.
Assignment 01
Instructions to perform the assignment:

• Download the GDP dataset from the “Resource” tab. You can copy the data provided to help you
with your assignment.
Common instructions:
• If you are new to Python, download the “Anaconda Installation Instructions” document from the
“Resources” tab to view the steps for installing Anaconda and the Jupyter notebook.
• Download the “Assignment 02” notebook and upload it on the Jupyter notebook to access it.
• Follow the cues provided to complete the assignment.
Assignment

Assignment 02
Evaluate the dataset of the Summer Olympics, London 2012 to:

• Find and print the name of the country that won maximum gold medals,
• Find and print the countries who won more than 20 gold medals,
• Print the medal tally,
• Print each country name with the corresponding number of gold medals, and
• Print each country name with the total number of medals won.

Assignment 02

• Download the “Olympic 2012 Medal Tally” dataset. Use the data provided to create relevant and
required variables.

Quiz
QUIZ
Which of the following arrays is valid?
1
a. [1, 0.3, 8, 6.4]
b. [“Lucy”, 16, “Susan”, 23, “Carrie”, 37]
c. [True, False, “False”, True]
d. [3.14j, 7.3j, 5.1j, 2j]

QUIZ
Which of the following arrays is valid?
1
a. [1, 0.3, 8, 6.4]
b. [“Lucy”, 16, “Susan”, 23, “Carrie”, 37]
c. [True, False, “False”, True]
d. [3.14j, 7.3j, 5.1j, 2j]

Explanation: A NumPy ndarray can hold only a single data type, which makes it homogenous. NumPy supports integers, floats,
Booleans, and even complex numbers. Of all the options provided, only the array containing complex numbers is homogenous.
All the other options contain more than one data type.
QUIZ
Which function is most useful to convert a multidimensional array into a one-
dimensional array?
2
a. ravel()
b. reshape()
c. resize() and reshape()
d. All of the above

QUIZ
Which function is most useful to convert a multidimensional array into a one-
dimensional array?
2
a. ravel()
b. reshape()
c. resize() and reshape()
d. All of the above
Explanation: The function ravel() is used to convert a multidimensional array into a one-dimensional array. Though reshape()
also functions in a similar way, it creates a new array instead of transforming the input array.
QUIZ
The np.trace() method gives the sum of _____.
3
a. the entire array
b. the diagonal elements from left to right
c. the diagonal elements from right to left
d. consecutive rows of an array

QUIZ
The np.trace() method gives the sum of _____.
3
a. the entire array
b. the diagonal elements from left to right
c. the diagonal elements from right to left
d. consecutive rows of an array
Explanation: The trace() function is used to find the sum of the diagonal elements in an array. It is carried out in an incremental
order of the indices. Therefore, it can only add diagonal values from left to right and not vice versa.
QUIZ
The function np.transpose() when applied on a one-dimensional array gives _____.
4
a. a reverse array
b. an unchanged original array
c. an inverse array
d. all elements with zeroes

QUIZ
The function np.transpose() when applied on a one dimensional array gives _____.
4
a. a reverse array
b. an unchanged original array
c. an inverse array
d. all elements with zeroes
Explanation: Transposing a one-dimensional array does not change it in any way. It returns an unchanged view of the original
array.
QUIZ
Which statement will slice the highlighted data? 11 14 21 32 53 64
5
a. [3 : 5]
b. [3 : 6]
c. [2 : 5]
d. [2 : 4]
QUIZ
Which statement will slice the highlighted data? 11 14 21 32 53 64
5
a. [3 : 5]
b. [3 : 6]
c. [2 : 5]
d. [2 : 4]
Explanation: Let’s assume that the index of the first element is m and the second element is n. Then, you need to use the
statement “[n : m + 1]” to slice the required dataset. In this case, the index of the element “21” is “2” and that of “53” is “4.” So, the
correct statement to use would be [2 : 5].
Key Takeaways
NumPy is a very powerful Python library for mathematical and scientific computing.
You can create and print NumPy arrays using different methods.
Arrays can be one-dimensional, two-dimensional, three-dimensional, or

multi-dimensional.
NumPy uses basic operations, data access techniques, and copy and view
techniques for data wrangling.
NumPy can also manipulate data using various array shape

manipulation techniques.
NumPy can perform linear algebra functions to fix problematic

datasets and execute mathematical operations.
This concludes “Mathematical Computing with Python (NumPy).”
The next lesson is “Scientific Computing with Python (SciPy).”
Lesson 06—Python: Scientific computing with Python
(SciPy)
What You’ll Learn
Why SciPy is needed
The characteristics of SciPy
The sub-packages of SciPy
SciPy Sub-packages such as Optimization,

Integration, Linear Algebra, Statistics, Weave,
and IO
The Real World: Multiple Scientific Domains
How to handle multiple scientific domains? The solution is SciPy.
Statistics
Space science Optimization
Image science Signal processing
Platform integration
Mathematical equations
Scientific Domains
SciPy: The Solution
SciPy has built-in packages that help in handling the scientific domains.
Mathematics
Integration Statistics
(Normal distribution)
Linear algebra
Multidimensional image
processing
Mathematics Language Integration
constants
SciPy and its Characteristics
Characteristics of SciPy are as follows:
Built-in mathematical libraries 1 High-level commands for data

and functions 2 manipulation and visualization
Simplifies scientific
application development 6
Efficient and fast data
3 processing
Large collection of sub-packages

for different scientific domains 5 Integrates well with multiple
4 systems and environments
SciPy Sub-package
SciPy has multiple sub-packages which handle different scientific domains.
cluster ndimage
Clustering algorithms N-dimensional image processing
constants odr
Physical and mathematical constant Orthogonal distance regression
fftpack optimize
Fast Fourier Transform routines Optimization and root-finding routines
integrate signal
Integration and ordinary differential equation solvers Signal processing
Spatial sparse
Spatial data structures and algorithms Sparse matrices and associated routines
interpolate weave
Interpolation and smoothing splines C/C++ integration
IO stats
Input and Output Statistical distributions and functions
special
linalg
Special functions
Linear algebra
SciPy Sub-package: Integration
SciPy provides integration techniques that solve mathematical sequences and series,
or perform function approximation.
General integration (quad) General multiple integration (dblquad, tplquad, nquad)
•integrate.quad(f, a, b)
•integrate.dblquad()
•integrate.tplquad()
•integrate.nquad()
The limits of all inner integrals need to be defined

as functions.
This example shows how to perform quad integration.
Import quad from

integrate sub-
package
Define function for
integration of x
Perform quad integration
for function of x for limit 0
to 1
Define function for ax + b
Declare value of a and b
Perform quad
integration and pass
functions and arguments
This example shows you how to perform multiple integration.
Import integrate package

sub-package
Define function for x + y
Perform multiple
integration using the
lambda built-in function
SciPy Sub-package: Optimization
Optimization is a process to improve performance of a system mathematically by fine-

tuning the process parameters.
SciPy provides several optimization algorithms such as bfgs, Nelder-Mead simplex,
Newton Conjugate Gradient, COBYLA, or SLSQP.
Root finding, Curve fitting

Minimization functions
optimize.minimize(f, x0, method=‘BFGS’)
lower limit in a
given range
root(f, x0, method=’hybr’)

optimize.curve_fit(f, xdata, ydata)
SciPy Sub Package: Optimization
Import numpy and
optimize from scipy
Define function for

X^2 + 5 sin x
Perform optimize
minimize function
using bfgs method
and options
Perform optimize minimize

function using bfgs method
and without options
SciPy Sub-package: Optimization
Define function for

X + 3.5 Cos x
Pass x value in argument for

root
Function value and

array values
Knowledge Check
KNOWLEDGE What are the specification limits provided for curve fitting function (optimize.curve.fit),
CHECK during the optimization process?
a. Upper limit value
b. Lower limit value
c. Upper and lower limit values
d. Only the optimization method

KNOWLEDGE What are the specification limits provided for curve fitting function (optimize.curve.fit), during the
CHECK
optimization process?
a. Upper limit value
b. Lower limit value
c. • Upper and lower limit values
d. Only the optimization method
Explanation: Both the upper and lower limit values should be specified for optimize.curve.fit function.
SciPy Sub-package: Linear Algebra
SciPy provides rapid linear algebra capabilities and contains advanced algebraic
functions.
Click each tab to know more.
Inverse of matrix Determinant Linear systems Single value
decomposition (svd)
This function is used to compute the inverse of the given matrix. Let’s take a look at the inverse matrix
operation.
Import linalg and

Define a numpy
matrix or array
View the type
Use inv function to

inverse the matrix
SciPy provides very rapid linear algebra capabilities and contains advanced algebraic functions.

decomposition (svd)
With this function you can compute the value of the determinant for the given matrix.
Import linalg and

Define an numpy matrix or
array
Use det function to find the

determinant value of the
matrix

decomposition (svd)
Linear equations Import linalg
2x + 3 y + z = 21
-x + 5y + 4z = 9
3x + 2y + 9z = 6
Use solve
method

decomposition (svd)
Import linalg
Define matrix
Find shape of ndarray which

is 2X3 matrix
Use svd function

U (Unitary matrix)
Sigma or square root of eigenvalues
VH is values collected into

unitary matrix
Demo—Calculate Eigenvalues
Demonstrate how to calculate eigenvalues.
Knowledge Check
KNOWLEDGE
CHECK Which of the following function is used for inversing the matrix?
a. SciPy.special
b. SciPy.linalg
c. SciPy.signal
d. SciPy.stats
KNOWLEDGE
CHECK
Which of the following function is used for inversing the matrix?
a. SciPy.special
b. SciPy.linalg
c. • SciPy.signal
d. SciPy.stats
The correct answer is . b
Explanation: SciPy.linalg is used to inverse the matrix.

SciPy Sub-package: Statistics
SciPy provides a very rich set of statistical functions which are as follows:
• This package contains distributions for which random variables are

generated.
• These packages enable the addition of new routines and
distributions. It also offers convenience methods such as pdf(), cdf()
• Following are the statistical functions for a set of data:
o linear regression: linregress()
o describing data: describe(), normaltest()
Cumulative Distribution Function provides the cumulative probability associated with a

function.
One standard
Age Range Frequency
Cumulative deviation
Frequency
0-10 19 19
68% of data
10-20 55 74
Total number of 95% of data
21-30 23 97 persons within
this age
31-40 36 133 99.7% of data
41-50 10 143
-3 -2 -1 01 1 2 3
51-60 17 160
F(x) = P(X≤x)
negative infinity
Probability Density Function, or PDF, of a continuous random variable is the derivative of its
Cumulative Distribution Function, or CDF.
Derivative of CDF
Functions of Random Variables – Continuous (Normal Distribution):
Import norm for normal distribution
rvs for Random variables
cdf for Cumulative Distribution Function
pdf for Probability Density

Function for random
distribution
loc and scale are used to adjust the location and scale of the data distribution.
SciPy Sub-package: Weave
The weave package provides ways to modify and extend any supported extension libraries.
• Includes C/C++ code within Python code

• Speed ups of 1.5x to 30x compared to algorithms written in pure Python
Two main functions of weave::

• inline() compiles and executes C/C++ code on the fly
• blitz() compiles NumPy Python expressions for fast execution
SciPy Sub-package: IO
The IO package provides a set of functions to deal with several kinds of file formats.
It offers a set of functions to deal with file formats that

includes:
• MatLab file
• IDL files
• Matrix Market files
• Wav sound files
• Arff files, and
• Netcdf files
Package provides additional files and it’s corresponding
methods.
Quiz
Assignment 01
Use SciPy to solve a linear algebra problem.

There is a test with 30 questions worth 150 marks. The test has two types of questions:
1. True or false – carries 4 marks each
2. Multiple choice – carries 9 marks each
Find the number of true or false and multiple choice questions.
Assignment 01
• If you are new to Python, download the “Anaconda Installation Instructions”
document from the “Resources” tab to view the steps for installing Anaconda and the
Jupyter notebook.
• Download the “Assignment 01” notebook and upload it on the Jupyter notebook to
access it.
Quiz
Assignment 02
Use SciPy to declare 20 random values for random values and perform the following:
1. CDF – Cumulative Distribution Function for 10 random variables.
2. PDF – Probability Density Function for 14 random variables.
Assignment 02
• If you are new to Python, download the “Anaconda Installation Instructions”
document from the “Resources” tab to view the steps for installing Anaconda and the
Jupyter notebook.
• Download the “Assignment 02” notebook and upload it on the Jupyter notebook to
access it.
Quiz
QUIZ
Which of the following is performed using SciPy?
1
a. Website
b. Plot data
c. Scientific calculations
d. System administration
QUIZ
Which of the following is performed using SciPy?
1
a. Website
b. Plot data
c. Scientific calculations
d. System administration
The correct answer is. c.

Explanation: SciPy has been specially made to perform scientific calculations. Generally,
Python is the programming language that has libraries to perform all listed activities.
QUIZ
Which of the following functions is used to calculate minima?
2
a. optimize.minimize()
b. integrate.quad()
c. stats.linregress()
d. linalg.solve()
QUIZ
Which of the following functions is used to calculate minima?
2
a. optimize.minimize()
b. integrate.quad()
c. stats.linregress()
d. linalg.solve()
The correct answer is. a.

Explanation: The function optimize.minimize() is used to calculate minima. integrate.quad () is used
for integral calculation, stats.linregress() is used for linear regression, and linalg.solve() is used to
solve a linear system.
QUIZ Which of the following syntaxes is used to generate 100 random variables from a
3 t-distribution with df = 10?
a. stats.t.pmf(df=10, size=100)
b. stats.t.pdf(df=10, size=100)
c. stats.t.rvs(df=10, size=100)
d. stats.t.rand(df=10, size=100)
QUIZ
Which of the following syntaxes is used to generate 100 random variables from a
3
t-distribution with df = 10?
a. stats.t.pmf(df=10, size=100)
b. stats.t.pdf(df=10, size=100)
c. stats.t.rvs(df=10, size=100)
d. stats.t.rand(df=10, size=100)
The correct answer is. c.

Explanation: The stats.t.rvs() function is used to generate random variables. stats.t.pmf()
function is used to generate the probability of mass function, and stats.t.pdf() is used to generate
probability density function. Note that stats.t.rand () does not exist.
QUIZ
Which of the following functions is used to run C or C++ codes in SciPy?
4
a. io.loadmat()
b. weave.inline()
c. weave.blitz()
d. io.whosmat()
QUIZ
Which of the following functions is used to run C or C++ codes in SciPy?
4
a. io.loadmat()
b. weave.inline()
c. weave.blitz()
d. io.whosmat()
The correct answer is. b.

Explanation: inline() function accepts C codes as string and compiles them for later use.
loadmat() loads variables from .mat file. whosmat() checks the variables inside a .mat file.blitz()
and then compiles NumPy expressions for faster running, but it can’t accept C codes.
Key Takeaways
SciPy has multiple sub-packages, which proves useful for different scientific
computing domains.
Integration can be used to solve mathematical sequences and series or
perform function approximation.
Optimization is the process to improve performance of a system

mathematically by fine-tuning the process parameters.
The SciPy linear algebraic functions include computing the

inverse of a matrix, calculating the determinant, solving linear
systems, and computing single value decomposition.
Statistical functions provide many useful sub-packages that

enable the building of a hypothesis, determining the
probability, and predicting the outcome.
The IO package offers a set of functions to deal with

several types of file formats.
This concludes “Scientific computing with Python”
The next lesson is “Data Manipulation with Pandas”
Lesson 7— Data Manipulation with Pandas

What You Will Learn
Pandas and its features
Different data structures of Pandas
Creating Series and DataFrame with data inputs
Viewing, selecting, and accessing elements in a data structure
Handling vectorized operations
Learning how to handle missing values
Analyzing data with different data operation methods

Why Pandas
NumPy is great for mathematical computing. Then why do we need Pandas?
Pandas with several

functionalities
NumPy
Why Pandas
NumPy is great for mathematical computing. Then why do we need Pandas?
Intrinsic data
alignment
Data Structures
Data operation
handling major
functions
use cases
Pandas
Data standardization Functions for handling

functions ? missing data
.
Pandas Features
The various features of Pandas makes it an efficient library for Data Scientists.
Powerful data
structure
Fast and efficient

High performance
data wrangling
merging and joining
of data sets
Pandas
Intelligent and Easy data aggregation
automated data and transformation
alignment
Tools for reading/

writing data
Data Structures
The four main libraries of Pandas data structure are:
• One-dimensional labeled array

Series
• Supports multiple data types
• Two-dimensional labeled array
Data Frame
• Input can be a Series
• Three-dimensional labeled array • Input can be another DataFrame
• Items  axis 0 Panel
• Major axis  rows • Four-dimensional labeled array
• Minor axis columns • Supports multiple data types
Panel 4D • Labels  axis 0
(Experimental) • Items  axis 1
• Major axis  rows
• Minor axis columns
Understanding Series
Series is a one-dimensional array-like object containing data and labels (or index).
Data 4 11 21 36
0 1 2 3
Label(index)
Data alignment is intrinsic and will not be broken until changed explicitly by program.
Series
Series can be created with different data inputs:
Data Input
• Integer
• ndarray 2 3 8 4
• String
• dict
• Python Object 0 1 2 3
• scalar
• Floating Point
• list Label(index)
Data Types
Series
How to Create Series
Key points to note while creating a series are as follows:

• Import Pandas as it is the main library
• Apply the syntax and pass the data elements as arguments
• Import NumPy while working with ndarrays
Basic Method
4 11 21 36
S = pd.Series(data, index = [index])
Series
Create Series from List
This example shows you how to create a series from a list:
Import libraries
Pass list as an argument
Data value
Index
Data type
We have not created index for data but notice that data alignment is done automatically
Create Series from ndarray
This example shows you how to create a series from an ndarray:
ndarray for countries
Pass ndarray as an argument
countries
Index
Data type
Create Series from dict
A series can also be created with dict data input for faster operations.
dict for countries and their gdp
Countries have been passed as an index

and GDP as the actual data value
GDP
Country
Data type
Create Series from Scalar
Scalar input
Index
Data
index
Data type
Accessing Elements in Series
Data can be accessed through different functions like loc, iloc by passing data element position or
index range.
Vectorized Operations in Series
Vectorized operations are performed by the data element’s position.
Add the series

Vectorized Operations in Series
Knowledge Check
KNOWLEDGE How is an index for data elements assigned while creating a Pandas series ? Select all
CHECK that apply.
a. Created automatically
b. Needs to be assigned
c. Once created can not be changed or altered
d. Index is not applicable as series is one-dimensional

How is an index for data elements assigned while creating a Pandas series ? Select all
KNOWLEDGE that apply.
CHECK
a.
Created automatically
b. Needs to be assigned
c. Once created can not be changed or altered
d.
Index is not applicable as series is one-dimensional
The correct answer is a, b .
Explanation: Data alignment is intrinsic in Pandas data structure and happens automatically. One can also assign index to data
elements.
KNOWLEDGE
CHECK
What will the result be in vector addition if label is not found in a series?
a. Marked as Zeros for missing labels
b. Labels will be skipped
c. Marked as NaN for missing labels
d. Will throw an exception, index not found

KNOWLEDGE
CHECK
What will the result be in vector addition if label is not found in a series?
a.
Marked as Zeros for missing labels
b. Labels will be skipped
c. Marked as NaN for missing labels
d.
Will throw an exception, index not found
Explanation: The result will be marked as NaN (Not a Number) for missing labels.
DataFrame
DataFrame is a two-dimensional labeled data structure with columns of potentially different types.
Data Input
• Integer
• ndarray 2 3 8 4
• String
• dict 5 8 10 1
• Python Object
• scalar
• Floating Point 0 1 2 3
• list
Label(index)
Data Types
DataFrame
Create DataFrame from Lists
Let’s see how you can create a DataFrame from lists:
Pass the list to the DataFrame

Create DataFrame from dict
This example shows you how to create a DataFrame from a series of dicts:
dict one dict two
Entire dict
View DataFrame
You can view a DataFrame by referring the column name or with the describe function.
Create DataFrame from dict of Series
Create DataFrame from ndarray
Create an ndarrays with years

Create a dict with the ndarray
Pass this dict to a new DataFrame

Create DataFrame from DataFrame
Create a DataFrame from a

DataFrame
Demo 01—View and Select Data
Demonstrate how to view and select data in a DataFrame.
Missing Values
Various factors may lead to missing data values:
Data not provided by the

source Software issue Data integration issue Network issue
Handle Missing Values
It’s difficult to operate on a dataset when it has missing values or uncommon indices.
Handle Missing Values with Functions
The dropna function drops all the values with uncommon indices.
Handle Missing Values with Functions
The fillna function fills all the uncommon indices with a number instead of dropping them.
Fill the missing values with zero

Handle Missing Values with Functions- Example
Data Operation
Data operation can be performed through various built-in methods for faster data processing.
Data Operation with Functions
While performing data operation, custom functions can be applied with the applymap method.
Declare a custom function
Test the function
Apply the function to the DataFrame

Data Operation with Statistical Functions
This example shows data operations with different statistical functions.
Create a DataFrame with two test
Apply the max function to find the

maximum score
Apply the mean function to find

the average score
Apply the std function to find the standard

deviation for both the tests
Data Operation Using Groupby
This example shows how to operate data using the groupby function.
Create a DataFrame with first and

last name as former presidents
Group the DataFrame with the first name
Group the DataFrame with the first name

Data Operation – Sorting
This example shows how to sort data
Sort values by first name

Demo 02—Data Operations
Demonstrate how to perform data operations.
Data Standardization
This example shows how to standardize a dataset.
Create a function to return the standardize value
Apply the function to the entire dataset
Standardized test data is applied for the entire

DataFrame
Knowledge Check
KNOWLEDGE
CHECK
What is the result of DataFrame[3:9]?
a. Series with sliced index from 3 to 9
b. dict of index position 3 and index position 9
c. DataFrame of sliced rows index from 3 to 9
d. DataFrame with data elements at index 3 to index9

KNOWLEDGE
CHECK
What is the result of DataFrame[3:9]?
a. Series with sliced index from 3 to 9
b. dict of index position 3 and index position 9
c. DataFrame of sliced rows index from 3 to 9
d. DataFrame with data elements at index 3 to index9
Explanation: This is DataFrame slicing technique with indexing or selection on data elements. When a user
passes the range 3:9, the entire range from 3 to 9 gets sliced and displayed as output.
KNOWLEDGE
CHECK
What does the fillna() method do?
a. Fills all NaN values with zeros
b. Fills all NaN values with one
c. Fills all NaN values with values mentioned in the parenthesis
d. Drops NaN values from the dataset

KNOWLEDGE
CHECK
What does the fillna() method do?
a. Fills all NaN values with zeros
b. Fills all NaN values with One
c. Fills all NaN values with values mentioned in the parenthesis
d. Drops NaN values from the dataset
Explanation: fillna is one of the basic methods to fill NaN values in a dataset with a desired value by passing
that in parenthesis.
File Read and Write Support
read_hdf
read_excel to_hdf read_clipboard
to_excel to_clipboard
read_csv read_html
to csv to_html
read_json read_pickle
to_json to_pickle
read_sql read_stata
read_sas
to_sql to_stata
to sas
Pandas SQL operation
Activity—Sequence it Right!
The code here is buggy. You have to correct its sequence to debug it. To do that, click any two code
snippets, which you feel are out of place, to swap their places.
Click any two code snippets to swap them.

Assignment

Assignment 01
Analyze the Federal Aviation Authority (FAA) dataset using Pandas to do the following:
1. View
a. Aircraft make name
b. State name
c. Aircraft model name
d. Text information
e. Flight phase
f. Event description type
g. Fatal flag
2. Clean the dataset and replace the fatal flag NaN with “No”
3. Find the aircraft types and their occurrences in the dataset
4. Remove all the observations where aircraft names are not available
5. Display the observations where fatal flag is “Yes”
Assignment 01

• Download the FAA dataset from the “Resource” tab. Upload the dataset to your Jupyter notebook to
view and evaluate it.
Assignment
Assignment 02
A dataset in CSV format is given for the Fire Department of New York City. Analyze the dataset to
determine:
1. The total number of fire department facilities in New York city
2. The number of fire department facilities in each borough
3. The facility names in Manhattan
Assignment 02

• Download the FDNY dataset from the “Resource” tab. You can upload the dataset to your Jupyter
notebook to use it.
Quiz
QUIZ
Which of the following data structures is used to store three-dimensional data?
1
a. Series
b. DataFrame
c. Panel
d. PanelND
QUIZ
Which of the following data structures is used to store three-dimensional data?
1
a. Series
b. DataFrame
c. Panel
d. PanelND
Explanation: Panel is a data structure used to store three-dimensional data.

QUIZ
Which method is used for label-location indexing by label?
2
a. iat
b. iloc
c. loc
d. std
QUIZ
Which method is used for label-location indexing by label?
2
a. iat
b. iloc
c. loc
d. std
Explanation: The loc method is used to for label-location indexing by label; iat is strictly integer location and
iloc is integer-location-based indexing by position.
QUIZ
While viewing a dataframe, head() method will _____.
3
a. return only the first row
b. return only headers or column names of the DataFrame
c. return the first five rows of the DataFrame
d. throw an exception as it expects parameter(number) in parenthesis

QUIZ
While viewing a dataframe, head() method will _____.
3
a. return only the first row
b. return only headers or column name of the DataFrame
c. return the first five rows of the DataFrame
d. throw an exception as it expects parameter(number) in parenthesis
Explanation: The default value is 5 if nothing is passed in head method. So it will return the first five rows
of the DataFrame.
Key Takeaways
Let us take a quick recap of what we have learned in the lesson:
Pandas is an open source library for data analysis and is an efficient

data wrangling tool in Python.
The four main libraries of Pandas are Series, DataFrame, Panel, and
Panel 4D.
DataFrame is a two-dimensional labeled data structure with columns
of potentially different data types.
To access data elements in a series, 'loc' and 'iloc' methods can be
used.
Key Takeaways
The 'iat' method enables selection of elements in a DataFrame by

index position and returns the corresponding data element.
Missing data values in Pandas can be resolved through
two built-in methods such as dropna and fillna.
Pandas supports multiple files for data analysis such as

Excel, PyTables, Clipboard, HTML, pickle, dta, SAS, SQL,
JSON, and CSV.
This concludes “Data Manipulation with Pandas.”
The next lesson is “Machine Learning with SciKit Learn.”
Lesson 8—Machine Learning with Scikit–Learn
What You’ll Learn
What machine learning is and why it is important
The machine learning approach
Relevant terminologies that help you understand a dataset
Features of supervised and unsupervised learning models
Algorithms such as regression, classification,

clustering, and dimensionality reduction
Why Machine Learning
If we stored the data generated in a day on Blu-ray disks and stacked them up, it would be equal to
the height of four Eiffel towers! Machine learning helps analyze this data easily and quickly.
Machine learning
Purpose of Machine Learning
Machine learning is a great tool to analyze data, find hidden data patterns and relationships, and
extract information to enable information-driven decisions and provide insights.
Identify patterns and

relationships
Data
Insights into
unknown data
Information-driven
decisions
Machine Learning Terminology
These are some machine learning terminologies that you will come across in this lesson:
Inputs Attributes
Features
Label Records
Response Observations
Outcome Examples
Target Samples
Machine Learning Approach
The machine learning approach starts with either a problem that you need to solve or a given
dataset that you need to analyze.
Strive for
accuracy
Train and test
the model
Choose the right
model
Identify the
problem type
Extract the
features from
Understand the the dataset
problem/dataset
Steps 1 and 2: Understand the Dataset and Extract its Features
Let us look at a dataset and understand its features in terms of machine learning.
Features Response
(attributes) (label)
Education Professional Training Hourly Rate
(Yrs.) (Yes/No) (USD)
16 1 90
15 0 65
12 1 70
Observations 18 1 130
(records)
16 0 110
16 1 100
15 1 105
31 0 70
Predictors
Steps 3 and 4: Identify the Problem Type and Learning Model
Machine learning can either be supervised or unsupervised. The problem type should be selected
based on the type of learning model.
Concept Problem Types Example
Supervised Learning Unsupervised Learning
• In supervised learning, the dataset used to train a model • In unsupervised learning, the response or the outcome of
should have observations, features, and responses. The the data is not known.
model is trained to predict the “right” response for a given
set of data points. • Supervised learning models are used to identify and
visualize patterns in data by grouping similar types of data.
• Supervised learning models are used to predict an
outcome. • The goal of this model is to “represent” data in a way that
meaningful information can be extracted.
• The goal of this model is to “generalize” a dataset so that
the “general rule” can be applied to new data as well.
Steps 3 and 4: Identify the Problem Type and Learning Model (contd.)
Data can either be continuous or categorical. Based on whether it is supervised or unsupervised
learning, the problem type will differ.
Data Data
Data Type Data Type
Continuous Categorical Continuous Categorical
Problem Dimensionality Problem

Regression Classification Clustering
Type reduction Type
Steps 3 and 4: Identify the Problem Type and Learning Model (contd.)
Some examples of supervised and unsupervised learning models are shown here.
Categories of news based on the topics Grouping of similar stories on different news networks
How it Works—Supervised Learning Model
In supervised learning, a known dataset with observations, features, and response is used to create and train
a machine learning algorithm. A predictive model, built on top of this algorithm, is then used to predict the
response for a new dataset that has the same features.
New or
Known Data
Unseen Data
Observations/ Observations/
Records Records
Features/
Attributes
Predictive Features/
Model Attributes
Response/
Label Machine
Learning
Algorithm Predicted
Response/Label
How it Works—Unsupervised Learning Model
In unsupervised learning, a known dataset has a set of observations with features. But the response is not
known. The predictive model uses these features to identify how to classify and represent the data points of
new or unseen data.
New or
Known Data
Unseen Data
Observations/ Observations/
Records Records
Machine Predictive
Features/ Features/
Learning Model
Attributes Attributes
Algorithm
Data
Representation
Steps 5 and 6: Train, Test, and Optimize the Model
To train supervised learning models, data analysts usually divide a known dataset into training and
testing sets.
Known Data
Train
(60%-80%)
Test Observations/
(20%-40%) Records
Features/
Attributes
Response/
Label Machine
Learning
Algorithm
Steps 5 and 6: Train, Test, and Optimize the Model (contd.)
Let us look at an example to see how the split approach works.
Model Training
Observation Response
ID Education Professional Hourly rate

training
10 16 1 90
Train set Train set
45 15 0 65
Test set Test set
83 12 1 70
45 18 1 130
54 16 0 110
67 16 1 100
71 15 1 105
31 15 0 70
Supervised Learning Model Considerations
Some considerations of supervised and unsupervised learning models are shown here.
Performance
Response optimization
Model Accuracy
Features
Generalization
Knowledge Check
KNOWLEDGE
CHECK In machine learning, which one of the following is an observation?
a. Features
b. Attributes
c. Records
d. Labels
KNOWLEDGE
CHECK In machine learning, which one of the following is an observation?
a. Features
b. Attributes
c. Records
d. Labels
Explanation: An observation is a set of examples, records, or samples.

KNOWLEDGE If data is continuous and has labels (response), then it fits which of the following
CHECK problem types?
a. Supervised learning: classification
b. Unsupervised learning: clustering
c. Unsupervised learning: dimensionality reduction
d. Supervised learning: regression

KNOWLEDGE If data is continuous and has labels (response), then it fits which of the following
CHECK problem types?
a. Supervised learning: classification
b. Unsupervised learning: clustering
c. Unsupervised learning: dimensionality reduction
d. Supervised learning: regression
Explanation: The regression algorithm belonging to the supervised learning model is best suited to analyze continuous data.
KNOWLEDGE
CHECK Identify the goal of unsupervised learning. Select all that apply.
a. To predict the outcome
b. To understand the structure of the data
c. To generalize the dataset
d. To represent the data

KNOWLEDGE
CHECK Identify the goal of unsupervised learning. Select all that apply.
a. To predict the outcome
b. To understand the structure of the data
c. To generalize the dataset
d. To represent the data
The correct answer is b, d.
Explanation: The goal of unsupervised learning is to understand the structure of the data and represent it. There is no right or
certain answer in unsupervised learning.
Scikit-Learn
Scikit is a powerful and modern machine learning Python library for fully and semi-automated data
analysis and information extraction.
Efficient tools to identify Free and open Rich set of libraries

and organize problems datasets for learning and
(Supervised/ Unsupervised) predicting
Model support for Model Open source

every problem type persistence community and
vendor support
Scikit-Learn—Problem-Solution Approach
Scikit-learn helps Data Scientists organize their work through its problem-solution approach.
Model Estimator Model Model

Predictions Accuracy
Selection Object Training Tuning
Scikit-Learn—Problem-Solution Considerations
While working with a Scikit-Learn dataset or loading your own data to Scikit -Learn, always consider these
four points:
 Create separate objects for feature and response.
 Ensure that features and response have only numeric values.
 Features and response should be in the form of a NumPy ndarray.
 Since features and response would be in the form of arrays, they would have shapes and
sizes.
 Features are always mapped as x, and response is mapped as y.

Knowledge Check
KNOWLEDGE
CHECK The estimator instance in Scikit-learn is a _____.
a. model
b. feature
c. dataset
d. response
KNOWLEDGE
CHECK The estimator instance in Scikit-learn is a _____.
a. model
b. feature
c. dataset
d. response
Explanation: The estimator instance or object is a model.

Supervised Learning Models: Linear Regression
Linear regression is a supervised learning model used to analyze continuous data.
It is easy to use as the model does not

require a lot of tuning.
It is the most basic and widely used

technique to predict a value of an attribute.
It runs very fast, which makes it more time-

efficient.
Supervised Learning Models: Linear Regression (contd.)
The linear regression equation is based on the formula for a simple linear equation.
𝑦 = m𝑥 + c Simple linear equation
𝑦 = β0 + β1 𝑥 Linear regression equation
Response Input features
Intercept Coefficient of x
Linear regression is the most basic technique to predict a value of an attribute.
Data point
Residual
y Residual
(response) Least square line
dy 𝑦 = β0 + β1 𝑥 + u
dx
d𝑦
(0, y) β1 = Slope/ gradient
d𝑥 Actual Predicted
β0 value value
x (predictor variable)
! The attributes are usually fitted using the “least square” approach.
Smaller the value of SSR or SSE, the more accurate the prediction will be, which would make the
model the best fit.
Data point
𝑦 = β0 + β1 𝑥 + u
y SSE
(response) Least square line
SSR SSR = (𝑦𝑖 − 𝑦)2
𝑦
Regression of sum of squares
𝑦 = β0 + β1 𝑥
(0, y)
β0
SSE = (𝑦𝑖 − 𝑦𝑖 )2
x (predictor variable)
Error of sum of squares
! The attributes are usually fitted using the “least square” approach.
Let us see how linear regression works in Scikit-Learn.
Normalizes the regression

variable before performing
the regression operation
Calculates the intercept

Class for this model
sklearn.linear_model.LinearRegression(fit_intercept=True, normalize=False, copy_X=True, n_jobs=1)
Copies the Number of jobs to use

regression variable for the computation
Demo 01—Loading a Dataset
Demonstrate how to load a built-in scikit-learn dataset
Demo 02—Linear Regression Model
Demonstrate how to create and train a linear regression model
Supervised Learning Models: Logistic Regression
Logistic regression is a generalization of the linear regression model used for classification problems.
𝑒 β0+β1𝑥
𝜋 = Pr 𝑦 = 1 𝑥 =
1 + 𝑒 β0+β1𝑥
Probability of y = 1, given x
Change in the log-odds
for a unit change in x
The purpose of K-NN is to predict the class for each observation.

Supervised Learning Models: Logistic Regression (contd.)
𝜋
Odds =
1−𝜋
Probability
𝜋 β +β1 𝑥
log = log 𝑒 0 = β0 + β1 𝑥
1−𝜋
Logarithm of odds Linear regression
Supervised Learning Models: Logistic Regression (contd.)
Inverse of
Specifies the norm used regularization
in penalization Calculates
the intercept
Implemented only
Class for L2 penalty
class sklearn.linear_model.LogisticRegression(penalty='l2', dual=False, tol=0.0001, C=1.0, fit_intercept=True, intercept_scaling=1,

class_weight=None, random_state=None, solver='liblinear', max_iter=100, multi_class='ovr', verbose=0, warm_start=False, n_jobs=1)
Seed or the random Algorithm to use in the

If true, reuse the
state instance optimization problem
solution of the Number of
previous call jobs in parallel
Can be ovr (binary)
computation
or multinomial
Supervised Learning Models: K Nearest Neighbors (K-NN)
K-nearest neighbors, or K-NN, is one of the simplest machine learning algorithms used for both
classification and regression problem types.
K=6
K=3
If you are using this method for binary classification, choose an odd number for k to avoid the case
of a "tied" distance between two classes.
Demo 03—K-NN and Logistic Regression Models
Demonstrate the use of K-NN and logistic regression models
Unsupervised Learning Models: Clustering
A cluster is a group of similar data points.
It is used:
• To extract the structure of the data
• To identify groups in the data
Greater similarity between data points results in better clustering.

Unsupervised Learning Models: K-means Clustering
K-means finds the best centroids by alternatively assigning random centroids to a dataset and selecting
mean data points from the resulting clusters to form new centroids. It continues this process iteratively
until the model is optimized.
Centroids (mean)
Assign Optimize
Find the number of clusters and assign Iterate and optimize the mean for each cluster for
mean its respective data points
Unsupervised Learning Models: K-means Clustering (contd.)
Let us see how the k-means algorithm works in Scikit-Learn.
Number of clusters to
form and number of
Number of times the K-
centroids to generate
means algorithm will be run Pre-compute
with different centroid for faster
Class
Selects initial cluster seeds operation
centers
sklearn.cluster.KMeans(n_clusters=8, init='k-means++', n_init=10, max_iter=300, tol=0.0001,
precompute_distances='auto',
verbose=0, random_state=None, copy_x=True, n_jobs=1)
Maximum number of
Initialize the iterations of the K-means
centers Number of jobs in algorithm for a single run
parallel
If true, does not computation
modify data
while pre-
computing
Demo 04—K-means Clustering
Demonstrate how to use k-means clustering to classify data points
Unsupervised Learning Models: Dimensionality Reduction
It reduces a high-dimensional dataset into a dataset with fewer dimensions. This makes it easier and
faster for the algorithm to analyze the data.
Unsupervised Learning Models: Dimensionality Reduction (contd.)
These are some techniques used for dimensionality reduction:
Drop data columns with missing values
Drop data columns with low variance
Drop data columns with high

correlations
Large dataset
(a few thousand columns and
rows)
Apply statistical functions - PCA
Unsupervised Learning Models: Principal Component Analysis (PCA)
It is a linear dimensionality reduction method which uses singular value decomposition of the data and
keeps only the most significant singular vectors to project the data to a lower dimensional space.
• It is primarily used to compress or reduce the

data.
• PCA tries to capture the variance, which helps
it pick up interesting features. Minor axis
Principal axis
• PCA is used to reduce dimensionality in the
dataset and to build our feature vector.
• Here, the principal axes in the feature space
represents the direction of maximum variance
in the data.
This method is used to capture variance.

Unsupervised Learning Models: Principal Component Analysis (PCA)
Let us look at how the PCA algorithm works in Scikit-Learn.
Number of components to
keep
Class
sklearn.decomposition.PCA(n_components=None, copy=True, whiten=Fals

e)
Overwrites the transform data after Removes data with

fitting them into the model lower variance
Demo 05—Principal Component Analysis (PCA)
Demonstrate how to use the PCA model to reduce the dimensions
of a dataset
Pipeline
Pipeline is mainly used to combine multiple models or estimators. Its characteristics are as follows:
All models in the pipeline

must be transformers. Once all the data is fit
It simplifies the process
The last model can either into the models or
where more than one
be a transformer or a estimators, the predict
model is required or used.
classifier, regressor, or method can be called.
other such objects.
Estimators are known as ‘model instance’.

Demo 06—Pipeline
Demonstrate how to build a pipeline
Model Persistence
Save model for the future use. No need to retrain your model every time when you need them.
It is possible to save a model by using Python's Pickle method.

Scikit-learn has a special replacement for pickle called joblib.
You can use joblib.dump and joblib.load methods.
These are more efficient for Big Data.
Demo 06—Model Persistence
Demonstrate how to persist a model for future use
Model Evaluation: Metric Functions
You can use the “Metrics” function to evaluate the accuracy of your model’s predictions.
metrics. accuracy_score
Classification metrics.average_precision_score
Clustering metrics.adjusted_rand_score
metrics.mean_absolute_error
Regression metrics.mean_squared_error
metrics.median_absolute_error
Knowledge Check
KNOWLEDGE
CHECK What is the best way to train a model?
a. Use the entire dataset as a training and testing set
b. Split the known dataset into separate training and testing sets
c. Ask the source to provide continuous data
d. Ask the source to provide categorical data

KNOWLEDGE
CHECK What is the best way to train a model?
a. Use the entire dataset as a training and testing set both
b. Split the known dataset into separate training and testing sets
c. Ask the source to provide continuous data
d. Ask the source to provide categorical data
Explanation: The best way to train a model is to split the known dataset into training and testing sets. The testing set varies
from 20% to 40%.
Quiz
Assignment 01
The given dataset contains ad budgets for different media channels and the corresponding ad sales of
XYZ firm. Evaluate the dataset to:
• Find the features or media channels used by the firm
• Find the sales figures for each channel
• Create a model to predict the sales outcome
• Split as training and testing datasets for the model
• Calculate the Mean Square Error (MSE)
Assignment 01
Instruction
Problem
s
Instructions on performing the assignment:

• Download the “Advertising Budget and Sales.csv” file from the “Resource” tab. You can load the
saved file to the Jupyter notebook that you would be using to complete the assignment.
• Follow the provided cues to complete the assignment.
Quiz
Assignment 02
The given dataset lists the glucose level readings of several pregnant women taken either during a
survey examination or routine medical care. It specifies if the 2 hour post-load plasma glucose was at
least 200 mg/dl. Analyze the dataset to:
1. Find the features of the dataset,
2. Find the response label of the dataset,
3. Create a model to predict the diabetes outcome,
4. Use training and testing datasets to train the model, and
5. Check the accuracy of the model.
Assignment 02

• Download the “pima-indians-diabetes.DATA” and “pima-indians-diabetes.NAMES” files from the
“Resources” tab. Load the .DATA file to the Jupyter notebook to work on it.
• Open the .NAMES file with a notepad application to view its text. Use this file to view the features of
the dataset and add them manually in your code.
Quiz
QUIZ
Which of the following is true with a greater value of SSR or SSE? Select all that apply.
1
a. The prediction will be more accurate, making it the best fit model.
b. The prediction will start becoming less accurate.
c. The outcome remains unaffected.
d. The model will not be the best fit for the attributes.
QUIZ
Which of the following is true with a greater value of SSR or SSE? Select all that apply.
1
a. The prediction will be more accurate, making it the best fit model.
b. The prediction will start becoming less accurate.
c. The outcome remains unaffected.
d. The model will not be the best fit for the attributes.
The correct answer is b, d.
Explanation: With higher SSR or SSE, the prediction will be less accurate and the model will not be the best fit for the attributes.
QUIZ
Class sklearn.linear_model.LogisticRegression, random_state _____.
2
a. indicates the seed of the pseudo random number generator used to shuffle data
b. defines the features state
c. represents the number of random iterations
d. specifies a random constant to be added to the decision function

QUIZ
Class sklearn.linear_model.LogisticRegression, random_state _____.
2
a. indicates the seed of the pseudo random number generator used to shuffle data
b. defines the features state
c. represents the number of random iterations
d. specifies a random constant to be added to the decision function
Explanation: The class “sklearn.linear_model.LogisticRegression, random_state” indicates the seed of the pseudo random
number generator used to shuffle data.
QUIZ
What are the requirements of the K-means algorithm? Select all that apply.
3
a. Number of clusters should be specified
b. More than one iteration should meet requisite criteria
c. Centroids should minimize inertia
d. Features should be labeled

QUIZ
What are the requirements of the K-means algorithm? Select all that apply.
3
a. Number of clusters should be specified
b. More than one iteration should meet requisite criteria
c. Centroids should minimize inertia
d. Features should be labeled
The correct answer is a, b, c.
Explanation: The K-means algorithm requires that the number of clusters be specified and that centroids that minimize inertia
be selected. It requires several iterations to fine tune itself and meet the required criteria to become the best fit model.
QUIZ
In Class sklearn.decomposition.PCA, the transform(X) method , where X is multi-
dimensional ______.
4
a. fits the model with X and applies the dimensionality reduction on X
b. transforms the data back to its original space
c. applies the dimensionality reduction on X
d. computes data co-variance with the generative model

QUIZ
In Class sklearn.decomposition.PCA, the transform(X) method , where X is multi-
dimensional _____.
3
a. fits the model with X and applies the dimensionality reduction on X
b. transforms the data back to its original space
c. applies the dimensionality reduction on X
d. computes data co-variance with the generative model
Explanation: In Class “sklearn.decomposition.PCA,” the transform(X) method applies the dimensionality reduction on X.
Key Takeaways
Scikit-learn has many built-in functions and algorithms which make it easy for Data
Scientists to build machine learning models.
Machine learning can easily and quickly extract information from large sources
of data.
Supervised and unsupervised machine learning models are two of the most
widely used learning models.
Supervised learning models are used to predict the outcome of a dataset.
Unsupervised learning models are used to find the structure of a

dataset.
For continuous data, you can use regression algorithms if it is a

supervised learning model. However, use dimensionality
reduction for unsupervised learning.
For categorical data, use classification algorithms if it is a

supervised learning model and clustering if it is an
unsupervised learning model.
This concludes “Machine Learning with Scikit-Learn.”
The next lesson is “Natural Language Processing (NLP) with Scikit-Learn.”
Lesson 9 — Natural Language Processing (NLP) with SciKit
Learn
What You Will Learn
What is Natural Language Processing

How Natural Language Processing is helpful
Modules to load content and category

Applying feature extraction techniques
Applying approaches of Natural Language Processing
Natural Language Processing (NLP)
Natural language processing is an automated way to understand and analyze natural human languages
and extract information from such data by applying machine algorithms.
Extract information
Analyze human languages
Machine algorithms and translations

(mathematics and statistics)
Data from various sources

Why Natural Language Processing
With the advancement in technology and services, the world is now a global village. However, following
are a few challenges while analyzing the huge data collection:
Analyzing tons of data
Identifying various languages
Applying quantitative analysis
Handling ambiguities
Why Natural Language Processing (contd.)
In NLP, full automation can be easily achieved by using modern software libraries, modules, and
packages.
Full Intelligent
automation processing
Knowledge about Modern software Machine

languages and world libraries models
NLP Terminology
Let us understand the NLP terminologies.
Determines where one word

Splits words, phrases, and idioms
ends and the other begins
Word
Tokenization
boundaries
Discover topics in a collection of Topic Maps to the valid root word

Stemming
documents models
NLP
Determines meaning and sense Disambiguation Tf-idf

of words (context vs. intent) Represents term frequency and inverse
Semantic document frequency
analytics
Compares words, phrases, and idioms in

a set of documents to extract meaning
The NLP Approach for Text Data
Let us look at the Natural Language Processing approaches to analyze text data.
Conduct basic text

processing
Analyze the meaning

Categorize and tag
words
Build feature based NLP

Classify text
structure
Analyze sentence
Extract information
structure
Demo 01- NLP Environmental Setup
Demonstrate the installation of NLP environment
Demo 02: Sentence Analysis
Demonstrate the sentence analysis
The NLP Applications
Let us take a look at the applications that use NLP.
Machine translation is used to translate one language into another. Google

Machine Translation Translate is an example. It uses NLP to translate the input data from one
language to another.
Speech Recognition
Sentiment Analysis
The NLP Applications (contd.)
Let us take a look at the applications that use NLP
The speech recognition application understands human speech and uses it as

Machine Translation input information. It is useful for applications like Siri, Google Now, and Microsoft
Cortana.
Speech Recognition
Sentiment Analysis
The NLP Applications (contd.)
Let us take a look at the applications that use NLP
Sentiment analysis is achieved by processing tons of data received from

Machine Translation different interfaces and sources. For example, NLP uses all social media
activities to find out the popular topic of discussion.
Speech Recognition
Sentiment Analysis
Quiz
KNOWLEDGE
CHECK
In NLP, tokenization is a way to
a. Find the grammar of the text
b. Analyze the sentence structure
c. Find ambiguities
d. Split text data into words, phrases, and idioms

KNOWLEDGE
CHECK
In NLP, tokenization is a way to
a. Find the grammar of the text
b. Analyze the sentence structure
c. Find ambiguities
d. Split text data into words, phrases, and idioms
The correct answer is . d
Explanation: Splitting text data into words, phrases, and idioms is known as tokenization and each individual word is known as
token.
Major NLP Libraries
The major NLP libraries used in Python are:
NLTK
Scikit-learn
NLP libraries
TextBlob
spaCy
The Scikit-Learn Approach
It is a very powerful library with a set of modules to process and analyze natural language data such as
texts and images and extract information using machine learning algorithms.
Built-in module Feature extraction Model training
Analyze the
Contains built-in A way to extract content based
modules to load the information from on particular
dataset’s content data which can be categories and
and categories. text or images. then train them
according to a
specific model.
The SciKit Learn Approach (contd.)
It is a very powerful library with a set of modules to process and analyze natural language data such as
texts and images and extract information using machine learning algorithms.
Pipeline building Performance Grid search for

mechanism optimization finding good
parameters
It’s a powerful
A technique in In this stage we way to search
Scikit-learn train the models parameters
approach to to optimize the affecting the
streamline the overall process. outcome for
NLP process model training
into stages. purposes.
Modules to Load Content and Category
Scikit-learn has many built-in datasets. There are several methods to load these datasets with the help of
a data load object.
Container
folder
Category 1
Data load object
Category 2
Data load object

Modules to Load Content and Category (contd.)
The text files are loaded with categories as subfolder names.
Container Extract features

folder
Category 1
NumPy array
SciPy matrix
Category 2
The attributes of a data load object are:
Bunch Contains fields and can be

Attributes accessed as dict keys or an object
Data load object Target names List of requested categories
Data Refers to an attribute in the memory

A dataset can be loaded using scikit-learn.
Import the dataset
Load dataset
Describe the dataset

Let us see how functions like type, .data, and .target help in analyzing a dataset.
View type of dataset
View data
View target
Feature Extraction
Feature extraction is a technique to convert the content into the numerical vectors to perform machine
learning.
For example: Large datasets or documents
Text feature extraction
For example: Patch extraction, hierarchical

clustering
Image feature extraction

Bag of Words
Bag of words is used to convert text data into numerical feature vectors with a fixed size.
Store
Counting
Store as the
value feature
Tokenizing
Number of
occurrences of
Assign a fixed each word
integer id to
each word
Token 1 Token 2 Token 3 Token 4
Document 1 42 32 119 3
Corpus of document Document 2 1118 0 0 89
Document 3 0 0 0 55
CountVectorizer Class Signature
Specifies number of
components to keep
Class class
sklearn.feature_extraction.text.CountVectorizer
(input='content', encoding='utf-8', Encoding used to
File name or decode the input
sequence of strings decode_error='strict', strip_accents=None,
Removes accents
lowercase=True, preprocessor=None,
Overrides string
tokenizer tokenizer=None, stop_words=None, Built-in stopwords list
token_pattern='(?u)\b\w\w+\b', ngram_range=(1, 1), Min Threshold
analyzer='word', max_df=1.0, min_df=1,
Max Threshold
max_features=None, vocabulary=None,
binary=False, dtype=<class 'numpy.int64'>)
Demo 03—Bag of Words
Demonstrate the Bag of Words technique
Text Feature Extraction Considerations
This utility deals with sparse matrix while storing them in memory.
Sparse Sparse data is commonly noticed when it comes to extracting feature
values, especially for large document datasets.
It implements tokenization and occurrence. Words with minimum two

Vectorizer letters get tokenized. We can use the analyzer function to vectorize the
text data.
It is a term weighing utility for term frequency and inverse document

frequency. Term frequency indicates the frequency of a particular term
Tf-idf
in the document. Inverse document frequency is a factor which
diminishes the weight of terms that occur frequently.
Decoding This utility can decode text files if their encoding is specified.
Model Training
An important task in model training is to identify the right model for the given dataset. The choice of
model completely depends on the type of dataset.
Models predict the outcome of new observations and datasets, and classify
Supervised documents based on the features and response of a given dataset.
Example: Naïve Bayes, SVM, linear regression, K-NN neighbors
Models identify patterns in the data and extract its structure. They are also used
Unsupervised to group documents using clustering algorithms.
Example: K-means
Naïve Bayes Classifier
It is the most basic technique for classification of text.
Advantages: Uses:
• It is efficient as it uses limited CPU • Naïve Bayes is used for sentiment
and memory. analysis, email spam detection,
• It is fast as the model training takes categorization of documents, and
less time. language detection.
• Multinomial Naïve Bayes is used when
multiple occurrences of the words
matter.
Naïve Bayes Classifier
Let us take a look at the signature of the multinomial Naïve Bayes classifier:
Learn Class prior probabilities

Class
class sklearn.naive_bayes.MultinomialNB(alpha=1.0, fit_prior=True, class_prior=None)
Smoothing parameter
(0 for no smoothing) Prior probabilities of the
classes
Grid Search and Multiple Parameters
Document classifiers can have many parameters and a Grid approach helps to search the best parameters
for model training and predicting the outcome accurately.
Category 1
Extract features of a document
Document classifier
Category 2
Grid Search and Multiple Parameters (contd.)
Document classifiers can have many parameters and a Grid approach helps to search the best parameters
for model training and predicting the outcome accurately.
Document classifier Parameter
Parameter Grid searcher
Parameter
Best parameter
Grid Search and Multiple Parameters (contd.)
In grid search mechanism, the whole dataset can be divided into multiple grids and a search can be run on
entire grids or a combination of grids.
Grid searcher
Best parameter Parameter 1
Parameter 2
Parameter 3
Pipeline
A pipeline is a combination of vectorizers, transformers, and model training.
Extracts features
around the word
of interest
Transformer Model Training

Vectorizer
(tf-idf) (document classifiers)
Converts a collection
Helps the
of text documents
model predict
into a numerical
accurately
feature vector
Demo 04—Pipeline and Grid Search
Demonstrate the Pipeline and grid search technique
Quiz
Assignment 01
Analyze the given Spam Collection dataset to:

1.View information on the spam data,
2.View the length of messages,
3. Define a function to eliminate stopwords,
4. Apply Bag of Words,
5. Apply tf-idf transformer, and
6. Detect Spam with Naïve Bayes model.
Assignment 01

• Download the Spam Collection dataset from the “Resource” tab. Upload it using the right syntax to
use and analyze it.
Quiz
Assignment 02
Problem Assignment
Analyze the Sentiment dataset using NLP to:
1. View the observations,

2. Verify the length of the messages and add it as a new column,
3. Apply a transformer and fit the data in the bag of words,
4. Print the shape for the transformer, and
5. Check the model for predicted and expected values.
Assignment 02

• Download the Sentiment dataset from the “Resource” tab. Upload it to your Jupyter notebook to
work on it.
Quiz
QUIZ
What is the tf-idf value in a document?
1
a. Directly proportional to the number of times a word appears
b. Inversely proportional to the number of times a word appears
c. Offset by frequency of the words in corpus
d. Increase with frequency of the words in corpus

QUIZ
What is the tf-idf value in a document?
1
a. Directly proportional to the number of times a word appears
b. Inversely proportional to the number of times a word appears
c. Offset by frequency of the words in corpus
d. Increase with frequency of the words in corpus
The correct answer is . a, c
Explanation: td-idf value reflects how important a word is to a document. It is directly proportional to the number of times a
word appears and is offset by frequency of the words in corpus.
QUIZ
In grid search if n_jobs = -1, then which of the following is correct?
2
a. Uses only 1 CPU core
b. Detects all installed cores and uses them all
c. Searches for only one parameter
d. All parameters will be searched on a given grid

QUIZ
In grid search if n_jobs = -1, then which of the following is correct?
2
a. Uses only 1 CPU core
b. Detects all installed cores and uses them all
c. Searches for only one parameter
d. All parameters will be searched on a given grid
Explanation: Detects all installed cores on the machine and uses all of them.
QUIZ
Identify the correct example of Topic Modeling from the following options:
3
a. Machine translation
b. Speech recognition
c. News aggregators
d. Sentiment analysis
QUIZ
Identify the correct example of Topic Modeling from the following options:
3
a. Machine translation
b. Speech recognition
c. News aggregators
d. Sentiment analysis
Explanation: ‘Topic model’ is statistical modeling and used to find latent groupings in the documents based upon the words. For
example, news aggregators.
QUIZ
How do we save memory while operating on Bag of Words which typically contain
4 high-dimensional sparse datasets?
a. Distribute datasets in several blocks or chunks
b. Store only non zero parts of the feature vectors
c. Flatten the dataset
d. Decode them
QUIZ
How do we save memory while operating on Bag of Words which typically contain
4 high-dimensional sparse datasets?
a. Distribute datasets in several blocks or chunks
b. Store only non zero parts of the feature vectors
c. Flatten the dataset
d. Decode them
Explanation: In features vector, there will be several values with zeros. The best way to save memory is to store only non zero
parts of the feature vectors.
QUIZ
What is the function of the sub-module feature_extraction.text.CountVectorizer?
5
a. Convert a collection of text documents to a matrix of token counts
b. Convert a collection of text documents to a matrix of token occurrences
c. Transform a count matrix to a normalized form
d. Convert a collection of raw documents to a matrix of TF-IDF features

QUIZ
What is the function of the sub-module feature_extraction.text.CountVectorizer?
5
a. Convert a collection of text documents to a matrix of token counts
b. Convert a collection of text documents to a matrix of token occurrences
c. Transform a count matrix to a normalized form
d. Convert a collection of raw documents to a matrix of TF-IDF features
The correct answer is . a
Explanation: The function of the sub-module feature_extraction.text.CountVectorizer is to convert a collection of text

documents to a matrix of token counts.
Key Takeaways
Let us take a quick recap of what we have learned in the lesson:
Natural Language Processing is an automated way to understand,

analyze human languages, and extract information from such data by
applying machine learning algorithms.
There are various approaches of Natural Language Processing to
analyze text data which are inter-dependent or can be independently
applied in a document.
There are two feature extraction techniques, which are text

feature extraction and image feature extraction.
Pipeline building can be used to streamline the NLP process into
stages.
Grid search mechanism is used to perform exhaustive search
on the best parameters that impacts the model.
This concludes ‘Natural Language Processing (NLP)
with Scikit-Learn’
The next lesson is ‘Data Visualization in Python with
Matplotlib and Bokeh’
Lesson 10—Data Visualization in Python using matplotlib
Explain what data visualization is and its importance in our world today
Understand why Python is considered one of the best data visualization tools
Describe matplotlib and its data visualization features in Python
List the types of plots and the steps involved in creating these plots
Data Visualization
Data visualization is a technique to present the data in a pictorial or graphical format.
Well, you might wonder why data visualization is important?

Data Visualization (contd.)
You are a Sales Manager in a leading global organization. The organization plans to study the
sales details of each product across all regions and countries. This is to identify the product
which has the highest sales in a particular region and up the production. This research will
enable the organization to increase the manufacturing of that product in that particular
region.
region.
region.
Data Visualization
The are
You maina benefits
Sales Manager
of datainvisualization
a leading global
are as
organization.
follows: The organization plans to research
the sales details of each product across all regions and countries. This is to identify the
product which has the highest sale in a particular region. This research will enable the
organization to increase its manufacturing of that product in the particular region.
Data Visualization Considerations
Three major considerations for data visualization:
Ensure the dataset is complete and relevant. This enables the Data Scientist to
use the new patterns obtained from the data in the relevant places.
Data Visualization Considerations (contd.)
Ensure you use appropriate graphical representation to convey the intended

message.
Data Visualization Considerations (contd.)
Use efficient visualization techniques that highlight all the data points.
Data Visualization Factors
There are some basic factors that one needs to be aware of before visualizing the data:
The visual effect includes the usage of appropriate shapes, colors, and sizes to represent
the analyzed data.
Data Visualization Factors (contd.)
The coordinate system helps organize the data points within the provided coordinates.
Data Visualization Factors (contd.)
The data types and scale choose the type of data, for example, numeric or categorical.
Data Visualization Factors
The informative interpretation helps create visuals in an effective and easily interpretable
manner using labels, title, legends, and pointers.
Data Visualization Tool—Python
How is data visualization performed for large and complex data?
What data visualization is?
How data visualization helps

interpret results with large data
Python Libraries
Many new Python data visualization libraries are introduced recently such as:
matplotlib
vispy pygal
bokeh folium
seaborn networkx
Python Data
Visualization Libraries
Python Libraries—matplotlib
Using Python’s matplotlib, the data visualization of large and complex data becomes easy.
matplotlib
Python 2D plotting library

Python Libraries—matplotlib (contd.)
There are several advantages of using matplotlib to visualize data. They are as follows:
Has high-quality With Jupyter notebook

graphics and plots to integration, the
print and view for a developers are free to
Can work well with spend their time
range of graphs
many operating implementing features
systems and graphics Has large community
back ends support and cross
platform support as
Is a multi-platform
it is an open source
data visualization
tool
tool; therefore, it is
Has full control
fast and efficient
over graphs or
plot styles
Advantages of using matplotlib to

visualize data
The Plot
A plot is a graphical representation of data, which shows the relationship between two variables
or the distribution of data.
Title
First Plot
1.1
Legend
1.0
0.9
0.8 Grid
Numbers
Y -axis 0.7
0.6
0.5
0.4
0.
0.3
2 0 1 3 4 5 6 7
Range
X-axis
Steps to Create a Plot
You can create a plot using four simple steps.
Step 04: Display the created plot
Step 03: Set the plot parameters
Step 02: Define or import the required dataset
Step 01: Import the required libraries

Steps to Create Plot – Example
First Plot
1.1
1.0
0.9
0.8
Numbers
0.7
0.6
0.5
0.4
0.3
0.2
0 1 3 4 5 6 7
Range
Steps to Create Plot – Example (contd.)
Generate random numbers numpy Import the required

libraries
Plot the numbers pyplot Step 01
set the grid style style

used numpy random
method
Defined the dataset Define or import the
view the created random required dataset
numbers Step 02
Print method
Set the plot

ggplot Set the style parameters
Set the legend Step 03
Set line width
Set coordinates labels
Set the title Display the created

plot
Plot the graph Step 04
Display the created plot
Knowledge Check
KNOWLEDGE
CHECK
Which of the following methods is used to set the title?
a. Plot()
b. Plt.title()
c. Plot.title()
d. Title()
KNOWLEDGE
CHECK
Which of the following methods is used to set the title?
a. Plot()
b. Plt.title()
c. Plot.title()
d. Title()
The correct answer is . b.

Explanation Plt.title() is used to set the title.
Line Properties
Line Properties Plot Graphics
alpha animated 1 linestyle 2 linewidth 3 marker

1 2
style
set the transparency set the transparency

of the line of the line
View Line Properties
matplotlib also offers various line colors.

Click View Line Properties to know more.
Line Properties (contd.)
Property Value Type

alpha float
Alias Color
animated [True | False]
b Blue
antialiased or aa [True | False] r Red
c Cyan
clip_box a matplotlib.transform.Bbox instance m Magenta
clip_on [True | False] g Green
a Path instance and a Transform y Yellow
clip_path
instance, a Patch k Black
color or c any matplotlib color w White
contains the hit testing function
dash_capstyle ['butt' | 'round' | 'projecting']
View Line Properties
linestyle or ls [ '-' | '--' | '-.' | ':' | 'steps' | ...]
linewidth or lw float value in points
marker [ '+' | ',' | '.' | '1' | '2' | '3' | '4' ] Click View Line Properties to know more.
Plot With (X,Y)
A leading global organization wants to know how many people visit its website in a particular time. This
analysis helps it control and monitor the website traffic.
2D plot
Users
Time
Plot With (X,Y)
List of users
Time
Use %matplotlib inline to display or view the plot on Jupyter notebook.

Plot with (x,y)
Web site traffic

1800
1600
Number of users 1400
1200
1000
800
600
400
200
0
6 8 10 12 14 16 18
Hrs
Controlling Line Patterns and Colors
Line Color (blue) Dashed (--)
Web site traffic

1800
1600
1400
Number of users
1200
1000
800
600
400
200
0
6 8 10 12 14 16 18
Hrs
Set Axis, Labels, and Legend Property
Using matplotlib, it is also possible to set the desired axis to interpret the result.
Axis is used to define the range on the x axis and y axis.
Set the axis

Web site traffic
200
0 Web
1500 traffic
Number of
users
1000
500
0
8 1 12 14 16
0 Hrs
Alpha and Annotation
Alpha is an attribute that controls the transparency of the line.

The lower the alpha value, the more transparent the line is.
Annotate() method is used to annotate the graph. It has several attributes which help
annotate the plot.
“Max” denotes the annotation text,

“ha” indicates the horizontal alignment,
“va” indicates the vertical alignment,
“xytext” indicates the text position,
“xy” indicates the arrow position, and
“arrowprops” indicates the properties of
the arrow.
Annotate() method is used to annotate the graph. It has several attributes which help
annotate the plot.
Multiple Plots
Monday
Web site traffic

2000
Web traffic
1500
Number of users
1000
500
0
8 10 12 14 16
Hrs
Multiple Plots
Web traffic data
Set different colors and line

widths for different days
Multiple Plots
Web site traffic
2000
Monday
Tuesday
Wednesday
1500
Number of users
1000
500
0
8 10 12 14 16
Hrs
Subplots
Subplots are used to display multiple plots in the same window.

With subplot, you can arrange plots in a regular grid.
The syntax for subplot is

It divides the current window into an m-by-n grid
subplot(m,n,p). and creates an axis for a subplot in the position
specified by p.
For example,
subplot(2,1,2) creates two subplots which are stacked vertically on a grid.

subplot(2,1,4) creates four subplots in one window.
Subplots (contd.)

With subplots, you can arrange plots in a regular grid.
Subplot(2,2,1) Subplot(2,2,2)
Subplot(2,1,1)
Grid divided
into two
vertically Grid divided
stacked plots Subplot(2,1,2) into four plots
Subplot(2,2,3) Subplot(2,2,4)
Layout
Layout and Spacing adjustments are two important factors to be considered while creating
subplots.
Use the plt.subplots_adjust() method with the parameters hspace and wspace to adjust the
distances between the subplots and move them around on the grid.
hspace
Top
Bottom
wspace
Knowledge Check
KNOWLEDGE
CHECK
Which of the following methods is used to adjust the distances between the subplots?
a. plot.subplots_adjust()
b. plt.subplots_adjust()
c. subplots_adjust()
d. plt.subplots.adjust()
KNOWLEDGE
CHECK
Which of the following methods is used to adjust the distances between the subplots?
a. plot.subplots_adjust()
b. plt.subplots_adjust()
c. subplots_adjust()
d. plt.subplots.adjust()

Explanation plt.subplots_adjust() used to adjust the distances between the subplots.
Types of Plots
You can create different types of plots using matplotlib:
Click each plot to know more.
Histogram
Scatter Plot
Heat Map
Pie Chart
Error Bar
Types of Plots (contd.)
Histogram Histograms are graphical representations of

a probability distribution. A histogram is a
kind of a bar chart.
Scatter Plot Using matplotlib and its bar chart function, bins
bins
you can create histogram charts.
Frequency
Heat Map Advantages of Histogram charts:
• They display the number of values within

Pie Chart a specified interval.
• They are suitable for large datasets as
they can be grouped within the intervals.
Error Bar Age
A scatter plot is used to graphically display the relationships between variables.

Histogram
However, to control a plot, it is recommended to use scatter() method.
Scatter Plot
It has several advantages:
• Shows the correlation between variables

Heat Map
• Is suitable for large datasets
• Is easy to find clusters
• Is possible to represent each piece of data as a
Pie Chart point on the plot
Error Bar
Histogram A heat map is a way to visualize two-dimensional data. Using heat maps, you can
gain deeper and faster insights about data than other types of plots.
Scatter Plot
• Draws attention to the risk-prone area
• Uses the entire dataset to draw meaningful insights
Heat Map • Is used for cluster analysis and can deal with large
datasets
Pie Chart
Error Bar
Histogram Pie charts are used to show percentage or proportional data.

matplotlib provides the pie() method to create pie charts.
Scatter Plot It has several advantages:
• Summarizes a large dataset in visual form

Heat Map • Displays the relative proportions of multiple
classes of data
• Size of the circle is made proportional to the
Pie Chart total quantity
Error Bar
Histogram An error bar is used to graphically represent the variability of data. It is used mainly
to identify errors. It builds confidence about the data analysis by revealing the
statistical difference between the two groups of data.
Scatter Plot
Heat Map
• Shows the variability in data and indicates the
errors.
• Depicts the precision in the data analysis.
Pie Chart
• Demonstrates how well a function and model are
used in the data analysis.
• Describes the underlying data.
Error Bar
Seaborn
Seaborn is a Python visualization library based on matplotlib. It provides a high-level

interface to draw attractive statistical graphics.
There are several advantages:
Possesses built- Has built-in statistical Has functions to

in themes for functions which reveal visualize
better hidden patterns in the matrices of data
visualizations dataset
Quiz
Assignment 01
Analyze the “auto mpg data” and draw a pair plot using seaborn library for mpg, weight, and origin.
Sources:
(a) Origin: This dataset was taken from the StatLib library maintained at Carnegie Mellon University.
•Number of Instances: 398
• Number of Attributes: 9 including the class attribute
• Attribute Information:
o mpg: continuous
o cylinders: multi-valued discrete
o displacement: continuous
o horsepower: continuous
Assignment 01
o weight: continuous
o acceleration: continuous
o model year: multi-valued discrete
o origin: multi-valued discrete
o car name: string (unique for each instance)
Quiz
Assignment 02
You have been provided with a dataset that lists Ohio State’s leading causes of death from the year
2012.
Using the two data points:
• Cause of deaths and
• Percentile
Draw a pie chart to visualize the dataset.
Assignment 02

• Download the dataset “Ohio_State_data”. Use the data provided to create relevant and required
variables.
Quiz
QUIZ
Which of the following libraries needs to be imported to display the plot on Jupyter
notebook?
1
a. %matplotlib
b. %matplotlib inline
c. import matplotlib
d. import style
QUIZ
Which of the following libraries needs to be imported to display the plot on Jupyter
notebook?
1
a. %matplotlib
b. %matplotlib inline
c. import matplotlib
d. import style

Explanation: To display the plot on Jupyter notebook “import‘%matplotlib inline.”
QUIZ
Which of the following keywords is used to decide the transparency of the plot line?
2
a. Legend
b. Alpha
c. Animated
d. Annotation
QUIZ
Which of the following keywords is used to decide the transparency of the plot line?
2
a. Legend
b. Alpha
c. Animated
d. Annotation

Explanation: Alpha decides the line transparency in line properties while plotting line plot/ chart.
QUIZ
Which of the following plots is used to represent data in a two-dimensional manner?
3
a. Histogram
b. Heat Map
c. Pie Chart
d. Scatter Plot
QUIZ
Which of the following plots is used to represent data in a two-dimensional manner?
3
a. Histogram
b. Heat Map
c. Pie Chart
d. Scatter Plot

Explanation: Heat Maps are used to represent data in a two-dimensional manner.
QUIZ
Which of the following statements limits both x and y axes to the interval [0, 6]?
4
a. plt.xlim(0, 6)
b. plt.ylim(0, 6)
c. plt.xylim(0, 6)
d. plt.axis([0, 6, 0, 6])
QUIZ
Which of the following statements limits both x and y axes to the interval [0, 6]?
4
a. plt.xlim(0, 6)
b. plt.ylim(0, 6)
c. plt.xylim(0, 6)
d. plt.axis([0, 6, 0, 6])
Explanation: plt.axis([0, 6, 0, 6]) statement limits both x and y axes to the interval [0, 6].
Key Takeaways
Data visualization is the technique to present the data in a pictorial or graphical format.
There are three major considerations for data visualization. They are clarity, accuracy, and
efficiency.
The matplotlib is a python 2D plotting library for data visualization and the creation of
interactive graphics/ plots.
A plot is a graphical representation of data which shows the relationship between
two variables or the distribution of data.
Seaborn is a Python visualization library based on matplotlib. It provides a

high-level interface to draw attractive statistical graphics.
This concludes “Data Visualization in Python using matplotlib”
The next lesson is “Web Scraping with BeautifulSoup.”

Lesson 11 — Web Scraping with BeautifulSoup
Define web scraping and explain the importance of web scraping
Lists the steps involved in the web scraping process
Describe basic terminologies such as parser, object, and tree associated with the
BeautifulSoup
Understand various operations such as searching, modifying, and navigating
the tree to yield the required result
What is Web Scraping
Web scraping is a computer software technique of extracting information from websites in an

automated fashion.
What Web Scraping is (contd.)

automated fashion.
Why Web Scraping
Every day, you find yourself in a situation where you need to extract data from the web.
Why Web Scraping (contd.)
Every day, you find yourself in a situation where you need to extract data from the web.
Web Scraping Process—Basic Preparation
There are two basic things to consider before setting up the web scraping process.
Understand the target data on the Internet
Finalize the list of websites

Web Scraping Process (contd.)
Once you have understood the target data and finalized the list of websites, you need to design
the web scraping process.
The steps involved in a typical web scraping process are as follows:
Step 1: A web request is sent to the targeted website to collect the required data.
Step 2: The information is retrieved from the targeted website in HTML or XML format from
web.
Step 3: The retrieved information is parsed to the several parsers based on the data format.
Parsing is a technique to read data and extract information from the available document.
Step 4: The parsed data is stored in the desired format. You can follow the same process to
scrap another targeted web.
Web Scraping Software vs. Web Browser
A web scraping software will interact with websites in the same way as your web browser.
A Web scraper is used to extract the information from web in routine and automated manner.
Web Browser Web Scraping Software
Displays the data Saves data from the web page to the local
file or database
Web Scraping Considerations
It’s important to read and understand the legal information and terms and conditions
mentioned in the website.
Web Scraping Considerations (contd.)
It’s important to read and understand the legal information and terms and conditions
mentioned in the website.
Legal Constraints
Notice
Copyright
Trademark Material
Patented Information
Web Scraping Tool—BeautifulSoup
SymPy Requests SQLAlchemy BeautifulSoup Twisted
Scrapy wxPython Pillow Pyglet matplotlib
Nose IPython SciPy Pygame NumPy

Web Scraping Tool: BeautifulSoup (contd.)
BeautifulSoup, is an easy, intuitive, and a robust Python library designed for web scraping.
SymPy Requests SQLAlchemy BeautifulSoup Twisted
Scrapy wxPython Pillow Pyglet matplotlib
Nose IPython SciPy Pygame NumPy

Web Scraping Tool: BeautifulSoup (contd.)
BeautifulSoup, is an easy, intuitive, and a robust Python library designed for web scraping.
Some of the reasons to choose BeautifulSoup are as follows:
Efficient tool for dissecting documents and extracting information from the web pages
Powerful sets of built-in methods for navigating, searching, and modifying a parse tree
Possess parser that supports both html and xml documents
Converts all incoming documents to Unicode automatically
Converts all outgoing documents to UTF-8 automatically

Common Data/ Page Formats on The Web
Common Data/ Page Formats on The Web (contd.)
An HTML page is one of

the oldest, easiest, and
the most popular
methods to upload
information on the web.
An HTML 5 is a new
HTML standard which
gained popularity with
the mobile devices.
XML is another popular

way to upload your
information on the web.
CSS is mainly used for the

consistent presentation
of data using cascaded
style sheets.
Application Program
Interface, or APIs, has
now become a common
practice to extract
information from the
web.
PDF is also widely used to

upload information and
reports.
JavaScript Object
Notation, or JSON, is a
lightweight and popular
format used for
information exchange on
the web.
The Parser
What is a parser?
How does it help Data Scientists in

the web scraping process?
The Parser
A Parser is a basic tool to interpret or render information from a web document.
A Parser is also used to validate the input information before processing it.
Program instructions Objects
Input Output Methods

Commands Parser
Markup tags Attributes

Importance of Parsing
Parsing data is one of the most important steps in the web scraping process.
Failing to parse the data would eventually lead to a failure of the entire process.
! !
Parser
Parser
Various Parser
There are various parsers supported by BeautifulSoup:
html.parser HTML parser is Python based, fast, and lenient.
Lxml html is not built using Python and it depends on C.

lxml html
However, it is fast and lenient in nature.
lxml xml Lxml xml is the only xml parser available and it also
depends on C.
HTML 5lib is another Python-based parser; however, it is

html5lib
slow and able to create valid HTML5.
Importance of Objects
A web document gets transformed into a complex tree of objects.
Objects
Object
Relationship
HTML
Parser
Tree
A tree is defined as a collection of simple and complex objects.

Types of Objects
BeautifulSoup transforms a complex HTML document into a complex tree of Python objects.
There are four types of objects. They are:
A tag object is an XML or HTML tag in the web document.

Tag
Tags have a lot of attributes and methods.
A NavigableString is a string or set of characters that

NavigableString
corresponds to the text present within a tag.
BeautifulSoup A BeautifulSoup represents the entire web document and

supports navigating and searching the document tree.
A Comment represents the comment or information

Comment section of the document. It is a special type of
NavigableString.
Demo: 01—Parsing web documents and extracting data using objects
This demo shows you how to scrape a web document, parse it, and use
objects to extract information.
Knowledge Check
KNOWLEDGE Which of the following object types represents a string or set of characters within a
CHECK tag?
a. Tag
b. NavigableString
c. BeautifuSoup
d. Comment
KNOWLEDGE Which of the following object types represents a string or set of characters within a
CHECK tag?
a. Tag
b. NavigableString
c. BeautifuSoup
d. Comment

Explanation: NavigableString is a string or set of characters that corresponds to the text present within a tag.
Understanding the tree
html
head Body
tiltle meta meta h1 p ul
a li li li
Understanding The Tree
html tag
Body tag
Division or a Section
Cascaded style sheets
Understanding The Tree (contd.)
BeautifulSoup Parent
html Direct child
div:
oraganizationlist
ul: ul: ul:

IT HR Finance
Siblings
li: li:
HRmanager HRmanager
div: div: div: div:

Class: name Class: ID Class: name Class: ID
Searching The Tree – Filters
With the help of the search filters technique, you can extract specific information from the parsed
document.
The filters can be treated as search criteria for extracting the information based on the elements
present in the document.
Searching The Tree – Filters (contd.)
There are various kinds of filters used for searching an information from a tree:
A string is the simplest filter. BeautifulSoup will perform a

String
match against the search string.
Regular A regular expression filters the match against the search

Expressions criteria.
List A list filters the string that matches against the search item
in the list.
A function filters the elements that matches against its only

Function
argument.
Searching the Tree—find_all()
BeautifulSoup defines a lot of methods for searching the parsed tree.
Searching methods
Methods
,, and Attributes
find_all() find()
Searching the tree with find_all()
The find_all() searches and retrieves all tags’ descendants that matches your filters.
The syntax for find_all(): Arguments
find_all(name, attrs, recursive, string, limit, **kwargs)
Method
Pass argument for tags Filter multiple attributes by

with names passing multiple keywords in
Pass argument for tags the argument
with attributes
Limit the search result to numeric value
Pass argument as Boolean passed in the argument
value for recursive operation
Search for string instead

of tags
Searching the tree with find ()
The find_all() finds the entire document looking for results.
To find one result, use find().
The find() method has a syntax similar to that of the find_all() method; however, there are some
key differences.
Method name Search Scope Match Found Match Not Found
Find_all() Scans entire document Returns list with values Returns empty list
Returns only the first match

Find() Searches only for passed argument Returns None
value
Searching the tree with other methods
Searching the parse tree can also be performed by various other methods such as the following:
Tree Search
find_parents() find_parent()
find_previous_siblings() find_previous_sibling()
find_all_next() find_next()
find_all_previous() find_previous()
find_next_siblings() find_next_sibling()
scans for all the matches scans only for the first match
Demo: 02—Demo: 02—Searching in a Tree with Filters
This demo shows the ways to search in a tree using filters.
Knowledge Check
KNOWLEDGE
CHECK
The method get_text() is used to _________.
a. parse the entire document
b. parse only part of the document
c. search the tree
d. navigate the tree

KNOWLEDGE
CHECK
The method get_text() is used to _________.
a. parse the entire document
b. parse only part of the document
c. search the tree
d. navigate the tree

Explanation The method get_text() is used to parse only part of the document.
Navigating options
With the help of BeautifulSoup, it is easy to navigate the parse tree based on the need.
There are four options to navigate the tree:
Navigating Down
Navigating Up
Navigating
Sideways
Navigating Back
and Forth
Navigating options
There are four options to navigate the tree. They are:
Navigating Down This technique shows you how to extract information from children
tags. Following are the attributes used to navigate down:
Navigating Up • .contents and .children

• .descendants
Navigating • .string
Sideways • .strings and stripped_strings
Navigating Back
and Forth
Navigating options
Navigating Down Navigating Up:

Every tag has a parent and two attributes, .parents and .parent,to
Navigating Up help navigate up the family tree.
Navigating
Sideways
Navigating Back
and Forth
Navigating options
Navigating Down Navigating Sideways:
This technique shows you how to extract information from the same
Navigating Up level in the tree.
The attributes used to navigate sideways are .next_sibling and
Navigating .previous_sibling.
Sideways
Navigating Back
and Forth
Navigating options
Navigating Down Navigating Back and Forth:
This technique shows you how to parse the tree back and forth.
Navigating Up Following are the attributes used to navigate back and forth are:
.next_element and .previous_element
Navigating .next_elements and .previous_elements
Sideways
Navigating Back
and Forth
Demo: 03—Navigating a Tree
This demo shows how to navigate the web tree using various techniques.
Knowledge Check
KNOWLEDGE
CHECK
Which of the following attributes is used to navigate up?
a. .next_element
b. .parent
c. .previous_elements
d. .next_sibling
KNOWLEDGE
CHECK
Which of the following attributes is used to navigate up?
a. .next_element
b. .parent
c. .previous_elements
d. .next_sibling

Explanation:The .parent attribute is used to navigate up.
Modifying The Tree
With BeautifulSoup, you can also modify the tree and write your changes as a new HTML or
XML document.
There are several methods to modify the tree:
.string()
append()
NavigableString()
.new_tag()
insert()
Insert_before() and insert_after()
clear()
extract()
decompose()
replace with ()
wrap()
unwrap()
Modifying The Tree
With BeautifulSoup, you can also modify the tree and write your changes as a new HTML or
XML document.
There are several methods to modify the tree:
.string()
append()
NavigableString()
.new_tag()
insert()
Insert_before() and insert_after()
clear()
extract()
decompose()
replace with ()
wrap()
unwrap()
Demo: 04—Modifying the Tree
This demo shows you ways to modify a web tree to get the desired result
with the help of an example.
Parsing Only Part of the Document
But how can you overcome this problem?
Use SoupStrainer class
Allows you to choose the part of the

document to be parsed
This feature of parsing a part of the document will not work with the html5lib parser.
Demo: 05—Parsing part of the document
This demo shows you how to parse only a part of document with the help of
an example.
Output : Printing and Formatting
Output
Prettify()
Printing
Unicode() or str()
Formatting
Output : Printing and Formatting
Prettify()
The prettify or pretty printing method turns a parse tree

into a decorative formatted Unicode string.
Output
Printing
Formatting
Unicode() or str()
Output : Printing and Formatting (contd.)
Prettify()
Output
Unicode() or str()
Printing The unicode()or str() method turns a parse tree into a non-
decorative formatting string.
Formatting
The formatters are used to generate different types of output with the desired formatting.
Output
Html and xml

Printing
Minimal
Formatting Html and xml formatting
will convert unicode
none
characters into html and
xml entities respectively.
Uppercase and
lowercase
Output
Html and xml

Printing
Minimal
Formatting The minimal formatting
none will processes content
with valid html/ xml tags.
Uppercase and
lowercase
Output
Html and xml

Printing
Minimal
Formatting None formatting will not
None modify the content or
string on output.
Uppercase and
lowercase
Output
Html and xml

Printing
Minimal
Uppercase and lowercase
Formatting formatting will convert
none string values to
uppercase and
lowercase, respectively.
Uppercase and
lowercase
Demo: 06—Formatting and Printing
This demo shows the ways to format, print, and encode the web
document.
Encoding
Document Encoding Output Encoding
• HTML or XML documents are

written in specific encodings such • When you write a document from
as, ASCII or UTF-8. Beautiful Soup, you get a UTF-8
• When we load the document into document irrespective of the
BeautifulSoup, it gets converted original encoding.
into Unicode. • If some other encoding is required,
• The original encoding can be we can pass it to prettify.
extracted from attribute .original
encoding of the Beautiful Soup
object.
Knowledge Check
Assignment 01
Scrape the Simplilearn website page and perform the following tasks:
• View and print the Simplilearn web page content in a proper format
• View the head and title
• Print all the href links present in the Simplilearn web page
Simplilearn website URL: http://www.simplilearn.com/

Assignment 01

• Use Simplilearn’s resource page URL in the Jupyter notebook to view and evaluate it.
Knowledge Check
Assignment 02
Scrape the Simplilearn website resource page and perform the following tasks:
• View and print the Simplilearn web page content in a proper format
• View the head and title
• Print all the href links present in the Simplilearn web page
• Search and print the resource headers of the Simplilearn web page
• Search resource topics
• View the article names and navigate through them
Simplilearn website URL: http://www.simplilearn.com/resources

Assignment 02

• Download the web scraping dataset from the “Resource” tab. Upload the dataset to your Jupyter
notebook to view and evaluate it.
Quiz
QUIZ
Which of the following is the only xml parser?
1
a. html.parser
b. lxml
c. lxml.xml
d. html5lib
QUIZ
Which of the following is the only xml parser?
1
a. html.parser
b. lxml
c. lxml.xml
d. html5lib
The correct answercis .

Explanation: lxml.xml is the only xml parser available for BeautifulSoup object.
QUIZ
In which of the following formats is the BeautifulSoup output encoded?
2
a. ASCII
b. Unicode
c. latin-1
d. UTF-8
QUIZ
In which of the following formats id the BeautifulSoup output encoded?
2
a. ASCII
b. Unicode
c. latin-1
d. UTF-8

Explanation: The output of the BeautifulSoup is always UTF-8 encoded.
QUIZ
Which of the following libraries is used to extract a web page?
3
a. Beautiful Soup
b. Pandas
c. Requests
d. NumPy
QUIZ
Which of the following libraries is used to extract a web page?
3
a. Beautiful Soup
b. Pandas
c. Requests
d. NumPy

Explanation: Requests is the right API to extract the web page.
QUIZ
Which of the following is NOT an object in BeautifulSoup?
4
a. Tag
b. NextSibling
c. NavigableString
d. Comment
QUIZ
Which of the following is NOT an object in BeautifulSoup?
4
a. Tag
b. Next sibling
c. NavigableString
d. Comment

Explanation: NextSibling is a navigation method.
Key Takeaways

automated fashion.
A Parser is a basic tool to interpret or render the information from a web document.
Objects are used to extract the required information from a tree structure by
searching or navigating through the parsed document.
A tree can be defined as a collection of simple and complex objects.
BeautifulSoup transforms a complex HTML document into a complex tree of Python objects.
This concludes “Web Scraping with BeautifulSoup”
The next lesson is “Python integration with Hadoop, MapReduce, and Spark”
Lesson 12—Python Integration with Hadoop MapReduce
and Spark
What You’ll Learn
Why Python should be integrated with Hadoop
Brief overview of the ecosystem and architecture of Hadoop
How MapReduce functions
How Apache Spark functions and what its benefits are
Write Python programs for Hadoop operations

Quick Recap: Need for Real Time Analytics
Demographics Orders Products Visits/Channels Social Media Customer Support

Sentiments and Surveys
Real Time
Analytics
lmprove Conversion Targeted Marketing Better Customer Increased Sales

Satisfaction
Quick Recap: Why Python
Data Scientists all over the world prefer to use Python for analytics because of its ease and support to
carry out all the aspects of Data Science.
Acquire
Wrangle
Explore
Model
Data Science
Visualize
Bokeh
Disparity in Programming Languages
However, Big Data can only be accessed through Hadoop which is completely developed and
implemented in Java. Also, analytics platforms are coded in different programming languages.
Hadoop
Infrastructure
(HDFS)
Python
Big Data
Spark Analytics
Platform
Data Science
Big Data Platform

Integrating Python with Hadoop
But as Python is a Data Scientist’s first language of choice, both Hadoop and Spark provide Python APIs
that allow easy access to the Big Data platform.
Hadoop
Infrastructure
(HDFS)
Python Python
APIs
Big Data
Spark Analytics
Platform
Data Science
Big Data Platform

Need for Big Data Solutions for Python
There are several reasons for creating Big Data solutions for Python.
Multiple Data Processing

Frameworks
Multiple
Programming
Languages
Data Scientist Python APIs

Multiple Big Data
Vendors
Hadoop: Core Components
Hadoop
HDFS MapReduce
(Hadoop Distributed File System)
• It is responsible for storing data on a cluster • MapReduce is a data processing framework to

• Data is split into blocks and distributed across process data on the cluster
multiple nodes in a cluster • Two consecutive phases: Map and Reduce
• Each block is replicated multiple times • Each map task operates on discrete portions of data
o Default is 3 times • After map, reduce works on the intermediate data
o Replicas are stored on different nodes distributed on nodes
Hadoop: The System Architecture
This example illustrates the Hadoop system architecture and the ways to store data in a cluster.
D
a
t
a
S
o Name node
u Large file
r
c Data nodes
(Hadoop cluster)
e
s File blocks
(64MB or 128MB)
Secondary name
node
MapReduce
The second core component of Hadoop is MapReduce, the primary framework of the HDFS architecture.
input
HDFS
sort Output
HDFS
Split 0 map copy
merge
HDFS
reduce part 0
replication
Split 1 map
HDFS
reduce part 1
replication
Split 2 map
MapReduce: The Mapper and Reducer
Let us discuss the MapReduce functions—mapper and reducer—in detail.
Mapper Reducer
• Mappers run locally on the data nodes • All intermediate values for a given
to avoid the network traffic. intermediate key are combined together
• Multiple mappers run in parallel into a list and given to a reducer.
processing a portion of the input data. • This step is known as ‘shuffle and sort’.
• The mapper reads data in the form of • The reducer outputs either zero or
key-value pairs. more final key-value pairs. These
• If the mapper writes generates an output, it is written are written to HDFS.
in the form of key-value pairs.
Hadoop Steaming: Python API for Hadoop
Hadoop Streaming acts like a bridge between your Python code and the Java-based HDFS, and lets you
seamlessly access Hadoop clusters and execute MapReduce tasks.
Hadoop
Streaming
Data Scientist Python API

Mapper in Python
Python supports map and reduce operations:
Suppose you have list of numbers you want to square =

[1, 2, 3, 4, 5, 6 ]
Square function is written as follows:

def square(num):
return num * num
You can square this list using the following code:

squared_nums = map(square, numbers)
Output would be:

[1, 4, 9, 16, 25, 36]
Reducer in Python
Reduce written in Python:
Suppose you want to sum the squared numbers:

[1, 4, 9, 16, 25, 36]
Use the sum function to add two numbers

def sum(a, b ):
return a + b
You can now sum the numbers using the reduce function
sum_squared = reduce(sum, squared_nums)
Output would be:

[91]
Cloudera QuickStart VM Set Up
Cloudera provides enterprise-ready Hadoop Big Data platform which supports Python as well.
To set up the Cloudera Hadoop environment, visit the Cloudera link:
http://www.cloudera.com/downloads/quickstart_vms/5-7.html
Cloudera recommends that you use 7-Zip to extract these files. To download and install it, visit the link:
http://www.7-zip.org/
Cloudera QuickStart VM: Prerequisites
• These 64-bit VMs require a 64-bit host OS and a virtualization product that can support a 64-bit guest
OS.
• To use a VMware VM, you must use a player compatible with WorkStation 8.x or higher:
• Player 4.x or higher
• Fusion 4.x or higher
• Older versions of WorkStation can be used to create a new VM using the same virtual disk (VMDK file),
but some features in VMware Tools are not available.
• The amount of RAM required varies by the run-time option you choose
QuickStart VMware Player: Windows, Linux & VMware Fusion: Mac
To launch the VMware, visit the VMware link:
https://www.vmware.com/products/player/ https://www.vmware.com/products/fusion/
playerpro-evaluation.html fusion-evaluation.html
QuickStart VMware Image
Launch VMware player with Cloudera VM
Launch Terminal
Account:
username: cloudera
password: cloudera
QuickStart VM Terminal
Step 01 Step 02
Unix command :
• pwd to verify present working directory
• ls -lrt to list files and directories
Demo 01—Using Hadoop Streaming for Calculating Word Count
Demonstrate how to create a MapReduce program and use Hadoop
Streaming to determine the word count of a document
Knowledge Check
KNOWLEDGE
CHECK
What is the usual size of the data block on HDFS?
a. 32 MB
b. 64 MB
c. 100 MB
d. 1 GB
KNOWLEDGE
CHECK
What is the usual size of the data block on HDFS
a. 32 MB
b. 64 MB
c. 100 MB
d. 1 GB

Explanation The usual data block size on HDFS is 64 MB.
MapReduce Uses Disk I/O Operations
HDFS HDFS HDFS HDFS

read write read write
Iteration 1 Iteration 2
Input
Query 1 Result 1
Query 2 Result 2
Input Query 3
Result 3
Apache Spark Uses In-Memory Instead of Disk I/O
HDFS
read
Iteration 1 Iteration 2
Memory (RAM) Memory (RAM)
Input
Query 1 Result 1
Query 2
Result 2
Input Query 3
Result 3
Distributed memory
10-100 X faster than network and disk

Hardware Requirements for MapReduce and Spark
Hard Drives Hard Drives
CPUs
CPUs Memory
MapReduce Spark
Apache Spark Resilient Distributed Systems (RDD)
Some basic concepts about Resilient Distributed Datasets (RDD) are listed here:
• The main programming approach of Spark is RDD.

• They are fault-tolerant collections of objects spread across a cluster that you can
operate on in parallel. They can automatically recover from machine failure.
• You can create an RDD either by copying the elements from an existing collection or
by referencing a dataset stored externally.
• RDDs support two types of operations: transformations and actions.
• Transformations use an existing dataset to create a new one.
• Example: Map, filter, join
• Actions compute on the dataset and return the value to the driver program.
• Example: Reduce, count, collect, save
If the available memory is insufficient, then the data is written to disk.

Advantages of Spark
Listed here are some of the advantages of using Spark:
Faster: 10 to 100 times faster than Hadoop MapReduce
• Simple data processing framework

Simplified: • Interactive APIs for Python for faster application
development
Efficient: Has multiple tools for complex analytics operations
Integrated: Can be easily integrated with existing Hadoop

infrastructure
PySpark : Python API for Spark
PySpark is the Spark Python API which enables data scientists to access Spark programming model
PySpark
Data Scientist Python API

PySpark : RDD Transformations and Actions
Transformation Action
Transformation Description Action Description
Returns RDD, formed by

Returns all elements of the
map() passing data element of the collect()
dataset as an array
source
Returns RDD based on Returns the number of
filter()
selection count() elements present in the
dataset
Maps items present in the
flatMap() Returns the first element in the
dataset and returns sequence first()
dataset
Returns key value pairs where
reduceByKey() values for which each key is Returns number of elements
aggregated by value take(n) (n) as specified by the number
in the parenthesis
SparkContext or SC is the entry point to spark for the spark application

Spark Tools
Spark MLlib
Spark GraphX
SQL (machine
Streaming (graph)
Learning)
Spark
Interactive Python APIs

Apache Spark Set Up
To set up the Apache Spark environment, access the link:
http://spark.apache.org/downloads.html
Please use 7-Zip to extract these files.

Apache Spark : Environment Variable Set Up
[installed directory]\spark-1.6.1-bin-
hadoop2.4\spark-1.6.1-bin-hadoop2.4
[installed directory] \spark-1.6.1-bin-

hadoop2.4\spark-1.6.1-bin-
hadoop2.4\bin
Apache Spark: Jupyter Notebook Integration
Setup the
pyspark
notebook
specific
variables
Run the pyspark

command
Check SparkContext
Demo 02—Using PySpark to Determine Word Count
Demonstrate how to use the Jupyter integrated PySpark API to
determine the word count of a given dataset
Knowledge Check
KNOWLEDGE What happens if the available memory is insufficient while performing RDD
CHECK transformations?
a. The RDD process waits for memory to be available
b. The process is cancelled by scheduler
c. The data is written to the disk
d. The RDD process fails

KNOWLEDGE What happens if the available memory is insufficient while performing RDD
CHECK transformations?
a. The RDD process waits for memory to be available
b. The process is cancelled by scheduler
c. The data is written to the disk
d. The RDD process fails
The correct answer is . c.

Explanation The data is written to the disk in case the memory is insufficient while performing transformations.
Quiz
Assignment 01
Instruction
Problem
s
To determine the word count of the given Amazon dataset:

• Create a MapReduce program to determine the word count of the Amazon dataset
• Submit the MapReduce task to HDFS and run it
• Verify the output
Click each tab to know more. Click the Resources tab to download the files for this
assignment.
Assignment 01
Instruction
Problem
s

• Download the “Amazon text dataset.txt” file from the “Resource” tab. Use the QuickStart VM
terminal to create a file and copy-paste the Amazon dataset into it.
Special instructions:
• This assignment is done purely on Cloudera’s QuickStart VM. You may need to learn a few basic
UNIX commands to operate the program.
• For any cues, refer the Hadoop Streaming demo provided in the lesson.
Quiz
Assignment 02
Instruction
Problem
s
Use the given dataset to count and display all the airports based in New York using PySpark. Perform
the following steps:
• View all the airports listed in the dataset
• View only the first 10 records
• Filter the data for all airports located in New York
• Clean up the dataset, if required
Assignment 02
Instruction
Problem
s

• Download the “Airport.csv” file from the “Resource” tab. You can load the saved file to the Jupyter
notebook that you would be using to complete the assignment..
Quiz
QUIZ
What are the core components of Hadoop? Select all that apply.
1
a. MapReduce
b. HDFS
c. Spark
d. RDD
QUIZ
What are the core components of Hadoop? Select all that apply.
1
a. MapReduce
b. HDFS
c. Spark
d. RDD
The correct answer is . a&b
Explanation: MapReduce and HDFS are the core components of Hadoop.

QUIZ
MapReduce is a data processing framework which gets executed _____.
2
a. at DataNode
b. at NameNode
c. on client side
d. in memory
QUIZ
MapReduce is a data processing framework which gets executed _____.
2
a. at DataNode
b. at NameNode
c. on client side
d. in memory
Explanation: The MapReduce program is executed at the data node and the output is written to the disk.
QUIZ
Which of the following functions is responsible for consolidating the results produced
3 by each of the Map() functions/tasks?
a. Reducer
b. Mapper
c. Partitioner
d. All of the above

QUIZ
Which of the following functions is responsible for consolidating the results produced
3 by each of the Map() functions/tasks?
a. Reducer
b. Mapper
c. Partitioner
d. All of the above
Explanation: Reducer combines or aggregates results produced by mappers.

QUIZ
What transforms input key-value pairs to a set of intermediate key-value pairs?
4
a. Mapper
b. Reducer
c. Combiner
d. Partitioner
QUIZ
What transforms input key-value pairs to a set of intermediate key-value pairs?
4
a. Mapper
b. Reducer
c. Combiner
d. Partitioner
Explanation: Mapper processes input data to intermediate key-value pairs which are in turn processed by reducers.
Key Takeaways
As Python is a Data Scientist’s preferred choice of language, it is important to provide

Big Data solutions that accommodates it.
There are two primary components of Hadoop architecture: Hadoop Distributed

File System or HDFS and MapReduce.
Both Hadoop and Spark provide Python APIs to help Data Scientists use the Big
Data platform.
MapReduce has two functions—mapper and reducer.
MapReduce carries out computations on data through disk I/O

operations while Apache Spark carries them out in-memory.
The main programming approach of Spark is RDD.
Spark is almost 10 to 100 times faster than Hadoop MapReduce.
There are mainly four components in Spark tools: Spark SQL,

Spark Streaming, Mllib, and GraphX.
This concludes “Python Integration with MapReduce and Spark”.
This is the final lesson of the Data Science with Python Course.
Quiz
Project 01: Stock Market Data Analysis
After learning about Data Science in depth, it is now time to implement the knowledge gained through
this course in real-life scenarios. We will provide you with four scenarios where you need to implement
data science solutions. To perform these tasks, you can use the different Python libraries such as
NumPy, SciPy, Pandas, scikit-learn, matplotlib, BeautifulSoup, and so on.
You will focus on acquiring stock data information for the companies listed.
The scope of the project is as follows:

Problem Instructions Solution
• Import the financial data using Yahoo data reader for the following companies:
o Yahoo
o Apple
o Amazon
o Microsoft
o Google
• Perform fundamental data analysis

o Fetch the last one year’s data
o View the values of Apple’s stock
o Display the plot of closing price
o Display the stock trade by volume
o Plot all companies’ data together for closing prices
• Perform Daily Return Analysis and show the relationship between different stocks
o Plot the percentage change plot for Apple’s stock
o Show a joint plot for Apple and Google
o Use PairPlot to show the correlation between all the stocks
• Perform risk analysis

Instructions to perform the project:

• Download the “Anaconda Installation Instructions” document from the “Resources” tab to view the
steps to install Anaconda and the Jupyter notebook.
• Download the “Project 01” notebook and upload it on the Jupyter notebook to access it.
• Follow the provided cues to complete the project.
We recommend you to first solve the project and then view the solution to assess your learning.
Hope you had a good experience working on the project “Stock Market Data Analysis.”
Go to the next screen to assess your performance.
Click Next to view the demo.

Quiz
Project 02: Titanic data set analysis
After learning about Data Science in depth, it is time to implement the knowledge gained through this
course in real-life scenarios. We are providing four real-life scenarios where you can implement data
science solutions. To develop solutions to these problems, you can use various Python libraries like
NumPy, SciPy, Pandas, Scikit-learn, Matplotlib, BeautifulSoup, and so on.
Project details are given below:
Project 02: Titanic Data Set Analysis
Titanic Dataset Analysis
On April 15, 1912, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 passengers
and crew. This tragedy shocked the world and led to better safety regulations for ships. Here, we ask
you to perform the analysis through the exploratory data analysis technique. In particular, we want
you to apply the tools of machine learning to predict which passengers survived the tragedy.
The details of these projects and their scope are listed in the following sections.
Click each tab to know more. Click the Resources tab to download the files for this project.
• Data acquisition of the Titanic dataset

o train dataset
o test dataset
• Perform the Exploratory Data Analysis (EDA) - for train dataset

o passengers age distribution
o passengers survival by age
o passengers survival breakdown
o passengers class distribution
o passengers embarkation by locations
• Perform machine learning to train the machine model and

o create user defined function to load train data set
o create user defined function to load test data set
o create machine model
o train the machine
o predict whether a passenger survived the tragedy or not
o persist the mode for future re-use
o
Instructions to perform the project:

• Download the “Anaconda Installation Instructions” document from the “Resources” tab to view the
steps to install Anaconda and the Jupyter notebook.
• Download the “Project 02” notebook and upload it on the Jupyter notebook to access it.
• Follow the provided cues to complete the project.
We recommend you to first solve the project and then view the solution to assess your learning.
Hope you had a good experience working on the project “Titanic data set analysis.”
Go to the next screen to assess your performance.
Click Next to view the demo.
To view the demo for this project, click Next.

Thank You

Data Science With Python

Uploaded by

Copyright:

Available Formats

Data Science With Python

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Data Science With Python

Uploaded by

Copyright:

Available Formats

Data Science with Python

Lesson 1—Data Science Overview

Know what Data Science is

Discuss the roles and responsibilities of a Data Scientist

List various applications of Data Science

Understand how Data Science and Big Data work together

Explore Data Science as a discipline

Understand how and why Data Science is gaining importance

Understand what Python is and what problems it resolves

Domain Expertise and Scientific Methods Technology

Mathematical and Scientific Python Application

Data Processing Library

Data analysis can be:

These technologies help Data Scientists to Technology

• Build and train machine learning models Operating System

A Data Scientist should be able to

Big Data is characterized by the following:

Volume Enormous amount of data generated from various sources

Large amount of data streaming in at great speeds, which requires quick

Variety Different formats of data: Structured, Semi-structured, and Unstructured

Autocomplete feature is fed by data models (Machine Learning)

Fast and real-time analytics is made possible Influencing Factors

Biometric data IOT Gateway Data transfer to Enterprise

Data transfer to Enterprise

Credit report, Credit history,

Make informed Engagement

Large collection of datasets

Some of the challenges Data Scientists face in the real

Efficient and multi platform support

Huge collection of libraries,

Big open source community

Integrates well with enterprise apps

Great vendor and product support

Python Tool for data analysis and processing

Big Data Processing Framework

Enterprise Big Data Platform

Data Science is a discipline that combines aspects of statistics, mathematics,

Data Scientists solve big problems in public and private sectors.

Python is a powerful language and a preferred tool for Data

a. asks the right questions

c. performs data wrangling and data visualization

d. All of the above

a. Asks the right questions

c. Performs data wrangling and data visualization

d. All of the above

The correct answer is d.

a. to scrub inappropriate content.

b. to build a Query Volume.

c. to tag the location to a query.

d. to find similar instances on the web.

a. to scrub inappropriate content

b. to build a Query Volume

c. to tag the location to a query

d. to find similar instances on the web

The correct answer is b, c .

a. Data wrangling, exploration, modeling, acquisition, and visualization

b. Data exploration, acquisition, modeling, wrangling, and visualization

c. Data acquisition, wrangling, exploration, modeling, and visualization

d. Data modeling, acquisition, exploration, wrangling, and visualization