Data Science with Python

Data Analytics Overview

 Learning Objectives

By the end of this lesson, you will be able to:

Describe Data Analytics process and its steps

List the skills and tools required for data analysis

Understand the challenges of the Data Analytics process

Explain Exploratory data analysis technique

Illustrate data visualization techniques

Describe Hypothesis testing

Data Analytics
 Why Data Analytics

Data by itself is just an information source. But, unless you understand it, you will not be able to use it effectively.

Date Description Deposit Withdrawal Balance

Apr 1 ATM Post Debit 100 $200,000

Apr 2 PayPal Transfer 231054 200 $202,000

Apr 3 Simplilearn course fee 150 $200,500

Information source;
Apr 4 Starbucks Café 210 $198,400 overall patterns not
clearly visible
Apr 5 Walmart TX 230 $196,100

Apr 6 eBay swiss watch 239 250 $193,600

Apr 7 Caterpillar black boots men 270 $190,900

Apr 8 Halo blue shirt 831 160 $189,300

 Why Data Analytics

When the transaction details are presented as a line chart, the deposit and withdrawal patterns become apparent.

Overall pattern
 Why Data Analytics

When the transaction details are presented as a line chart, the deposit and withdrawal patterns become
apparent. It helps view and analyze general trends and discrepancies.

Discrepancy
 Introduction to Data Analytics

Data Analytics is a combination of processes to extract information from datasets.

Data Analytics Process
Data Analytics Process
 Business Problem

The process of analytics begins with questions or business problems of stakeholders.

Sales Inventory

Customers Traffic volume

Business problems trigger the need to

analyze data and find answers.
 Data Acquisition

Collect data from various sources for analysis to answer the question raised in step 1.

Twitter, Facebook,
LinkedIn, and other social
media and information
sites provide streaming
APIs.

Data Scientist Expertise:

File handling
• Database Skills Data
• File formats
• Web scraping
Server logs can be
extracted from enterprise
system servers to analyze
and optimize application
performance.
 Data Wrangling and Exploration

Data wrangling is the most important phase of the data analytic process.

Data cleansing Data Data pattern

Data discovery
manipulation

Data Wrangling Data Exploration

 Data Wrangling: Challenges

This phase includes data cleansing, data manipulation, data aggregation, data split, and reshaping of data.

Causes of challenges in the data wrangling phase:

• Unexpected data format

• Erroneous data
• Voluminous data to be manipulated
• Classifying data into linear or clustered
• Determining relationship between observation, feature, and response

Data wrangling is the most challenging phase and takes up 70% of the Data Scientist’s time.
 Data Exploration: Model Selection

This phase includes data cleansing, data manipulation, data aggregation, data split, and reshaping of data.

Model selection
• Based on the overall data analysis process
• Should be accurate to avoid iterations
• Depends on pattern identification and algorithms
• Depends on hypothesis building and testing
• Leads to building mathematical statistical functions
 Exploratory Data Analysis (EDA)

APPROACH FOCUS ASSUMPTIONS EDA TECHNIQUES

Quantitative:
EDA techniques
The focus is on data; Provides numeric
EDA approach make minimal or no
its structure, outputs for the
studies the data to assumptions. They
outliers, and models inputted data
recommend suitable present and show
suggested by the Graphical:
models that best fit all the underlying
data. Uses statistical
the data. data without any
functions for
data loss.
graphical output
 EDA: Quantitative Technique

EDA: Quantitative technique has two goals, measurement of central tendency and spread of data.

Measurement of Central Tendency

Mean Mean is the point which indicates how centralized the data points are.
• Suitable for symmetric distributions

Median Median is the exact middle value.

• Suitable for skewed distributions and for catching outliers in the
dataset

Mode Mode is the most common value in the data (frequency).

Measurement of Spread
Variance Variance is approximately the mean of the squares of the deviations.

Standard deviation Standard deviation is the square root of the variance.

Inter-quartile range Inter-quartile range is the distance between the 75th and 25th percentile.
It’s essentially the middle 50% of the data.
 EDA: Graphical Technique

Histograms and scatter plots are two popular graphical techniques to depict data.

Histogram graphically summarizes the distribution of

25

a univariate dataset.

It shows:
Frequency
20

• the center or location of data (mean, median, or mode)

• the spread of data
15

• the skewness of data

• the presence of outliers
• the presence of multiple modes in the data
10

1 1 2 2
0 5 Per 0
Miles 5
Gallon
 EDA: Graphical Technique

Histograms and scatter plots are two popular graphical techniques to depict data.

Scatter plot represents relationships between two variables.

It can answer these questions visually:

• Are variables X and Y related?

• Are variables X and Y linearly related?
• Are variables X and Y non-linearly related?
• Does change in variation of Y depend on X?
• Are there outliers?
Hypothesis
 Conclusion or Prediction

This step involves reaching a conclusion and making predictions based on the data analysis.

• Involves heavy use of mathematical and statistical functions

• Requires model selection, training, and testing to help in forecasting

• Is called machine learning as data analysis is fully or semi-automated with

minimal or no human intervention

 Meaning of Hypothesis

Hypothesis is used to establish the relationship between dependent and independent variables.

Hypothesis building begins in the data exploration stage, but

becomes more mature in the conclusion or prediction phase.

Data Exploration Stage Conclusion and Prediction

Key Considerations of Hypothesis Building

Testable explanations of a problem or observation
Used in quantitative and qualitative analyses to provide research solutions
Involves two variables, one dependent on another
Independent variable manipulated by the researcher
Dependent variable changes when the independent variable changes
 Hypothesis Building Using Feature Engineering

Domain knowledge leads to hypothesis building using feature engineering.

Feature engineering involves domain expertise to:

• Make sense of data
• Construct new features from raw data automatically
• Construct new features from raw data manually
 Hypothesis Building Using a Model
There are three phases to hypothesis building, which are model building, model evaluation, and model deployment.

Phase 1: Model Building

• Identify best input variables
• Evaluate the model’s capacity to forecast with these
variables

Phase 2: Model Evaluation

• Train and test the model for accuracy
• Optimize model accuracy, performance, and
comparisons with other models

Phase 3: Model Deployment

• Use the model for prediction
• Use the model to compare actual outcome with
expectations
 Hypothesis Testing

Draw two samples from the population and calculate the difference between their means.

μ1 Calculating the
difference
S1 between the two
means is
hypothesis
testing.
μ2
S2
Hypothesis Testing

Alternative Hypothesis
• Proposed model outcome is
accurate and matches the data.
• There is a difference between the
means of S1 and S2.

Null Hypothesis
• Opposite of the alternative
hypothesis.
• There is no difference between
the means of S1 and S2.
 Hypothesis Testing Process

Choosing the training and test dataset, and evaluating them with the null and alternative hypothesis.

Usually the training dataset is between 60% to 80% of the big dataset and the test dataset is between
20% to 40% of the big dataset.
Data Visualization
 Communication

The last step of data analysis is communication, where the analyzed data is formally presented to stakeholders.

Forms of Data analysis presentations:

• Visual graphs
• Plotting maps
• Reports
• Whitepaper reports
• PowerPoint presentations
 Data Visualization

Data visualization techniques are used for effective communication of data.

Benefits of data visualization: Examples of data visualization:

• Simplifies quantitative information through visuals • Presenting information about new and existing
• Shows the relationship between data points and customers on the website and their behavior when
variables they access the website
• Identifies patterns • Representing web traffic pattern for the website,
• Establishes trends for example, more activity on the website in the
morning than in the evening
 Plotting

Plotting is a data visualization technique used to represent underlying data through graphics.

Features of plotting:

• Plotting is like telling a story about data using

different colors, shapes, and sizes.
• Plotting shows the relationship between variables.
• Example:
o Change in value of Y results in change in
value of X
o X is independent of y
 Data Types for Plotting

There are two types of numerical data:

Discrete Data: Distinct or counted values
Example: Number of employees in a company or number of students in a class
Continuous Data: Values within a range that can be measured
Example: Height can be measured in feet or inches and weight can be measured in pounds or
Numerical Data kilograms

There are two types of categorical data:

Cluster or group: Grouped values
Example: Students can be divided into different groups based on height: Tall, Medium, and Short
Ordinal data: Grouped values as per ranks
Example: A ranking system; a five-point scale with ranks like Agree, Strongly agree, and
Categorical Data Disagree

Data is measured in time blocks, such as, date, month, year, and time (hours, minutes, and
seconds

Time Series
 Types of Plot

Different data types can be visualized using various plotting techniques.

 Data Analytics: An Iterative Process
Data Analytics is an iterative process involving tracing back the steps, often to ensure that you are on the right track.

Process Result: Question is answered or business problem is solved.

 Data Analytics: Skills and Tools
Skills and tools required for each step of the data analysis process.

Question or Data Data Data Conclusion or Communication or

Business Problem Acquisition Wrangling Exploration Predictions Data Visualization
• Ability to ask • BeautifulSoup • CSV or other • NumPy • Scikit-Learn – • Pandas
appropriate for web file knowledge • SciPy the main • Database
questions and scraping • NumPy • Pandas machine • Matplotlib
know the • CSV or other • Pandas • Matplotlib learning library • PPT
business file knowledge • Database • CSV or other • CSV or other
• Domain • NumPy • SciPy file knowledge file knowledge
knowledge • Pandas • NumPy
• Passion for • Database • Pandas
data • Database
• Analytical • SciPy
approach
Knowledge Check
Knowledge
Check What is the goal of data acquisition?
Select all that apply.
1

a. Collect data from various data sources

b. Answer business questions through graphics

c. Collect web server logs

d. Scrape the web through web APIs

 Knowledge
Check What is the goal of data acquisition?
Select all that apply.
1

a. Collect data from various data sources

b. Answer business questions through graphics

c. Collect web server logs

d. Scrape the web through web APIs

The correct answer is a, c, d

Data acquisition is a process to collect data from various data sources, such as RDBMS, No SQL databases, web server logs and
also scrape the web through web APIs.
Knowledge
Check What is Exploratory data analysis technique?
Select all that apply.
2

a. Analysis of data using quantitative techniques

b. Conducted only on a small subset of data

c. Analysis of data using graphical techniques

d. Suggests models that best fit the data

 Knowledge
Check What is Exploratory data analysis technique?
Select all that apply.
2

a. Analysis of data using quantitative techniques

b. Conducted only on a small subset of data

c. Analysis of data using graphical techniques

d. Suggests models that best fit the data

The correct answer is a, c, d

Most EDA techniques are graphical in nature with a few quantitative techniques and also suggest models that best fit the data.
They use almost the entire data with minimum and no assumptions.
Knowledge
Check Which plotting technique is used for continuous data?
Select all that apply.
3

a. Regression plot

b. Line chart

c. Histogram

d. Heat map
 Knowledge
Check Which plotting technique is used for continuous data?
Select all that apply.
3

a. Regression plot

b. Line chart

c. Histogram

d. Heat map

The correct answer is b, c

Line charts and histograms are used to plot continuous data.

Knowledge
Check
Which Python library is the main machine learning library?
4

a. Pandas

b. Matplotlib

c. Scikit-learn

d. NumPy
 Knowledge
Check
Which Python library is the main machine learning library?
4

a. Pandas

b. Matplotlib

c. Scikit-learn

d. NumPy

The correct answer is c

SciKit-learn is the main machine learning library in Python.

Knowledge
Check Which of the following includes data transformation, merging, aggregation, group by operation,
and reshaping?
5

a. Data acquisition

b. Data visualization

c. Data wrangling

d. Machine learning
 Knowledge
Check Which of the following includes data transformation, merging, aggregation, group by operation,
and reshaping?
5

a. Data acquisition

b. Data visualization

c. Data wrangling

d. Machine learning

The correct answer is c

Data wrangling includes data transformation, merging, aggregation, group by operation, and reshaping.
Knowledge
Check
Which measure of central tendency is used to catch outliers in the data?
6

a. Mean

b. Median

c. Mode

d. Variance
 Knowledge
Check
Which measure of central tendency is used to catch outliers in the data?
6

a. Mean

b. Median

c. Mode

d. Variance

The correct answer is b

Median is the exact middle value and most suitable to catch outliers.
Knowledge
Check
In hypothesis testing, the proposed model is built on:
6

a. Entire dataset

b. Test dataset

c. Small subset

d. Training dataset
 Knowledge
Check
In hypothesis testing, the proposed model is built on:
6

a. Entire dataset

b. Test dataset

c. Small subset

d. Training dataset

The correct answer is d

The proposed model is built on the training dataset in hypothesis testing.

Knowledge
Check
Beautiful soup library is used for _____.
7

a. Data wrangling

b. Web scraping

c. Plotting

d. Machine learning
 Knowledge
Check
Beautiful soup library is used for _____.
7

a. Data wrangling

b. Web scraping

c. Plotting

d. Machine learning

The correct answer is b

BeautifulSoup is used for web scraping and mainly used in the data acquisition phase.
 Key Takeaways

You are now able to:

Describe Data Analytics process and its steps

List the skills and tools required for data analysis

Understand the challenges of the Data Analytics process

Explain Exploratory data analysis technique

Illustrate data visualization techniques

Describe Hypothesis testing

Thank You