Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
59 views

Datamining Lecture6

This document provides an overview of linear regression, logistic regression, neural networks, and deep learning. It discusses regression problems for predicting continuous values, linear regression with one or multiple inputs, calculating error terms, and limitations of linear models. Logistic regression is introduced as an alternative for classification problems. Neural networks are described as collections of perceptrons that can perform multi-class classification. Deep learning neural networks are distinguished by having more hidden layers.

Uploaded by

Vi Le
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
59 views

Datamining Lecture6

This document provides an overview of linear regression, logistic regression, neural networks, and deep learning. It discusses regression problems for predicting continuous values, linear regression with one or multiple inputs, calculating error terms, and limitations of linear models. Logistic regression is introduced as an alternative for classification problems. Neural networks are described as collections of perceptrons that can perform multi-class classification. Deep learning neural networks are distinguished by having more hidden layers.

Uploaded by

Vi Le
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 41

DATA MINING

LECTURE 6
Linear Regression
Logistic Regression
Neutral Networks
A Little Bit of Deep Learning

CSC177
Dr. Victor Chen
Regression Problem
• The problem of predicting continuous values is
called regression problem
• General approach: find a continuous function that
models the continuous points.
Linear Regression with one input

A simple regression has the coefficient β and the


constant α. The equation is then:

y=α+β*x

where α is y-intercept and β is slope


Multiple Linear Regression

Linear Regression with more than one


input

A multiple regression is in the following form:

y = α + β1 * X1 + β2 * X2 + … + βk * Xk

where k is the number of variables, or parameters.


Linear regression for 2D data
• Given a dataset of the
form
(𝑥1 , 𝑦1 ) , … , (𝑥𝑛 , 𝑦𝑛 ) ,
find a linear continuous
function that minimizes
the Sum of Squares of
Error (SSE)
Linear Regression

Dependent variable (y)


y’ = b0 + b1X
є

b1 = slope
= ∆y/ ∆x
b0 (y intercept)

Independent variable (x)


Regression

Sum of Squares of Error (SSE)


Sum of squared errors
෍ 𝑦𝑖′ − 𝑦𝑖 2
Dependent variable 𝑖

Independent variable (x)

Sum of Squares of Error (SSE) is the sum of all the squared errors.
Sum of Squares of Regression (SSR)

Dependent variable

Data mean

Independent variable (x)

Sum of Squares of Regression (SSR) is the sum of the squared differences


between each prediction and the mean of data.
Sum of Squares of Total (SST)
SST = SSE + SSR
The Coefficient of Determination

R-Squared score
• R-Squared score can be used as a single summary
number to measure the quality of linear regression
model
• The value of R2 can range between 0 and 1.
• The higher R2, the more accurate the regression
model is.
Nonlinear Regression
Nonlinear functions can also be fit as
regressions

Any nonlinear continuous functions can also be fit as regressions,


including power, Logarithmic, Exponential, and Logistic.
12

A Real Example: Social Network Analysis


High school dating

High school friendship


13

A Real Example: Social Network Analysis


• In link prediction, given any two arbitrary users,
we aim to answer:
• What is the likelihood that a particular link (relationship)
exists between them?
14

Markov Networks v.s. Bayesian


Networks

• Haiquan Chen, Wei-Shinn Ku, Haixun Wang, Liang Tang, Min-Te Sun:
Scaling Up Markov Logic Probabilistic Inference for Social Graphs. IEEE Trans. Knowl. Data
Eng. 29(2): 433-445 (2017)
15

Experimental validation
Now Logistic Regression…
Linear Regression Doesn’t Work
• A linear function/regression is not good
• It may produce probabilities beyond [0, 1]

Note: APACHE II is one of several ICU scoring systems.


Logistic Regression
Logistic (Sigmoid) function maps any input x between [0, 1]
1
𝑓 𝑡 =
1 + 𝑒 −𝑥

Logistic regression is a linear


regression on the Sigmoid
function

1
𝑃 𝐶𝑥 =
1 + 𝑒 −(α+β⋅𝒙)
Q: What is the logistic regression model for more than one dimension?
Logistic Regression
• For 2-class problem, the probability threshold is
set to 0.5 so
• If the predicted probability >= 0.5, predict “y = 1”,
• If the predicted probability > 0.5, predict “y = 0”,
How β affects the model
Compare Two Models In One Dimension

Q: What x value makes the model output a probability of 0.5?


Logistic regression in 2D space

Predict Iris flower


species based on
sepal length and
sepal width only

Coefficients

𝛽1 = −1.9
𝛽2 = −0.4
𝛼 = 13.04
Estimating the coefficients
• Use gradient descent algorithm to find the near-
optimal coefficients for linear/logistic regression
Gradient descent for two parameters
Gradient descent for two parameters
Gradient descent implementation
Sklearn implementation
• http://scikit-
learn.org/stable/modules/generated/sklearn.linear_model.
LinearRegression.html

• http://scikit-
learn.org/stable/modules/generated/sklearn.linear_model.
LogisticRegression.html
Logistic/Linear Regression Advantages
• Linear regression produces a continuous output.
• Logistic regression produces class membership
with predicted probability .
• The coefficients can be used for understanding
the feature importance.
• Works for relatively large datasets
Neutral networks
• Logistic regression can be considered as the
simplest form of a neutral network, which is a
collection of perceptron.
• Perceptron is seen as an analogy to a biological
neuron.
Perceptron (Neuron)
2-class classification with one neuron
2-class classification with two neurons,
one for each class
Putting multiple neurons in parallel we can predict multiple classes
sigmoid vs softmax
• sigmoid function is • softmax function is
used for the two-class used for the multiclass
logistic regression logistic regression
Softmax implementation
What is softmax([1, 2, 3])

e1
• y1 = 1 2 3 = 0.09
e +e +e
e2
• y1 = 1 2 3 = 0.24
e +e +e
e3
• y3 = 1 2 3 = 0.67
e +e +e

Output:

• [0.09, 0.24, 0.67]


3-class classification with three neurons
Putting multiple neurons in parallel we can predict multiple classes
Neutral Network = Multi Layer Perceptron
• Multiple classes can be predicted by putting many
neurons in parallel.
Sklearn implementation
• http://scikit-
learn.org/stable/modules/neural_networks_supervised.html#
Deep learning
Neural networks vs deep learning neural
networks
• "Normal" neural networks • Deep learning neural
usually have one to two network have more hidden
hidden layers and are layers and can be used for
used for SUPERVISED both UNSUPERVISED
classification. and SUPERVISED
learning tasks.
An example deep learning neural network

You might also like