Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
89 views

Python, Data Science, and Unsupervised Learning

The document contains a disclaimer stating that any opinions or information presented do not necessarily reflect the views of blibli.com and blibli.com assumes no responsibility for the accuracy of the information. Hendri Karisma is introduced as a senior research engineer at blibli.com working on fraud detection and recommendation systems using machine learning.

Uploaded by

Hendri Karisma
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
89 views

Python, Data Science, and Unsupervised Learning

The document contains a disclaimer stating that any opinions or information presented do not necessarily reflect the views of blibli.com and blibli.com assumes no responsibility for the accuracy of the information. Hendri Karisma is introduced as a senior research engineer at blibli.com working on fraud detection and recommendation systems using machine learning.

Uploaded by

Hendri Karisma
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 32

Disclaimer

Presentations are intended for educational


purposes only and do not replace
independent professional judgment.
Statements of fact and opinions expressed
are those of the participants individually
and don’t necessarily reflect those of
blibli.com.

Blibli.com does not endorse or approve,


and assumes no responsibility for, the
content, accuracy or completeness of the
information presented.
Python, data science, and
unsupervised learning
Hendri Karisma
hendri.karisma@gdn-commerce.com / situkangsayur@gmail.com
Hendri Karisma

• Sr. Research and Development


Engineer at blibli.com (PT. Global
Digital Niaga)
• Rnd Team for Machine Learning
• Working for Fraud Detection System.
Current working in dynamic
recommendation system project.
Definition of Informatics

“Automation of Information” –
Prof. Dr. Ing. Iping Supriana
Solution Approachment
• Analytical (Exact)
Example :
– analytics solution :
– Numerical solution

– Error = | 7.25 – 22/3| = |7.25-7.33|=0.08333


• Numerical (Aprox)
– Is numerical methods just about ML method that we know in the
book?
– Newton raphson, Gauss Elimination, Gauss-Jordan, Jacobi method,
Gauss-Seidel, Lagrange, Newton Gregory, Richardson Interpolation,
etc.
Machine Learning Definition

“A computer program is said to learn


from experience E with respect to
some class of tasks T and performance
measure P, if its performance at tasks
in T, as measured by P, improves with
experience E.” – Prof. Tom Mitchel
How it works
Machine Learning Perspective

● Information Theory (Decission Tree :


ID-Tree, C4.5, etc)
● Probability (Bayessian : Naive

Bayes, Belief Network, etc)


● Graphical Model (Belief network, HMM,

CRF, Neural Network, etc)


● Numerical Method or Regression

(Stochastic Gradient Descent/Ascent:


Linear Regression, Multiple Linear
Regression, Neural Network, E-M
Algorithm, HMM)
Machine Learning

• Supervised
• Unsupervised
• Reinforcement Learning
• Semi-Supervised
• Deep Learning
The four layer of data mining
Tools/libs in python

● Numpy ● Other Tech (to


● Scipy support ML) :
● Pandas – Apache Kafka
● Scikit-learn – Apache Spark
● Matplotlib – Db : mongo, postgre
● seaborn – elasticsearch
● Tensorflow – CUDA/OpenCL

*pydata.org
*anaconda
Numpy, scipy, padas, and sk-learn

● Numpy & scipy: Arrays, Indexing, Slicing,


and Iterating, Reshaping, Shallow vs deep
copy, Broadcasting, Indexing (advanced),
Matrices, Matrix decompositions, Scipy on
top numpy
● Pandas : Reading data, Selecting columns
and rows, Filtering, Vectorized string
operations, Missing values, Handling time,
Time series, On top numpy.
● SK-Learn : Feature extraction, Classification,
Regression, Clustering, Dimension reduction,
Model selection
What we do in blibli using python

● Data flow
● Data pooling
● Data preprocessing
● Machine Learning Service/app
Our system that using python for ML

● Personalize recommendation system


● Data engineering (especially the
data flow for ML engine)
● Machine learning engine
● Fraud detection experiments
EM Algorithms

Repeat until convergence{

}
What??
EM Algorithms

There are 3 keys that (as far as I know) almost


always used in EM-Algorithm :
● Data Distribution
● Maximum Likelihood Estimation (MLE)
● Estimation-Maximization (EM)

*Today we will use the Gaussian distribution for


sample case
EM Algorithms

The algorithm has 2 main steps just like the name


of the algorithm:
– Expectation :

– Maximization:

*repeat until get maximum likelihood :


Gaussian Distribution
Gaussian Distribution
Gaussian Multivariate

● Gaussian Distribution :

● Gaussian Distribution Multivariate :


Mixture Gaussian
EM-Algorithm for Mixture Gaussian

● Expectation :

● Maximization :

*Log likelihood :
Fraud – without target class/labels

● These are anomalous data


● Anomaly data usually have one or
some small group of data
● A lot of features without labels
------------------------------------------
● We need unsupervised algorithm
(EM-Algorithm)
Case Anomaly Detection

● Credit Card data with fraudulant data.


Case Anomaly Detection
Case Anomaly Detection
Case Anomaly Detection
Case Anomaly Detection
Problem Performance
Distributed System/Scale Out

Supervisor/Service
Using python

Python script

Presistence

Computation
THANK YOU
Any question?

*we are hiring*

You might also like