How to validate a model?
What is a best model ?
Types of data
Types of errors
The problem of over fitting
The problem of under fitting
Bias Variance Tradeoff
Cross validation
K-Fold Cross validation
Boot strap Cross validation
This document discusses various evaluation measures used in machine learning, including accuracy, precision, recall, F1 score, and AUROC for classification problems. For regression problems, the output is continuous and no additional treatment is needed. Classification accuracy is defined as the number of correct predictions divided by the total predictions. The confusion matrix is used to calculate true positives, false positives, etc. Precision measures correct positive predictions, while recall measures all positive predictions. The F1 score balances precision and recall for imbalanced data. AUROC plots the true positive rate against the false positive rate.
The document discusses machine learning classification using the MNIST dataset of handwritten digits. It begins by defining classification and providing examples. It then describes the MNIST dataset and how it is fetched in scikit-learn. The document outlines the steps of classification which include dividing the data into training and test sets, training a classifier on the training set, testing it on the test set, and evaluating performance. It specifically trains a stochastic gradient descent (SGD) classifier on the MNIST data. The performance is evaluated using cross validation accuracy, confusion matrix, and metrics like precision and recall.
The document discusses the random forest algorithm. It introduces random forest as a supervised classification algorithm that builds multiple decision trees and merges them to provide a more accurate and stable prediction. It then provides an example pseudocode that randomly selects features to calculate the best split points to build decision trees, repeating the process to create a forest of trees. The document notes key advantages of random forest are that it avoids overfitting and can be used for both classification and regression tasks.
Introduction to Maximum Likelihood EstimatorAmir Al-Ansary
This document provides an overview of maximum likelihood estimation (MLE). It discusses key concepts like probability models, parameters, and the likelihood function. MLE aims to find the parameter values that make the observed data most likely. This can be done analytically by taking derivatives or numerically using optimization algorithms. Practical considerations like removing constants and using the log-likelihood are also covered. The document concludes by introducing the likelihood ratio test for comparing nested models.
This Logistic Regression Presentation will help you understand how a Logistic Regression algorithm works in Machine Learning. In this tutorial video, you will learn what is Supervised Learning, what is Classification problem and some associated algorithms, what is Logistic Regression, how it works with simple examples, the maths behind Logistic Regression, how it is different from Linear Regression and Logistic Regression applications. At the end, you will also see an interesting demo in Python on how to predict the number present in an image using Logistic Regression.
Below topics are covered in this Machine Learning Algorithms Presentation:
1. What is supervised learning?
2. What is classification? what are some of its solutions?
3. What is logistic regression?
4. Comparing linear and logistic regression
5. Logistic regression applications
6. Use case - Predicting the number in an image
What is Machine Learning: Machine Learning is an application of Artificial Intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed.
- - - - - - - -
About Simplilearn Machine Learning course:
A form of artificial intelligence, Machine Learning is revolutionizing the world of computing as well as all people’s digital interactions. Machine Learning powers such innovative automated technologies as recommendation engines, facial recognition, fraud protection and even self-driving cars.This Machine Learning course prepares engineers, data scientists and other professionals with knowledge and hands-on skills required for certification and job competency in Machine Learning.
- - - - - - -
Why learn Machine Learning?
Machine Learning is taking over the world- and with that, there is a growing need among companies for professionals to know the ins and outs of Machine Learning
The Machine Learning market size is expected to grow from USD 1.03 Billion in 2016 to USD 8.81 Billion by 2022, at a Compound Annual Growth Rate (CAGR) of 44.1% during the forecast period.
- - - - - -
What skills will you learn from this Machine Learning course?
By the end of this Machine Learning course, you will be able to:
1. Master the concepts of supervised, unsupervised and reinforcement learning concepts and modeling.
2. Gain practical mastery over principles, algorithms, and applications of Machine Learning through a hands-on approach which includes working on 28 projects and one capstone project.
3. Acquire thorough knowledge of the mathematical and heuristic aspects of Machine Learning.
4. Understand the concepts and operation of support vector machines, kernel SVM, naive bayes, decision tree classifier, random forest classifier, logistic regression, K-nearest neighbors, K-means clustering and more.
5. Be able to model a wide variety of robust Machine Learning algorithms including deep learning, clustering, and recommendation systems
- - - - - - -
Principal Component Analysis, or PCA, is a factual method that permits you to sum up the data contained in enormous information tables by methods for a littler arrangement of "synopsis files" that can be all the more handily envisioned and broke down.
Logistic regression is a machine learning classification algorithm that predicts the probability of a categorical dependent variable. It models the probability of the dependent variable being in one of two possible categories, as a function of the independent variables. The model transforms the linear combination of the independent variables using the logistic sigmoid function to output a probability between 0 and 1. Logistic regression is optimized using maximum likelihood estimation to find the coefficients that maximize the probability of the observed outcomes in the training data. Like linear regression, it makes assumptions about the data being binary classified with no noise or highly correlated independent variables.
Data Analysis: Evaluation Metrics for Supervised Learning Models of Machine L...Md. Main Uddin Rony
This document discusses various machine learning evaluation metrics for supervised learning models. It covers classification, regression, and ranking metrics. For classification, it describes accuracy, confusion matrix, log-loss, and AUC. For regression, it discusses RMSE and quantiles of errors. For ranking, it explains precision-recall, precision-recall curves, F1 score, and NDCG. The document provides examples and visualizations to illustrate how these metrics are calculated and used to evaluate model performance.
KNN algorithm is one of the simplest classification algorithm and it is one of the most used learning algorithms. KNN is a non-parametric, lazy learning algorithm. Its purpose is to use a database in which the data points are separated into several classes to predict the classification of a new sample point.
In machine learning, support vector machines (SVMs, also support vector networks[1]) are supervised learning models with associated learning algorithms that analyze data and recognize patterns, used for classification and regression analysis. The basic SVM takes a set of input data and predicts, for each given input, which of two possible classes forms the output, making it a non-probabilistic binary linear classifier.
The document discusses exploratory data analysis and provides examples of how it can be used. It summarizes two case studies: one where an energy utility detected billing fraud by analyzing meter reading patterns, and another where month of birth was found to correlate with exam scores for students in Tamil Nadu. The document then outlines the exploratory data analysis process and provides a high-level overview of U.S. and Indian birth date patterns identified through analysis of large datasets.
Linear Regression vs Logistic Regression | EdurekaEdureka!
YouTube: https://youtu.be/OCwZyYH14uw
** Data Science Certification using R: https://www.edureka.co/data-science **
This Edureka PPT on Linear Regression Vs Logistic Regression covers the basic concepts of linear and logistic models. The following topics are covered in this session:
Types of Machine Learning
Regression Vs Classification
What is Linear Regression?
What is Logistic Regression?
Linear Regression Use Case
Logistic Regression Use Case
Linear Regression Vs Logistic Regression
Blog Series: http://bit.ly/data-science-blogs
Data Science Training Playlist: http://bit.ly/data-science-playlist
Follow us to never miss an update in the future.
YouTube: https://www.youtube.com/user/edurekaIN
Instagram: https://www.instagram.com/edureka_learning/
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
This document discusses association rule mining. Association rule mining finds frequent patterns, associations, correlations, or causal structures among items in transaction databases. The Apriori algorithm is commonly used to find frequent itemsets and generate association rules. It works by iteratively joining frequent itemsets from the previous pass to generate candidates, and then pruning the candidates that have infrequent subsets. Various techniques can improve the efficiency of Apriori, such as hashing to count itemsets and pruning transactions that don't contain frequent itemsets. Alternative approaches like FP-growth compress the database into a tree structure to avoid costly scans and candidate generation. The document also discusses mining multilevel, multidimensional, and quantitative association rules.
Module 4: Model Selection and EvaluationSara Hooker
Delta Analytics is a 501(c)3 non-profit in the Bay Area. We believe that data is powerful, and that anybody should be able to harness it for change. Our teaching fellows partner with schools and organizations worldwide to work with students excited about the power of data to do good.
Welcome to the course! These modules will teach you the fundamental building blocks and the theory necessary to be a responsible machine learning practitioner in your own community. Each module focuses on accessible examples designed to teach you about good practices and the powerful (yet surprisingly simple) algorithms we use to model data.
To learn more about our mission or provide feedback, take a look at www.deltanalytics.org.
Machine Learning - Accuracy and Confusion MatrixAndrew Ferlitsch
Abstract: This PDSG workshop introduces basic concepts on measuring accuracy of your trained model. Concepts covered are loss functions and confusion matrices.
Level: Fundamental
Requirements: No prior programming or statistics knowledge required.
Logistic regression in Machine LearningKuppusamy P
Logistic regression is a predictive analysis algorithm that can be used for classification problems. It estimates the probabilities of different classes using the logistic function, which outputs values between 0 and 1. Logistic regression transforms its output using the sigmoid function to return a probability value. It is used for problems like email spam detection, fraud detection, and tumor classification. The independent variables should be independent of each other and the dependent variable must be categorical. Gradient descent is used to minimize the loss function and optimize the model parameters during training.
The document discusses hyperparameters and hyperparameter tuning in deep learning models. It defines hyperparameters as parameters that govern how the model parameters (weights and biases) are determined during training, in contrast to model parameters which are learned from the training data. Important hyperparameters include the learning rate, number of layers and units, and activation functions. The goal of training is for the model to perform optimally on unseen test data. Model selection, such as through cross-validation, is used to select the optimal hyperparameters. Training, validation, and test sets are also discussed, with the validation set used for model selection and the test set providing an unbiased evaluation of the fully trained model.
A small introduction to some techniques of BBT as Boundary Value, Equivalence Class, Decision Table. According to this presentation, you can know how to apply them in the real world.
Performance metrics are used to evaluate machine learning algorithms and models. Key methods include confusion matrix, accuracy, precision, recall, specificity, and F1 score. The confusion matrix is a table that allows visualization of model performance, while accuracy measures correct predictions over total predictions. Precision focuses on avoiding false positives and recall focuses on avoiding false negatives. The F1 score calculates the harmonic mean of precision and recall to provide a single combined metric. These metrics help select the best performing algorithm and optimize model performance.
This document discusses evaluating machine learning model performance. It covers classification evaluation metrics like accuracy, precision, recall, F1 score, and confusion matrices. It also discusses regression metrics like MAE, MSE, and RMSE. The document discusses techniques for dealing with class imbalance like oversampling and undersampling. It provides examples of evaluating models and interpreting results based on these various performance metrics.
This document summarizes a presentation on model evaluation given at the 4th annual Valencian Summer School in Machine Learning. It discusses the importance of evaluating models to understand how well they will perform on new data and identify mistakes. Various evaluation metrics are introduced like accuracy, precision, recall, F1 score, and Phi coefficient. The dangers of evaluating on training data are explained, and techniques like train-test splits and cross-validation are recommended to get less optimistic evaluations. Regression metrics like MAE, MSE, and R-squared error are also covered. Different evaluation techniques for specific problem types like imbalanced classification, time series forecasting, and model selection are discussed.
Ways to evaluate a machine learning model’s performanceMala Deep Upadhaya
Some of the ways to evaluate a machine learning model’s performance.
In Summary:
Confusion matrix: Representation of the True positives (TP), False positives (FP), True negatives (TN), False negatives (FN)in a matrix format.
Accuracy: Worse happens when classes are imbalanced.
Precision: Find the answer of How much the model is right when it says it is right!
Recall: Find the answer of How many extra right ones, the model missed when it showed the right ones!
Specificity: Like Recall but the shift is on the negative instances.
F1 score: Is the harmonic mean of precision and recall so the higher the F1 score, the better.
Precision-Recall or PR curve: Curve between precision and recall for various threshold values.
ROC curve: Graph is plotted against TPR and FPR for various threshold values.
Confusion matrix and classification evaluation metricsMinesh A. Jethva
This document discusses classification evaluation metrics and their limitations. It introduces the confusion matrix and metrics calculated from it such as precision, recall, F1-score, and accuracy. The summary highlights that these metrics can be "hacked" and misleading. More robust alternatives like balanced accuracy and MCC are presented that account for true negatives and are not as affected by class imbalance. Comprehensive reporting of multiple metrics from different perspectives is recommended for fully understanding a model's performance.
“You can download this product from SlideTeam.net”
Our professionally designed PowerPoint presentation is sure to impress executives, inspire team members and other audience. With a complete set of thirty one slides, this PPT is the most comprehensive summary of Vendor Audit Power Point Presentation Slides you could have asked for. The content is extensively researched and designs are professional. Our PPT designers have worked tirelessly to craft this deck using beautiful PowerPoint templates, graphics, diagrams and icons. On top of that, the deck is 100 persent editable in PowerPoint so that you can enter your text in the placeholders, change colors if you wish to, and present in the shortest time possible. https://bit.ly/3F0XZfB
Being Right Starts By Knowing You're WrongData Con LA
Data Con LA 2020
Description
The recent proliferation of predictive analytics within companies is of limited benefit unless these companies learn to measure, understand, and embrace a critical concept: error. There is no such thing as a perfect predictive model and all tools using any sort of predictive model will have error. Despite being relatively easy to implement and understand, consistent error measurement continues to be underutilized or even completely avoided. In this session we will discuss
*Why embracing error is so valuable to companies.
*We will then review basic ways to measure error in commonly used models and in data source systems such as CRMs and ERPs.
*Most importantly, we will review some ways to approach company leadership with the concept of error.
Speaker
Ryan Johnson, GoGuardian, Director of Science and Analytics
Our professionally designed PowerPoint presentation is sure to impress executives, inspire team members and other audience. With a complete set of thirtyone slides, this PPT is the most comprehensive summary of Vendor Audit Power Point Presentation Slides you could have asked for. The content is extensively researched and designs are professional. Our PPT designers have worked tirelessly to craft this deck using beautiful PowerPoint templates, graphics, diagrams and icons. On top of that, the deck is 100 persent editable in PowerPoint so that you can enter your text in the placeholders, change colors if you wish to, and present in the shortest time possible.
Sensitivity and specificity are important metrics for evaluating predictive models. Sensitivity refers to the probability that a model correctly predicts a positive outcome, while specificity refers to the probability that it correctly predicts a negative outcome. There is often a tradeoff between the two - more stringent models will have higher specificity but lower sensitivity, while more relaxed models will be the opposite. It is important to consider an application's goals to determine whether prioritizing sensitivity or specificity would be more effective. Various statistical ratios like true and false positive/negative rates can provide further insight into a model's performance.
This document discusses various machine learning model validation techniques and ensemble methods such as bagging and boosting. It defines key concepts like overfitting, underfitting, bias-variance tradeoff, and different validation metrics. Cross validation techniques like k-fold and bootstrap are explained as ways to estimate model performance on unseen data. Bagging creates multiple models on resampled data and averages their predictions to reduce variance. Boosting iteratively adjusts weights of misclassified observations to build strong models, but risks overfitting. Gradient boosting and XGBoost are powerful ensemble methods.
This document discusses techniques for evaluating and improving classifiers. It begins by explaining how to evaluate a classifier's accuracy using metrics like accuracy, precision, recall, and F-measure. It introduces the confusion matrix and shows how different parts of the matrix relate to these metrics. The document then discusses issues like overfitting, underfitting, bias and variance that can impact a classifier's performance. It explains that the goal is to balance bias and variance to minimize total error and achieve optimal classification.
The document provides guidance on running effective black hat sessions to analyze competitors.
Key points covered include:
1) Thorough preparation is essential, including developing a win-loss analysis and bidder profile documents.
2) The session should have the right mix of participants and be structured to avoid groupthink, with multiple teams role-playing competitors.
3) Facilitation techniques like Porter's four corners model, organizational culture frameworks, and predictive markets can generate insights beyond typical "MBA thinking".
4) An action plan and follow up process is needed to measure predictions and continuously improve competitive intelligence over time.
Nobody likes false negatives. When your Nagios probes fail to detect a problem, it can hurt your sales, your reputation, and even your ego (especially your ego). The solution: tune the thresholds. Right? You can handle a couple spurious late-night pages if it means you’ll reliably detect real failures.
I will argue that – while easy – exchanging false negatives for false positives does more harm than good. Borrowing the medical concepts of specificity and sensitivity, I’ll show how deceptive this tradeoff can be. I’ll also make the case that putting in the extra effort to minimize both types of falsehoods is necessary and healthy. When the alarm goes off, you shouldn’t have to spend precious minutes sniffing for smoke.
This document summarizes key concepts from a presentation on A/B testing fundamentals. It discusses:
1. The different possible outcomes of A/B tests and how they relate to concepts like true positives, false positives, etc.
2. The difference between false positive rate and false discovery rate. False positive rate considers the probability of a false positive from a single test, while false discovery rate accounts for running multiple tests.
3. How to balance factors like error rates, effect size detection, and test duration by making tradeoffs between them, such as running tests longer to reduce error rates or detect smaller effects.
Evaluation metrics for binary classification - the ultimate guideneptune.ml
Presentation from PyData Warsaw 2019 by Jakub Czakon.
Choosing a proper metric is a crucial yet difficult part of the machine learning project. In this talk, you will learn about a number of common and lesser-known metrics and performance charts as well as typical decisions when it comes to choosing one for your project. Hopefully, with all that knowledge you will be fully equipped to deal with metric-related problems in your future projects!
This document discusses best practices for interpreting A/B test results and making decisions based on those results. It cautions against relying solely on significance values and p-values, noting that these only indicate the likelihood of the data given the null hypothesis, not the likelihood of the null hypothesis being true. It emphasizes considering confidence intervals, choosing the right metrics like profit or revenue, and weighing the risks of false positives versus missed opportunities when deciding whether to roll out a change. The key takeaways are to incorporate risk into test design, focus on the real goal of increasing profit, and recognize that the stakes are generally lower for online experiments than medical trials.
Ever wondered about the full form of Chat GPT?🤔 It stands for Chat Generative Pre-Trained Transformer. For those diving into the world of Transformers, I've been using this PPT during my lectures📚. Thought it might be handy for some of you too! Check it out and let me know what you think!🌟
The document provides notes on neural networks and regularization from a data science training course. It discusses issues like overfitting when neural networks have too many hidden layers. Regularization helps address overfitting by adding a penalty term to the cost function for high weights, effectively reducing the impact of weights. This keeps complex models while preventing overfitting. The document also covers activation functions like sigmoid, tanh, and ReLU, noting advantages of tanh and ReLU over sigmoid for addressing vanishing gradients and computational efficiency. Code examples demonstrate applying regularization and comparing models.
This document provides an overview of gradient boosting methods. It discusses that boosting is an ensemble method that builds models sequentially by focusing on misclassified examples from previous models. The gradient boosting algorithm updates weights based on misclassification rates and gradients. Key parameters for gradient boosting models include the number of trees, interaction depth, minimum observations per node, shrinkage, bag fraction, and train fraction. Tuning these hyperparameters is important for achieving the right balance of underfitting and overfitting.
This document provides an overview of neural networks in R. It begins with recapping logistic regression and decision boundaries. It then discusses how neural networks allow for non-linear decision boundaries through the use of intermediate outputs and multiple logistic regression models. Code examples are provided to demonstrate building neural networks with intermediate outputs to classify data with non-linear decision boundaries.
The document discusses decision trees, which are a type of predictive modeling that can be used for segmentation. It provides examples of how to segment a population of customers into subgroups based on attributes like employment status and income. The key aspects of decision trees covered include how they are constructed from a root node down to leaf nodes, different algorithms for building decision trees, measures for determining the best attributes to split on like information gain, and techniques for validating and pruning trees to avoid overfitting.
This document provides a step-by-step guide to learning R. It begins with the basics of R, including downloading and installing R and R Studio, understanding the R environment and basic operations. It then covers R packages, vectors, data frames, scripts, and functions. The second section discusses data handling in R, including importing data from external files like CSV and SAS files, working with datasets, creating new variables, data manipulations, sorting, removing duplicates, and exporting data. The document is intended to guide users through the essential skills needed to work with data in R.
1. The document outlines the steps in building a credit risk model, including defining the objective, applying exclusions, determining the observation and performance windows, defining "bad" accounts, performing segmentation, selecting variables, building the regression model, and validating and recalibrating the model.
2. Segmentation involves dividing the population into subgroups for separate modeling in order to better separate "good" and "bad" accounts. Common segmentation variables include product type, account tenure, credit file thickness, and portfolio type.
3. Determining the "bad" definition and performance window involves analysis of account roll rates and waterfalls to identify what constitutes a "bad"
Introduction to Analytics
Introduction to SAS
Introduction to Satistics
Introduction to Predictive Modeling
Introduction to Forecasting
Introduction to Bigdata
This document provides a step-by-step guide to learning SAS. It begins with an introduction to SAS and its windowing environment. Next, it discusses SAS datasets and variables, including importing data into SAS and basic procedures and functions. The document then covers combining datasets in SAS before concluding with next steps. It assumes some basic database and analytics knowledge and provides disclaimers about its intended use as a high-level summary.
This case study tests two hypotheses about customer satisfaction scores: 1) that the average satisfaction score for Samsunge customers is at least 75%, and 2) that the average satisfaction scores for Samsunge and Appleo customers are the same. Data on satisfaction scores for 100 customers from each company was provided. The approach is to use SAS to analyze the data, performing appropriate statistical tests on the Samsunge scores alone to test the first hypothesis, and a mean comparison test to analyze both companies' scores and test the second hypothesis. The results of the statistical tests will determine whether the null hypotheses can be accepted or rejected.
FiberBits, an internet service provider, has seen a 42% customer attrition rate over the last 3 years and wants to build a predictive model to identify customers most likely to quit in the next 2 years. The model will be built using historical data on nearly 10,000 customers containing variables like income, time as a customer, complaints, billing amounts, and technical issues. Customers identified as higher risk will be offered incentives like vouchers to encourage them to stay.
Step-1 Tableau Introduction
Step-2 Connecting to Data
Step-3 Building basic views
Step-4 Data manipulations and Calculated fields
Step-5 Tableau Dashboards
Step-6 Advanced Data Options
Step-7 Advanced graph Options
This document provides an introduction to machine learning, including:
- It discusses how the human brain learns to classify images and how machine learning systems are programmed to perform similar tasks.
- It provides an example of image classification using machine learning and discusses how machines are trained on sample data and then used to classify new queries.
- It outlines some common applications of machine learning in areas like banking, biomedicine, and computer/internet applications. It also discusses popular machine learning algorithms like Bayes networks, artificial neural networks, PCA, SVM classification, and K-means clustering.
List of data sets and data set sources
Sample data sets for machine learning
Data sets for predictive modeling and visualizations
Economic and Social Data sets
Business and Financial datasets
The document provides an introduction to the concepts of big data and how it can be analyzed. It discusses how traditional tools cannot handle large data files exceeding gigabytes in size. It then introduces the concepts of distributed computing using MapReduce and the Hadoop framework. Hadoop makes it possible to easily store and process very large datasets across a cluster of commodity servers. It also discusses programming interfaces like Hive and Pig that simplify writing MapReduce programs without needing to use Java.
The document provides an introduction to the R programming language. It discusses that R is an open-source programming language for statistical analysis and graphics. It can run on Windows, Unix and MacOS. The document then covers downloading and installing R and R Studio, the R workspace, basics of R syntax like naming conventions and assignments, working with data in R including importing, exporting and creating calculated fields, using R packages and functions, and resources for R help and tutorials.
The document provides an overview of cluster analysis techniques. It discusses the need for segmentation to group large populations into meaningful subsets. Common clustering algorithms like k-means are introduced, which assign data points to clusters based on similarity. The document also covers calculating distances between observations, defining the distance between clusters, and interpreting the results of clustering analysis. Real-world applications of segmentation and clustering are mentioned such as market research, credit risk analysis, and operations management.
This document discusses preparing data for analysis. It covers the need for data exploration including validation, sanitization, and treatment of missing values and outliers. The main steps in statistical data analysis are also presented. Specific techniques discussed include calculating frequency counts and descriptive statistics to understand the distribution and characteristics of variables in a loan data set with 250,000 observations. SAS procedures like Proc Freq, Proc Univariate, and Proc Means are demonstrated for exploring the data.
How to Manage Advanced Pricelist in Odoo 17Celine George
Maintaining relationships with customers is important for a business. Customizing prices will help to maintain the relationships with customers. Odoo provides a pricing strategy called pricelists. We can set appropriate prices for the clients. And advanced price rules will help to set prices based on different conditions. This slide will show how to manage advanced pricelists in odoo 17.
Plato and Aristotle's Views on Poetry by V.Jesinthal Maryjessintv
PPT on Plato and Aristotle's Views on Poetry prepared by Mrs.V.Jesinthal Mary, Dept of English and Foreign Languages(EFL),SRMIST Science and Humanities ,Ramapuram,Chennai-600089
A history of Innisfree in Milanville, PennsylvaniaThomasRue2
A history of Innisfree in Milanville, Damascus Township, Wayne County, Pennsylvania. By TOM RUE, July 23, 2023. Innisfree began as "an experiment in democracy," modeled after A.S. Neill's "Summerhill" school in England, "the first libertarian school".
What is the Use of API.onchange in Odoo 17Celine George
The @api.onchange decorator in Odoo is indeed used to trigger a method when a field's value changes. It's commonly used for validating data or triggering actions based on the change of a specific field. When the field value changes, the function decorated with @api.onchange will be called automatically.
How to install python packages from PycharmCeline George
In this slide, let's discuss how to install Python packages from PyCharm. In case we do any customization in our Odoo environment, sometimes it will be necessary to install some additional Python packages. Let’s check how we can do this from PyCharm.
Types of Diode and its working principle.pptxnitugatkal
A diode is a two-terminal polarized electronic component which mainly conducts current in one direction and blocks in other direction.
Its resistance in one direction is low (ideally zero) and high (ideally infinite) resistance in the other direction.
How to Use Serial Numbers to Track Products in Odoo 17 InventoryCeline George
Mainly lots or serial numbers are used for tracking the products. Lots are actually the codes that applied for collection of products. But serial numbers are distinct numbers allocated for a particular product. Lots and serial numbers in the products will help to manage the inventory, to trace the products that reached their expiry date. This slide will show how to use lots and serial numbers to track products in odoo 17 inventory.
Introduction to Literary Criticism 10 (1).pptxjessintv
Introduction to Literary Criticism prepared by Mrs.V.Jesinthal Mary,Asst.Professor,Dept of English and other foreign Languages (EFL), SRMIST Science and Humanities, Ramapuram,
Chennai-600089
How to Configure Field Cleaning Rules in Odoo 17Celine George
In this slide let’s discuss how to configure field cleaning rules in odoo 17. Field Cleaning is used to format the data that we use inside Odoo. Odoo 17's Data Cleaning module offers Field Cleaning Rules to improve data consistency and quality within specific fields of our Odoo records. By using the field cleaning, we can correct the typos, correct the spaces between them and also formats can be corrected.
How to Load Custom Field to POS in Odoo 17 - Odoo 17 SlidesCeline George
This slide explains how to load custom fields you've created into the Odoo 17 Point-of-Sale (POS) interface. This approach involves extending the functionalities of existing POS models (e.g., product.product) to include your custom field.
Multi Language and Language Translation with the Website of Odoo 17Celine George
In this slide, we'll explore the Multi Language and Language Translation features in Odoo 17 Website. We'll show you how to easily set up and manage these powerful tools.
Odoo 17 Project Module : New Features - Odoo 17 SlidesCeline George
The Project Management module undergoes significant enhancements, aimed at providing users with more robust tools for planning, organizing, and executing projects effectively.
Odoo 17 Project Module : New Features - Odoo 17 Slides
Model selection and cross validation techniques
1. Model Selection and Cross
Validation techniques
www.statinfer.com
1
statinfer.com
2. Statinfer.com
Data Science Training and R&D
#647, 100 feet road, Indra Nagar, Bangalore, India
Corporate Training
Online Video Courses
Contact us
info@statinfer.com
+91-080 4851 1820
+91-98867 60678
Follow us on Social
Media for more course
material
Twitter: https://twitter.com/statinfer
Facebook: https://www.facebook.com/statinfer/
Google Plus: https://plus.google.com/u/0/104559910764943078938
LinkedIn: https://www.linkedin.com/company/statinfer-software-
solutions-llp
3. Note
•This presentation is just class notes. This course material is prepared
by statinfer team, as an aid for training sessions.
•The best way to treat this is as a high-level summary; the actual
session went more in depth and contained detailed information and
examples
•Most of this material was written as informal notes, not intended for
publication
•Please send questions/comments/corrections to info@statinfer.com
•Please check our website statinfer.com for latest version of this
document
-Team Statinfer
4. Contents
•How to validate a model?
•What is a best model ?
•Types of data
•Types of errors
•The problem of over fitting
•The problem of under fitting
•Bias Variance Tradeoff
•Cross validation
•K-Fold Cross validation
•Boot strap Cross validation 4
statinfer.com
6. Model Validation
•Checking how good is our model
•It is very important to report the accuracy of the model along with the
final model
•The model validation in regression is done through R square and Adj R-
Square
•Logistic Regression, Decision tree and other classification techniques
have the very similar validation measures.
•Till now we have seen confusion matrix and accuracy. There are many
more validation and model accuracy metrics for classification models
6
statinfer.com
7. Classification-Validation measures
•Confusion matrix, Specificity, Sensitivity
•ROC, AUC
•Kappa, F1 Score
•KS, Gini
•Concordance and discordance
•Chi-Square, Hosmer and Lemeshow Goodness-of-Fit Test
All of them are measuring the model accuracy only. Some metrics work
really well for certain class of problems. Confusion matrix, ROC and
AUC will be sufficient for most of the business problems 7
statinfer.com
9. Classification Table
0(Positive) 1(Negative)
0(Positive)
True positive (TP)
Actual condition is Positive, it is
truly predicted as positive
False Negatives(FN)
Actual condition is Positive, it is
falsely predicted as negative
1(Negative)
False Positives(FP)
Actual condition is Negative, it is
falsely predicted as positive
True Negatives(TN)
Actual condition is Negative, it is
truly predicted as negative
9
Actual Classes
Predicted Classes
• Accuracy=(TP+TN)/(TP+FP+FN+TN)
• Misclassification Rate=(FP+FN)/(TP+FP+FN+TN)
Sensitivity and Specificity are derived from confusion matrix
statinfer.com
10. Sensitivity and Specificity
• Sensitivity : Percentage of positives that are successfully classified as positive
• Specificity : Percentage of negatives that are successfully classified as negatives
0(Positive) 1(Negative)
0(Positive)
True positive (TP)
Actual condition is
Positive, it is truly
predicted as positive
False Negatives(FN)
Actual condition is
Positive, it is falsely
predicted as negative
Sensitivity=
TP/(TP+FN) or TP/
Overall Positives
1(Negative)
False Positives(FP)
Actual condition is
Negative, it is falsely
predicted as positive
True Negatives(TN)
Actual condition is
Negative, it is truly
predicted as negative
Specificity =
TN/(TN+FP) or TN/
Overall Negatives
Actual Classes
Predicted Classes
statinfer.com
10
11. Sensitivity and Specificity
•By changing the threshold, the good and bad customers classification
will be changed hence the sensitivity and specificity will be changed
•Which one of these two we should maximize? What should be ideal
threshold?
•Ideally we want to maximize both Sensitivity & Specificity. But this is
not possible always. There is always a tradeoff.
•Sometimes we want to be 100% sure on Predicted negatives,
sometimes we want to be 100% sure on Predicted positives.
•Sometimes we simply don’t want to compromise on sensitivity
sometimes we don’t want to compromise on specificity
•The threshold is set based on business problem 11
statinfer.com
13. When Sensitivity is a high priority
•Predicting a bad customers or defaulters before issuing the loan
13
0(Yes-Defaulter) 1(Non-Defaulter)
0(Yes-Defaulter)
True positive (TP)
Actual customer is bad and
model is predicting them as
bad
False Negatives(FN)
Actual customer is bad
and model is predicting
them as good
Sensitivity=
TP/(TP+FN) or TP/
Overall Positives
1(Non-Defaulter)
False Positives(FP)
Actual customer is good and
model is predicting them as
bad
True Negatives(TN)
Actual customer is good
and model is predicting
them as good
Specificity =
TN/(TN+FP) or TN/
Overall Negatives
Actual Classes
Predicted Classes
statinfer.com
14. When Sensitivity is a high priority
•Predicting a bad defaulters before issuing the loan
14
0(Yes-Defaulter) 1(Non-Defaulter)
0(Yes-Defaulter)
True positive (TP)
Actual customer is bad and
model is predicting them as
bad. Rejected a Loan of
100,000
False Negatives(FN)
Actual customer is bad
and model is predicting
them as good Issued a
loan of 100,00
Sensitivity=
TP/(TP+FN) or TP/
Overall Positives
1(Non-Defaulter)
False Positives(FP)
Actual customer is good and
model is predicting them as
bad. Rejected a Loan of
100,000
True Negatives(TN)
Actual customer is good
and model is predicting
them as good. Issued a
loan of 100,00
Specificity =
TN/(TN+FP) or TN/
Overall Negatives
Actual Classes
Predicted Classes
statinfer.com
15. When Sensitivity is a high priority
•The profit on good customer loan is not equal to the loss on one bad
customer loan
•The loss on one bad loan might eat up the profit on 100 good customers
•In this case one bad customer is not equal to one good customer.
•If p is probability of default then we would like to set our threshold in such a
way that we don’t miss any of the bad customers.
•We set the threshold in such a way that Sensitivity is high
•We can compromise on specificity here. If we wrongly reject a good
customer, our loss is very less compared to giving a loan to a bad customer.
•We don’t really worry about the good customers here, they are not harmful
hence we can have less Specificity
15
statinfer.com
17. When Specificity is a high priority
•Testing a medicine is good or poisonous
17
0(Yes-Good) 1(Poisonous)
0(Yes-Good)
True positive (TP)
Actual medicine is good and
model is predicting them as
good
False Negatives(FN)
Actual medicine is good
and model is predicting
them as poisonous
Sensitivity=
TP/(TP+FN) or TP/
Overall Positives
1(Poisonous)
False Positives(FP)
Actual medicine is
poisonous and model is
predicting them as good
True Negatives(TN)
Actual medicine is
poisonous and model is
predicting them as
poisonous
Specificity =
TN/(TN+FP) or TN/
Overall Negatives
Actual Classes
Predicted Classes
statinfer.com
18. When Specificity is a high priority
•Testing a medicine is good or poisonous
18
0(Yes-Good) 1(Poisonous)
0(Yes-Good)
True positive (TP)
Actual medicine is good and
model is predicting them as
good. Recommended for
use
False Negatives(FN)
Actual medicine is good
and model is predicting
them as poisonous.
Banned the usage
Sensitivity=
TP/(TP+FN) or TP/
Overall Positives
1(Poisonous)
False Positives(FP)
Actual medicine is
poisonous and model is
predicting them as good.
Recommended for use
True Negatives(TN)
Actual medicine is
poisonous and model is
predicting them as
poisonous. Banned the
usage
Specificity =
TN/(TN+FP) or TN/
Overall Negatives
Actual Classes
Predicted Classes
statinfer.com
19. When Specificity is a high priority
•In this case, we have to really avoid cases like , Actual medicine is
poisonous and model is predicting them as good.
•We can’t take any chance here.
•The specificity need to be near 100.
•The sensitivity can be compromised here. It is not very harmful not to
use a good medicine when compared with vice versa case
19
statinfer.com
20. Sensitivity vs Specificity - Importance
•There are some cases where Sensitivity is important and need to be
near to 1
•There are business cases where Specificity is important and need to be
near to 1
•We need to understand the business problem and decide the
importance of Sensitivity and Specificity
20
statinfer.com
22. LAB - Sensitivity and Specificity
•Build a logistic regression model on fiber bits data
•Create the confusion matrix
•Find the accuracy
•Calculate Specificity
•Calculate Sensitivity
22
statinfer.com
28. ROC Curve
•Each threshold gives a sensitivity and specificity pair.
•What is the optimal sensitivity and specificity for a given problem?
•ROC curves helps us in choosing optimal sensitivity and specificity
pair.
•ROC tells us, how many mistakes are we making to identify all the
positives?
statinfer.com
28
29. ROC Curve
•ROCROC(Receiver operating characteristic)
tells us, how many mistakes are we making
to identify all the positives?
•ROC tells us, how many mistakes(False
positives) are we making to identify all the
positives?
•Curve is drawn by taking False positive rate
on X-axis and True positive rate on Y- axis
29
statinfer.com
30. ROC Curve - Interpretation
30
• How many mistakes are we making to identify all
the positives?
• How many mistakes are we making to identify 70%,
80% and 90% of positives?
• 1-Specificty(false positive rate) gives us an idea on
mistakes that we are making
• We would like to make 0% mistakes for identifying
100% positives
• We would like to make very minimal mistakes for
identifying maximum positives
• We want that curve to be far away from straight line
• Ideally we want the area under the curve as high as
possible
statinfer.com
32. ROC and AUC
• We want that curve to be far away from straight line. Ideally we want the area
under the curve as high as possible
• ROC comes with a connected topic, AUC. Area Under
• ROC Curve Gives us an idea on the performance of the model under all possible
values of threshold.
• We want to make almost 0% mistakes while identifying all the positives, which
means we want to see AUC value near to 1
32
statinfer.com
35. LAB: ROC and AUC
•Calculate ROC and AUC for Product Sales Data/Product_sales.csv
logistic regression model
•Calculate ROC and AUC for fiber bits logistic regression model
35
statinfer.com
39. When to use AUC over Accuracy?
•AUC is not same as accuracy. Accuracy is calculated at one cut-off
point.
•Use AUC when you want to work with probabilities and scoring rather
than simply classifying on one threshold
•Use AUC when each point probability is important for you than
accuracy on two classes.
•Use AUC in case of class imbalance. When one false positive
misclassification is not same as on negative misclassification
statinfer.com
39
41. What is a best model? How to build?
•A model with maximum accuracy /least error
•A model that uses maximum information available in the given data
•A model that has minimum squared error
•A model that captures all the hidden patterns in the data
•A model that produces the best perdition results
41
statinfer.com
42. Model Selection
•How to build/choose a best model?
•Error on the training data is not a good meter of performance on
future data
•How to select the best model out of the set of available models ?
•Are there any methods/metrics to choose best model?
•What is training error? What is testing error? What is hold out sample
error?
42
statinfer.com
44. LAB: The most accurate model
•Data: Fiberbits/Fiberbits.csv
•Build a decision tree to predict active_user
•What is the accuracy of your model?
•Grow the tree as much as you can and achieve 95% accuracy.
44
statinfer.com
49. The Training Error
•The accuracy of our best model is 95%. Is the 5% error model really
good?
•The error on the training data is known as training error.
•A low error rate on training data may not always mean the model is
good.
•What really matters is how the model is going to perform on unknown
data or test data.
•We need to find out a way to get an idea on error rate of test data.
•We may have to keep aside a part of the data and use it for validation.
•There are two types of datasets and two types of errors
49
statinfer.com
50. Two types of datasets
•There are two types of datasets
• Training set: This is used in model building. The input data
• Test set: The unknown dataset. This dataset is gives the accuracy of the final model
•We may not have access to these two datasets for all machine learning problems.
In some cases, we can take 90% of the available data and use it as training data
and rest 10% can be treated as validation data
• Validation set: This dataset kept aside for model validation and selection. This is a
temporary subsite to test dataset. It is not third type of data
•We create the validation data with the hope that the error rate on validation
data will give us some basic idea on the test error
50
Overall Data
Training data Validation datastatinfer.com
51. Types of errors
•The training error
• The error on training dataset
• In-time error
• Error on the known data
• Can be reduced while building the model
•The test error
• The error that matters
• Out-of-time error
• The error on unknown/new dataset.
“A good model will have both training and test error very near to each
other and close to zero”
51
statinfer.com
53. The problem of over fitting
•In search of the best model on the given data we add many predictors, polynomial
terms, Interaction terms, variable transformations, derived variables, indicator/dummy
variables etc.,
•Most of the times we succeed in reducing the error. What error is this?
•So by complicating the model we fit the best model for the training data.
•Sometimes the error on the training data can reduce to near zero
•But the same best model on training data fails miserably on test data.
statinfer.com
54. The problem of over fitting
•Imagine building multiple models with small changes in
training data.
•The resultant set of models will have huge variance in their
parameter estimates.
•If we build a model on sample1 of training data.
• Sample2 will almost have the same properties as sample1 but the
coefficients in the model change drastically if the model is over fitted.
• Same with case of training sample3
•Hence over fitted models are the models with huge variance
(variance in model parameters)
• The model is made really complicated, that it is very sensitive
to minimal changes
•In simple terms –Variance is how much the model
parameters changes with small changes in training data
Training Data
Sample1 Sample2 Sample3
M1 M2 M3
Q11
Q21
Q31
Q12
Q22
Q32
Q13
Q23
Q33
statinfer.com
54
55. The problem of over fitting
•By complicating the model the variance of the parameters estimates
inflates
•Model tries to fit the irrelevant characteristics in the data
•Over fitting
• The model is super good on training data but not so good on test data
• We fit the model for the noise in the data
• Less training error, high testing error
• The model is over complicated with too many predictors
• Model need to be simplified
• A model with lot of variance
55
statinfer.com
57. LAB: Model with huge Variance
•Data: Fiberbits/Fiberbits.csv
•Take initial 90% of the data. Consider it as training data. Keep the final
10% of the records for validation.
•Build the best model(5% error) model on training data.
•Use the validation data to verify the error rate. Is the error rate on
the training data and validation data same?
57
statinfer.com
60. The problem of under-fitting
•Simple models are better. Its true but is that always true? May not be
always true.
•We might have given it up too early. Did we really capture all the
information?
•Did we do enough research and future reengineering to fit the best
model? Is it the best model that can be fit on this data?
•By being over cautious about variance in the parameters, we might miss
out on some patterns in the data.
•Model need to be complicated enough to capture all the information
present.
60
statinfer.com
61. The problem of under-fitting
•If the training error itself is high, how can we be so sure about the
model performance on unknown data?
•Most of the accuracy and error measuring statistics give us a clear idea
on training error, this is one advantage of under fitting, we can
identify it confidently.
•Under fitting
• A model that is too simple
• A mode with a scope for improvement
• A model with lot of bias
61
statinfer.com
63. LAB: Model with huge Bias
•Lets simplify the model.
•Take the high variance model and prune it.
•Make it as simple as possible.
•Find the training error and validation error.
63
statinfer.com
66. Model Bias and Variance
•Over fitting
• Low Bias with High Variance
• Low training error – ‘Low Bias’
• High testing error
• Unstable model – ‘High Variance’
• The coefficients of the model change with small changes in the data
•Under fitting
• High Bias with low Variance
• High training error – ‘high Bias’
• testing error almost equal to training error
• Stable model – ‘Low Variance’
• The coefficients of the model doesn’t change with small changes in the data
66
statinfer.com
67. Model Bias and Variance
67
Bias Low to high
VarianceLowtohigh
Model aim is to hit
the center of circle
statinfer.com
69. Bias-Variance Decomposition
•Overall Model Squared Error = Irreducible Error + Bias2 + Variance
•Overall error is made by bias and variance together
•High bias low variance, Low bias and high variance, both are bad for
the overall accuracy of the model
•A good model need to have low bias and low variance or at least an
optimal where both of them are jointly low
•How to choose such optimal model. How to choose that optimal model
complexity
69
statinfer.com
71. Two ways of reading bias and variance
•Variance and bias vs Model Complexity
•Testing and Training Error vs Model Complexity
statinfer.com
71
72. Bias Variance Tradeoff
72
Model Complexity
Optimal
Models
Bias2
Variance
statinfer.com
Variance and bias vs Model Complexity
73. Test and Training error
73
Model Complexity
Test Error
Training
Error
Error
Optimal
complexity
statinfer.com
Testing and Training Error vs Model Complexity
74. Choosing optimal model
•Choosing optimal model reduces both bias and variance.
•Unfortunately There is no standard scientific method
•How to choose optimal model complexity that gives minimum test
error?
•Training error is not a good estimate of the test error.
•We can use
• Hold out data validation
• K-fold Cross validation methods
• Boot strap cross validation to choose the optimal and consistent model
74
statinfer.com
76. Holdout data Cross validation
• The best solution is out of time validation. Or the testing error should be given high
priority over the training error.
• A model that is performing good on training data and equally good on testing is
preferred.
• We may not have to test data always. How do we estimate test error?
• We take the part of the data as training and keep aside some potion for validation. May
be 80%-20% or 90%-10%
• Data splitting is a very basic intuitive method
76
Training data Hold-out
Overall Training data
To build the
model
To validate
model
statinfer.com
77. LAB: Holdout data Cross validation
•Data: Fiberbits/Fiberbits.csv
•Take a random sample with 80% data as training sample
•Use rest 20% as holdout sample.
•Build a model on 80% of the data. Try to validate it on holdout sample.
•Try to increase or reduce the complexity and choose the best model
that performs well on training data as well as holdout data
77
statinfer.com
83. Ten-fold Cross - Validation
• Divide the data into 10 parts(randomly)
• Use 9 parts as training data(90%) and the tenth part as holdout data(10%)
• We can repeat this process 10 times
• Build 10 models, find average error on 10 holdout samples. This gives us an idea on
testing error
83
statinfer.com
85. K-fold Cross Validation
•A generalization of cross validation.
•Divide the whole dataset into k equal parts
•Use kth part of the data as the holdout sample, use remaining k-1 parts
of the data as training data
•Repeat this K times, build K models. The average error on holdout
sample gives us an idea on the testing error
•Which model to choose?
• Choose the model with least error and least complexity
• Or the model with less than average error and simple (less parameters)
• Finally use complete data and build a model with the chosen number of parameters
•Note: Its better to choose K between 5 to 10. Which gives 80% to 90%
training data and rest 20% to 10% is holdout data 85
statinfer.com
87. LAB- K-fold Cross Validation
•Build a tree model on the fiber bits data.
•Try to build the best model by making all the possible adjustments to
the parameters.
•What is the accuracy of the above model?
•Perform 10 –fold cross validation. What is the final accuracy?
•Perform 20 –fold cross validation. What is the final accuracy?
•What can be the expected accuracy on the unknown dataset?
87
statinfer.com
95. Bootstrap Methods
•What if we have just 100 observations overall. If we do a 10 fold cross
validation then each part has only 10 observations.
•K-Fold might fail while validating the models in each iteration.
•Boot strapping is a powerful tool to get an idea on accuracy of the
model and the test error, especially when dataset size is small.
•Can estimate the likely future performance of a given modeling
procedure, on new data not yet realized.
95
statinfer.com
96. Bootstrap Method
96
Training Data (size-N)
Sample (size-N) Sample (size-N) Sample (size-N) Sample (size-N)
1 2 3 B
Samples with
replacement
statinfer.com
97. The Algorithm
•We have a training data is of size N
•Draw random sample with replacement of size N – This gives a new
dataset, it might have repeated observations, some observations might
not have even appeared once.
•Create B such new datasets. These are called boot strap datasets
•Build the model on these B datasets, we can test the models on the
original training dataset.
statinfer.com
97
98. Bootstrap Example
•Example
1. We have a training data is of size 500
2. Boot Strap Data-1: Create a dataset of size 500. To create this
dataset, draw a random point, note it down, then replace it back.
Again draw another sample point. Repeat this process 500 times.
This makes a dataset of size 500. Call this as Boot Strap Data-1
3. Multiple Boot Strap datasets :Repeat the procedure in step -2
multiple times. Say 200 times. Then we have 200 Boot Strap
datasets
4. We can build the models on these 200 boost strap datasets and the
average error gives a good idea on overall error.
5. We can even use the original training data as the test data for each of
the models
98
statinfer.com
99. LAB: Bootstrap cross validation
•Create a new dataset by taking a random sample of size 250; Name it
as fiber_sample_data
•In fiber_sample_data, draw a boot strap sample with sufficient sample
size
•Build a tree model and get an estimate on true accuracy of the model
99
statinfer.com
103. F1 – Score
statinfer.com
103
0(Positive) 1(Negative)
0(Positive)
True positive (TP)
Actual condition is
Positive, it is truly
predicted as positive
False Negatives(FN)
Actual condition is
Positive, it is falsely
predicted as negative
1(Negative)
False Positives(FP)
Actual condition is
Negative, it is falsely
predicted as positive
True Negatives(TN)
Actual condition is
Negative, it is truly
predicted as negative
Actual Classes
Predicted Classes
𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑇𝑃
𝑇𝑃 + 𝐹𝑁
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =
𝑇𝑃
𝑇𝑃 + 𝐹𝑃
• Recall is a fraction and Precision is a
fraction.
• F1 score is Harmonic mean of Recall and
Precision
104. When to use F1 – Score
•F1 score need to be calculated separately for individual classes
•Use it while dealing with while dealing with imbalanced classes.
Where one class really dominates the other
•Use F1 score in It is very useful for multi class problems. F1 is also
known as per class accuracy.
•Limitations:
• Different values of F1 score for different threshold values
• It is very difficult to set threshold for F1 score.
statinfer.com
104
105. LAB: F1-Score
•Product Sales Data/Product_sales.csv
•Build a logistic regression model on product sales data.
•Create a confusing matrix.
•Calculate F1 score
statinfer.com
105
109. Conclusion
•We studied
• Validating a model, Types of data & Types of errors
• The problem of over fitting & The problem of under fitting
• Bias Variance Tradeoff
• Cross validation & Boot strapping
•Training error is what we see and that is not the true performance
metric
•Test error plays vital role in model selection
•Cross Validation and Boot strapping techniques give us an idea on test
error
•Choose the model based on the combination of Accuracy, AIC, Cross
Validation and Boot strapping results
•Bootstrap is widely used in ensemble models & random forests.
109
statinfer.com
113. Statinfer.com
Data Science Training and R&D
statinfer.com
113
Corporate Training
Classroom Training
Online Training
Contact us
info@statinfer.com
venkat@statinfer.com