Support Vector Machines

Abstract

In this chapter we are going to study the concept of support vector machines as developed by Vapnik and others. The concept was first proposed as an alternative to neural networks, when neural networks were not performing up to the grand expectations that they came with. SVM proposed a very targeted mathematical approach towards finding the optimal solution in case of classification or regression. We will first study the original SVM theory that tries to solve the problem of linear classification. Then, we will see how it can be further generalized for nonlinear problems with use of kernels and also how it is extended for solving the problems of regression. The theory of SVM proposed an elegant solution towards optimization and generalization and more importantly was extremely successful in getting results that neural network-based methods only hoped for at the time.

Appendix

New Lagrangian to be minimized is given as

$$\displaystyle \begin{aligned} \min_{\mathbf{w}} \left\{ \left(\frac{1}{n}\sum_{i=1}^{n}\max(0,1-y_i[(\mathbf{w}.\mathbf{x}) - w_0])\right) + \lambda \Phi(\mathbf{w}) \right\} \end{aligned} $$

(8.20)

Here, with the $\max$ function, we are essentially ignoring the cases when there is error in the classification.