Contents

Prefaceiii

Learning and Intuitionvii

1. Data and Information1

   1. Data Representation . . . . . . . . . . . . . . . . . . . . . . . . .2

   2. Preprocessing the Data . . . . . . . . . . . . . . . . . . . . . . .4

2. Data Visualization7

3. Learning11

   1. In a Nutshell . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15

4. Types of Machine Learning17

   1. In a Nutshell . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20

5. Nearest Neighbors Classification21

   1. The Idea In a Nutshell . . . . . . . . . . . . . . . . . . . . . . . .23

6. The Naive Bayesian Classifier25

   1. The NaiveBayes Model. . . . . . . . . . . . . . . . . . . . . .25

   2. Learning a Naive Bayes Classifier. . . . . . . . . . . . . . . . .27

   3. Class-Prediction for New Instances . . . . . . . . . . . . . . . . .28

   4. Regularization . . . . . . . . . . . . . . . . . . .. . . . . . . . .30

   5. Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31

   6. The Idea In a Nutshell . . . . . . . . . . . . . . . . . . . . . . . .31

7. The Perceptron33

   1. The Perceptron Model. . . . . . . . . . . . . . . . . . . . . . .34

i

iiCONTENTS

7.2A Different Cost function: Logistic Regression. . . . . . . . . .37

7.3The Idea In a Nutshell . . . . . . . . . . . . . . . . . . . . . . . .38

1. Support Vector Machines39

   1. The Non-Separable case. . . . . . . . . . . . . . . . . . . . . .43

2. Support Vector Regression47

3. Kernel ridge Regression51

   1. Kernel Ridge Regression . . . . . . . . . . . . . . . . . . . . . .52

   2. An alternative derivation. . . . . . . . . . . . . . . . .. . . . .53

4. Kernel K-means and Spectral Clustering55

5. Kernel Principal Components Analysis59

   1. Centering Data in Feature Space . . . . . . . . . . . . . . . . . .61

6. Fisher Linear Discriminant Analysis63

   1. Kernel Fisher LDA . . . . . . . . . . . . . . . .. . . . . . . . .66

   2. A Constrained Convex Programming Formulation of FDA. . . .68

7. Kernel Canonical Correlation Analysis69

   1. Kernel CCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . .71

8. Essentials of Convex Optimization73

   1. Lagrangians and all that . . . . . . . . . . . . . . . . . . . . . . .73

9. Kernel Design77

   1. Polynomials Kernels. . . . . . . . . . . . . . . . . . . . . . . .77

   2. All Subsets Kernel. . . . . . . . . . . . . . . . . . . . . . . . .78

   3. The Gaussian Kernel. . . . . . . . . . . . . . . . . . . . . . . .79