Lecture 10

EE531 Statistical Learning Theory

Contents
Introduction
Discriminant functions(DF)
Linear discriminant
Fishers linear discriminant
Perceptron
Linear regression with indicator variables

Probabilistic approach
Discriminative model(DM) : Logistic regression
Generative model(GM) : LDA
Logistic regression vs LDA

Multiple classes
one-versus-the rest classifier
one-versus-one classifier

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Introduction

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Approaches :Discriminant function

and posterior probability model

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Linear Discriminant Function:

Introduction

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Linear Discriminant Function: Fisher(1)

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Linear Discriminant Function: Fisher (2)

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Linear Discriminant Function: Fisher(3)

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Discriminant Function: Perceptron(1)

Frank Rosenblatt
1928-1971

Marvin Minsky
1927-

perceptron

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Discriminant Function: Perceptron(2)

4
10

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Discriminant Function:
Linear regression with indicator variables
Linear regression with indicator variables
In the binary case, target label
, then perform linear regression
(with least squares error criterion) as before.

Example 1-D data

Class 0

Class 1

There are strategies for extending this to multiple classes, but none of
them work very well.
Well get the same vector this way as from LDA, in the two-class
problem, as long as our class priors
. If the priors are
different, the direction will be the same, but the intercept will change.
EE531 Statistical Learning Theory

Spring, 2016

11

Korea Advanced Institute of Science and Technology

DM: Logistic Regression (1)

0.8

0.6

0.4

0.2

0
-10

-5

10

12

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

DM: Logistic Regression (2)

13

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

DM: Logistic Regression (3)

How to solve these equations? Iteratively reweighted least squares iteration
(an instance of Newton-Raphson). Usually converges because likelihood is
convex.
If the model is correct, then
converges to the true as amount of data
increases.
Since all
, we have
which means that the observed number of class 1 instances equals the
expected number.

14

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Example: Logistic Regression

Input : the amount of Dosage given to a mouse
Class : killed or alive
The training data is given as follows:
Dosage
1.6907
1.7242
1.7552
1.7842
1.8113
1.8369
1.861
1.8839

number of
mice
59
60
62
56
63
59
62
60

number
killed
6
13
18
28
52
53
61
60

P(killed|Dosage)
0.10
0.22

0.29
0.50
0.83
0.90
0.98
1.00

Trained logistic regressor:

15

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

GM: Linear Discriminative Analysis(LDA)

16

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Logistic regression vs LDA

17

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Logistic regression vs LDA (3)

18

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Multiple Classes (1)

One-versus-the rest classifier
One-versus-one classifier/majority vote

19

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Multiple Classes (2)

Assign

to class

if

Decision boundary

Decision region is convex

20

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Summary
Introduction
Discriminant functions(DF)
Linear discriminant
Fishers linear discriminant
Perceptron
Linear regression with indicator variables

Probabilistic approach
Discriminative model (DM): Logistic regression
Generative model (GM): LDA
Logistic regression vs LDA

Multiple classes
one-versus-the rest classifier
one-versus-one classifier

21

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Appendix: Newton-Raphson Method

22

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology