Lecture 11

EE531 Statistical Learning Theory

Contents
Linear separator
Non-probabilistic methods
Perceptron learning algorithm
Support vector machine : Maximum margin classifier

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Linear separator: properties

Some relevant properties of
1) For any two points

so is normal to
Define
to be the unit normal.
2) For any point
in ,
3) The signed distance of any point
to
is
4) The geometric margin of example
defined by
is

with respect to hyperplane

A point is misclassified iff its margin is <0.

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Perceptron learning alg.: formulation

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Perceptron learning alg.: pseudo code

Algorithm :
Initialize
and
(to anything)
While there were misclassified examples on the previous iteration do
For

to

If
( Increases margin )

Given new

then

, predict sign

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Perceptron learning alg.: Properties

Properties
Easy to implement
Guaranteed to converge to a separator in finitely many iterations, if data are
linearly separable.
Many solutions can be found, depending on initial
Can require many steps : the smaller the gap, the longer it takes to find
Cycles if data are not separable

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Perceptron learning alg.: Dual rep.

Dual representation

Restated algorithm
Initialize
to
While there were misclassifications
For

to

If

then
7

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Perceptron learning alg.: Theorem

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Support vector machine: introduction

Maximizing the margin leads to a particular choice of decision boundary.
The location of the boundary is determined by a subset of the data points,
known as support vectors, which are indicated by the circles.

Infinite number of
possible linear
separators (decision
boundaries)

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Support vector machine: introduction (2)

Support vector machines
Names a whole family of algorithms. Well start with the maximum margin
separator. The idea is to find the separator with the maximum margin from all the
data points. Well see, later, a theoretical argument that this might be a good
idea. Seems a little less haphazard than a perceptron.

10

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Support vector machine: formulation

Lagrange
multiplier

positive if
constraint is
satisfied

11

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Support vector machine: use of KuhnTucker

12

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Support vector machine: Lagrange

formulation

13

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Support vector machine: Solution

14

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Support vector machine: advantages

Whats good about this?
few support vectors in practice sparse representation
maximizing margin (minimizing
) means choosing the simplest possible
hypothesis
generalization error is related to the proportion of support vectors

15

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Summary
Linear separator
Non-probabilistic methods
Perceptron learning algorithm
Support vector machine : Maximum margin classifier

16

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Appendix: Perceptron mistake

guarantee

17

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Appendix:
Perceptron mistake guarantee(Proof)

18

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Appendix:
Perceptron mistake guarantee(Proof)

19

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology

Appendix:
Perceptron mistake guarantee(Proof)

20

EE531 Statistical Learning Theory

Spring, 2016

Korea Advanced Institute of Science and Technology