Академический Документы
Профессиональный Документы
Культура Документы
Instance-based Learning
1-Nearest
Neighbor
Four
things
make
a
memory
based
learner:
1. A
distance
metric
Euclidian
(and
many
more)
2. How
many
nearby
neighbors
to
look
at?
One
1. A
weigh:ng
func:on
(op:onal)
Unused
Consistency
of
1-NN
Consider
an
es*mator
fn
trained
on
n
examples
e.g.,
1-NN,
regression,
...
1-NN overts?
k-Nearest
Neighbor
Four
things
make
a
memory
based
learner:
1. A
distance
metric
Euclidian
(and
many
more)
2. How
many
nearby
neighbors
to
look
at?
k
1. A
weigh:ng
func:on
(op:onal)
Unused
2.
K-nearest
neighbor
for
funcFon
Hng
smooth
away
noise,
but
there
are
clear
deciencies.
What
can
we
do
about
all
the
discon*nui*es
that
k-NN
gives
us?
Example i:
10
w.x + b
= 0
11
w.x + b
= 0
12
w.x + b
= 0
13
w.x + b
= 0
14
x+
margin 2
=
-1
w.x
+
b
=
0
w.x
+
b
w.x + b
= +1
x-
15
x+
margin 2
=
-1
w.x
+
b
=
0
w.x
+
b
w.x + b
= +1
x-
16
w.x + b
= 0
= +1
= -1
w.x + b
w.x + b
margin 2
17
=
-1
w.x
+
b
=
0
w.x
+
b
w.x + b
= +1
18
19
20
0/1
loss
Slack
penalty
C
Not
QP
anymore
Also
doesnt
dis*nguish
near
misses
and
really
bad
mistakes
LogisFc
regression:
Log
loss:
22
Constrained op*miza*on
23
Solve:
24
25
26
27
w.x + b
= 0
28
29
30
31
32
d=4
m
input
features
d
degree
of
polynomial
d=3
d=2
number
of
input
dimensions
34
grows
fast!
d
=
6,
m
=
100
about
1.6
billion
terms
35
Dot-product of polynomials
36
37
Polynomial
kernels
All
monomials
of
degree
d
in
O(d)
opera*ons:
38
Common
kernels
Polynomials
of
degree
d
Polynomials
of
degree
up
to
d
Gaussian
kernels
Sigmoid
39
Overvng?
Huge
feature
space
with
kernels,
what
about
overvng???
Maximizing
margin
leads
to
sparse
set
of
support
vectors
Some
interes*ng
theory
says
that
SVMs
search
for
simple
hypothesis
with
large
margin
O^en
robust
to
overvng
40
41
Classify as
42
Loss function
High dimensional
features with
kernels
SVMs
Logistic
Regression
Hinge loss
Log-loss
Yes!
No
43
Loss function
High dimensional
features with
kernels
Solution sparse
Semantics of
output
SVMs
Logistic
Regression
Hinge loss
Log-loss
Yes!
Yes!
Often yes!
Margin
Real probabilities
45