. . , . .

. ..

, 2007

1

,



( , , , , , , )

1 , 12 , 24 + 12
,

. , +

( ), http://mmphome.1gb.ru
: (VetrovD@yandex.ru) (DKropotov@yandex.ru)


1
1.1 . .
1.1.1 . . . . . . . .
1.1.2 . .
1.1.3 (
1.1.4 . . . . . . . .
1.1.5 . . . . . . .
1.1.6 . . . . . .
1.2 .
1.2.1 .
1.2.2 . .
1.2.3 . . . . . . . . . . . . .
1.3 : .

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

3
. 4
. 4
. 5
. 6
. 6
. 7
. 8
. 9
. 9
. 10
. 10
. 12

2
2.1 : . . . . . . . . . . . . . . .
2.2 . .
2.2.1 . . . . . . . . . . . . . . . . .
2.2.2 . . . . . . . . . . . . . . .
2.3 . . . . . . . . . . . . . .
2.3.1 . . . . . . . . . . . . . . . . . . . . .
2.3.2 . . . . . . . . . . . . . . . . . . . .
2.3.3 . . . . . . . . . . . . . . .
2.4 EM- . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4.1 . . . . .
2.4.2 . . . . . . . .
2.4.3 . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

14
15
16
16
17
18
18
19
19
21
21
22
23

3
3.1 :
3.2 . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 . . . . . . . . . . . . .
3.2.2 . . . . . . . . . . . . . . . .
3.2.3 . . . . . . . . . . . . .
3.3
3.3.1 . . . . . . . . . . . . . . . . . . .
3.3.2 IRLS . . . . . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

25
26
27
27
28
29
30
30
31

. . . . . .
. . . . . .
. . . . . .
)
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.

4
4.1 : . . . . . . . . . . . . . . . . . . . . . . . .
4.2 . . . . . . . . . . .
4.2.1 . . . . . . . . . . . . . . . . . . . .
4.2.2 . . . . . . . . . . . . . . . . .
4.2.3 . . . . . . . . . . . . . . .
4.2.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.5 . . . . . . . . . . . . . . . . . . . . . .
4.3 . . . . . . . . . . . . . . .
4.4 . . . . . . . . . . . . . . . . . . .
4.4.1 . . .
4.4.2 , . . . . .

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

34
35
37
37
38
43
44
46
48
50
50
51

5
5.1 : . . . . . . . . .
5.2 . . . . . . . . .
5.2.1
5.2.2 . . . . . . .
5.3 . . . . . . . . . . . .
5.3.1 - . . . . . . . . . . . . . . .
5.3.2 - . . . . . . .
5.3.3 . .
5.3.4 . . . . . . . . .

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

55
56
56
56
57
59
59
61
62
63

6 .
6.1 : . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.1.1 Sum- Product- rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.1.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.3.1 . . . . . . . . . . . . . . .
6.3.2 . . . . . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

65
66
66
67
67
67
68
69
69
70

7 .
7.1 : Ad Hoc . . . . . . . . . . . . . . . . .
7.2 . . . . . . . . . . . . . . . . . . . . . . . . . .
7.2.1 . . . . . . . . . . . . .
7.2.2 . . . . . . . . . . . . . . . . . . . . .
7.2.3 . . . . . . . . . . . . . . . . . . . . .
7.3 . . . . . . . . . . . . . . . . . . . .
7.3.1 . . . . . . . . . . . . . . . . . . . . . . . .
7.3.2 . . . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

73
74
74
74
75
77
77
77
79

8
8.1 : . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.2 . . . . . . . . . . . . . . . . . . . . . . . .
8.3 . . . . . . . . . . . . . . . . . . . . .

82
83
84
88

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

9
9.1 : . . . . .
9.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.2.1 RVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.2.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.2.3

.
.
.
.
.

.
.
.
.
.

94
95
96
96
97
98

10
102
10.1 : . . . . . . . . . . . . . . . . . . . . . . 103
10.2 . . . . . . . . . . . . . . . . . . . . . . . . 104
10.2.1 104
10.2.2 . . . . . . . . . . . . . 105
11
11.1 : -
11.2 . . . . . . . . . . . . .
11.2.1 . . . . . . . . . . . . . .
11.2.2 .
11.3 - . . . . . . . . . . . . .
11.3.1 . . . . . . . . . .
11.3.2 - . . . . .
11.3.3 - . . .

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

109
110
111
111
113
115
115
116
117

12 .
12.1 : . . . . . . . . .
12.1.1 . . . . . . . . . . . . . . . . . . . . . . . . . .
12.1.2 . . . . . . . . . . . . . . . . . . . . . . . .
12.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.2.1 . . . . . . . . . . . . . . . . . . . . . . . .
12.2.2 . . . . . . . . . . . . . . . . . . . . . . .
12.2.3 . . . . . . . . . . . . . . . . . . . . . .
12.3 . . . . . . . . . . . . . . . .
12.3.1 . . . . . . . . . . . . .
12.3.2 . . . . . . . . . .
12.3.3 . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

119
120
120
121
121
121
124
125
126
126
128
129

-
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

, . ,
.

 1.

1.1

(case-based reasoning)
(model-based reasoning)



, / , , .

1.1.1

,
X = {xi }ni=1 , xi =
(xi,1 , . . . , xi,d )
x t, ,
T = {1, . . . , l}
(), x
t ( )
{p(s|x)}ls=1 (. . 1.1)

. 1.1. . , . ,

:
:

 1.

:
,
: /
:

1.1.2

X = {xi }ni=1 ,
xi = (xi,1 , . . . , xi,d )
x t
(), x
t, (t , t+ )
p(t|x)

. 1.2. . ,

: , ,
:
, ,
:
:
:

 1.

1.1.3

( )

()

X = {xi }ni=1 , xi = (xi,1 , . . . , xi,d )
(),
Sk
() X = j=1 Ck , Cj {x1 , . . . , xm }, Ci Cj =
(. . 1.3)

. 1.3. . . , ,

: -
: , ,
:

: , , -

1.1.4

, , ,
X = {xi }ni=1 , xi =
(xi,1 , . . . , xi,d ), A (x) = 1
, ,

 1.

(), x
A x, p(A (x) = 1|x)
(. . 1.4)

. 1.4. . , ,

: /
:
:
:
: (, , ) ,

1.1.5

-

X = {x[i]}ni=1 , x[i] = (x1 [i], . . . , xd [i]),

(), {
x[i]}n+q
i=n+1 , n+q
{(x [i], x+ [i])}i=n+1 p(x[n + 1], . . . , x[n +
q]|x[1], . . . , x[n]) q (. . 1.5)
, ,

 1.

10

. 1.5. . ( ).

:
:
:
:
:

1.1.6

X = {xi }ni=1 , xi =
(xi,1 , . . . , xi,d )
, , (. . 1.6)
... ...
((0.45 x4 32.1)&(6.98 x7 6.59) (3.21 x2 3.345)),
( ( )
)

: ()

: ,
:
:
:

 1.

11

. 1.6. . . ,

1.2
1.2.1

I





:


,

II

.
.. - .
,
..

n
d

(active learning)

 1.

1.2.2

12

. ,


,


,

(
, ..)



, t x , p(t|x)

, ,




, -, .

1.2.3

()
(X, t) = (xi , ti )ni=1 , xi Rd , t T
( , ,
..)
,
p(x, t)
p(t|x), ..

 1.

13

(a)

(b)

. 1.7. . (a) ,
. (b)
, (a)
3

. 1.8. .
, . ,
, ,

, , (. . 1.7 1.8).


-
- . :
(. , . , 1974)
(. , 1978)
- (, 1974, , 1978)
(, 1992)

 1.

14

(, (weight decay) )

(SVM)

...

1.3

: .

X : R
(a, b)
Z
P (a X b) =

p(x)dx,
a

p(x) X,
Z

p(x) 0,

p(x)dx = 1

,
p(x|). c
f () = p(x|),
..
.
X
X, x = (x1 , . . . , xn )


M L = arg max f () = arg max p(x|) = arg max

n
Y

p(xi |)

i=1

, n
, . n

 1.

15

2
X w1 N (x|1 , 12 ) + + wm N (x|m , m
)
2
= (m, 1 , 12 , . . . , m , m
, w1 , . . . , wm )

p(x|)

n
Y

p(xi |)

i=1

n X
m
Y
i=1 j=1

kxi j k2
wj

exp
2j2
2j

m
M L = n,

j,M L = xj ,

j,M
L = 0,

w
M L,j =

max

1
n

( ),

!! m ( ) !!
,

.
. .
. -,
() .

16

 2.

2.1

17

1
(x )2
X N (x|, 2 ) p(x|, 2 ) =
exp
2 2
2
2 = DX , E(X EX)2

p(x|m,s )

= EX,

3s

. 2.1.

,
,


X N (x|, ) p(x|, ) =

1
exp (x )T 1 (x ) ,
2
det

= EX, = E(X )(X )T n

, ()
ij = E(Xi i )(Xj j ) = Cov(Xi , Xj )
,
Cov(Xi , Xj )
[1, 1]
(Xi , Xj ) , p
DXi DXj

 2.

18

(. /)
,
: . .

(a)

(b)

(c)

. 2.2. . (a)
, (b) , (c)
,

2.2
2.2.1

X = {xi }ni=1 ,
xi = (xi,1 , . . . , xi,d )
t T
,
t = {ti }ni=1

(x, t)
() ,
, ,
p(x, t) ( , , )

 2.

19

S(t, t) , t
t
, t = t
Sr (t, t) = (t t)2 Sc (t, t) = I{t 6= t}


p(x, t) ,
. ,

Z

ES(t, t) = S(t, t(x))p(x, t)dxdt min,

t(x) , x


,

2.2.2

Sc (t, t) = I{t 6= t}
tB (x) = arg maxtT p(x, t)

Z Z
ES(t, t) =
S(t, t(x))p(x, t)dxdt =
l Z
X

Z
S(s, t(x))p(x, s)dx = 1

s=1

p(x, t(x))dx

Z
1

Z
max p(x, t)dx = 1
t

p(x, tB (x))dx = ES(t, tB )

,



,

 2.

2.3
2.3.1

20

,
,
, , .

k ( ) 1 . . . k
ni i (. . 2.3).
p(x) =

ni
I{x i }
n|i |

. 2.3.

( i )
i
!! !!
d k d

 2.

2.3.2

21

D, x.
R
P = D p(x)dx. n ,
k , k nP
, D , , P p(x)V , V
D, ..
k
p(x) =
nV

D ,
k, .. D
D ,

k(u) = k(u) 0,

p(x) =

1
n

k(u)du = 1.

Pn
i=1

k(x xi )

, , p(x) 0
Z

p(x)dx =

1X
n i=1

Z
k(x xi )dx = 1

T , , k(u) = 12 exp u 2 u
, .. :
k(u) , h1d k( u
h )

2.3.3

!! h !!
, ,

, , , , ,

: , ,

 2.

22

. 2.4.

,
p(x) =

k
,
nV

V D, x
, D x,
k
X .

, p(x) p(x) n

k
lim k =
lim
=0
n
n n
,
, .. V 0 n

!! k !!

k n
, p(x) , .. , ,
p(x)

1
x

 2.

23

. 2.5. K

2.4
2.4.1

EM-

, p(x|)


p(X|) =

n
Y

p(xi |) max

i=1

, , (..
)

N (x|, 2 ) .


L(X|, ) =

n
X
(xi )2
i=1

n
X
(xi )

L
=

i=1
n

2 2

2 2

n log

n
log(2) max
,
2
n

= 0 M L =

1X
xi
n i=1
n

n
1X
L X (xi )2
2
=

=
0

=
(xi )2
M
L
3

n
i=1
i=1

 2.

2.4.2

24

,
Pl
Pl
: X j=1 wj p(x|j ), j=1 wj = 1, wj 0
R
R
: X w()p(x|())d, w()d = 1, w() 0
, ,
,

n
l
X
X
L(X|) =
log
wj p(xi | j )
i=1

j=1

,

-
, , .. j(i) ( ,
z),

n
X
L(X, Z|) =
log wj(i) p(xi | j(i) )
i=1

- , .. Z,
: Z (-),
(-)
-
: X, , w
, w
-:
p(X, Z|, w)
p(Z|X, , w) = P
Z p(X, Z|, w)

 2.

25

. 2.6.

-:
EZ log p(X, Z|, w) =

p(Z|X, , w) log p(X, Z|, w) max

,w

Z = Z0 , . (
) Z, log p(X, Z0 |, w)

-,

2.4.3

Pl
X j=1 wj N (x|j , j )) Rd (. . 2.6)

-
j , wj , j
-: z i {0, 1},
xi
(zij ) = Pl

P
j

zij = 1, ,

wj N (xi |j , j ))

k=1

wk N (xi |k , k ))

-: zi ,
new
=
j
new
=
j

n
1 X
(zij )xi ,
Nj i=1

wjnew =

Nj
,
n

Nj =

n
1 X
(zij )(xi new
)T (xi new
)
j
j
Nj i=1

-,

n
X
i=1

(zij )

 2.

26

-

- ,
,
- l
!! l !!

. .. . . .
,

27

 3.

3.1

28

, Ax = b
A (
), x = A1 b
, , .. A . AT
AT Ax = AT b

1 T
x = AT A
A b

1 T
AT A
A A, x

AT A , ,

- AT A
AT A + I,
I , .
> 0

1 T
x = AT A + I
A b

.


, , (. . 3.1)

1 T
, AT A
A A1

 3.

29

(0.0175,0.0702)

5x

1.

=1

+
1
-x

+x
x1

-2x2=1

(a)

(b)

. 3.1. (a) (
), (b) ,
,

3.2
3.2.1

x
t


y(x, w)
w = arg max F (X, t, w)
w

:
,

y(x, w) =

m
X

wj j (x) = wT (x)

j=1

j (x)
(,
- , )

(x) = xk

 3.

30

(x) = xk1 xk2 . . . xkl

(x) = exp(kx x0 kp ), , p > 0.
( w)

S(t, t) t

Z Z
ES(t, y(x, w)) =
S(t, y(x, w))p(x, t)dxdt min
w

p(t|x)

.
S(t, t) = (t t)2 ;
S(t, t) = |t t| ;
S(t, t) = 1 (t t) .
, ES(t, y(x, w)),
y(x) = Ep(t|x);
y(x) = med p(t|x);
y(x) = mod p(t|x) = arg maxt p(t|x).
,
,

3.2.2

S(t, t) = (t t)2

y = w, = (ij ) = (j (xi )) Rnm
,
ky tk2 = kw tk2 min
w

w ,
kw tk2
[wT T w 2wT T t + tT t]
=
= 2T w 2T t = 0
w
w
w = (T )1 T t

 3.

31

,
w = t
X .

T Rmm m > n
,

1 T
w = T + I
t

t = y = T + I 1 T t = Ht

H, hat-matrix

( )



( ) .

3.2.3

.
t p(t|x)
, t .
y(x), x
t = y(x) + ,

N (|0, 2 )

y(x),

 3.

32

y(x)

1
(ti yi )2

p(t|y) =
exp
max
2 2
2
i=1
n
Y

, ,
n
n
X
X
(ti yi )2 =
(ti wT (xi ))2 min
i=1

i=1

,

,

p(w|t, X) =

p(t|X, w)p(w)
max,
w
p(t, X)

w,
2

p(w) N w 0, I .
p(w|t, X)

1
m/2

2
2
2
exp

kw

tk
+
kwk
m+n
2
2
2

w ,
w = (T + I)1 t
,

3.3
3.3.1

t {1, +1}
, ,
t(x) = sign(y(x)) = sign

m
X
j=1

wj j (x)

 3.

33

: ?
+ t = +1
t = 1
: y(x),
x

, ,


p(t|x, w) =

1
1 + exp(ty(x))

(. . 3.2). ,
,

P
t

p(t|x, w) = 1 p(t|x, w) > 0, ,

0.8

0.6

0.4

0.2

0
5

. 3.2. .
ti = 1, ti = 1

p(t|X, w) =

n
Y
i=1

3.3.2

p(ti |xi , w) =

n
Y

i=1 1 + exp ti

1
Pm

j=1 wj j (xi )

IRLS

,
,
2 log p(t|x, w)
0
w2

 3.

34

, .
L(w) = log p(t|X, w),
, , , ,

:
f (x) min
w

1
f (x) ' g(x) = f (x0 ) + (f (x0 )) (x x0 ) + (x x0 )T (f (x0 ))(x x0 )
2
g(x ) = f (x0 ) + (f (x0 ))(x x0 ) = 0
T

x = x0 (f (x0 ))1 (f (x0 ))

g(x)
f(x)

x1

x0
2

. 3.3. . f (x) = log(1 + exp(x)) + x5 . x0 = 6

f (x) g(x). x1 = 2.4418

L(w)

wnew = wold H 1 L(w),
H = L(w)

1
si = 1+exp(t
, :
i yi )
L(w) = T diag(t)s,

L(w) = T R

 3.

35

s1 (1 s1 )

0
R=

...
0
wnew = wold (T R)1 T diag(t)s =

0
...
s2 (1 s2 ) . . .
...
...
...
0

...
sn (1 sn )

(T R)1 T Rwold T RR1 diag(t)s = (T R)1 T Rz,

z = wold R1 diag(t)s
( )
, (
R),

T R ( m > n),
(T R + I)
!! !!
!! j (x), !!

.
. .. ,
, ,
-. , ,
.

36

 4.

37

. 4.1. . f g

4.1

f (x) : Rd R . , :
f (x) extr
x

, ( ), :
f (x) = 0
, :
f (x) extr
x

g(x) = 0

(. . 4.1)
, g(x) g(x) = 0. x x +
.
g(x + ) ' g(x) + T g(x)
.. g(x + ) = g(x), T g(x) ' 0. kk 0 T g(x) = 0. ..
g(x) = 0, g(x) .

f (x) (
f (x) ,
, ), ..:
f + g = 0
6= 0 . .

L(x, ) , f (x) + g(x)

 4.

38

x2
*

(x 1,x 2)
x1
g(x1,x2)=0

. 4.2. . .
. (x1 , x2 ) = (1/2, 1/2) .

x L = 0

L=0

f + g = 0
g(x) = 0

. (. . 4.2)
f (x1 , x2 ) = 1 x21 x22 max
x1 ,x2

g(x1 , x2 ) = x1 + x2 1 = 0
:

L(x, ) = 1 x21 x22 + (x1 + x2 1)

:
2x1 + = 0
2x2 + = 0
x1 + x2 1 = 0
: (x1 , x2 ) = ( 12 , 12 ), = 1.
(. . 4.3)

f (x) max
x

g(x) 0

g(x) > 0
g(x) = 0

f (x) = 0, x L = 0, = 0
f (x) = g(x), x, L = 0, > 0

 4.

39

. 4.3. . ,
g(x) 0, f g

:
g(x) = 0

--
fi : X R, i = 0, 1, . . . , m ,
X , A X . :
f0 (x) min;

fi (x) 0, i = 1, . . . , m, x A

(P )

1.
absmin(P ) ,
1. x
Pm
Rm+1 , L(x) = i=0 i fi (x) :
a) minxA L(x) = L(
x)
b) i fi (
x) = 0, i = 1, . . . , m
c) i 0
a)c) 0 6= 0, x
absmin(P )
2. x
a)c)
3. x
x A : fi (
x) < 0, i = 0, . . . , m
absmin(P )
( ), x

4.2

. (X, t) =
{xi , ti }ni=1 , x Rd , t T = {1, 1}.
A : Rd T ,
x t .

4.2.1

[ ., 1964]

 4.

40

. 4.4. .

xi ti qi . :
n
X
f (x) =
ti qi K(x, xi )
i=1

K(x, y) y,
x. ,
K(x, y) 0 kx yk +
K(x, y) max kx yk 0
:

f (x) + K(x, xk ), tk = 1 f (xk ) 0

f (x) K(x, xk ), tk = 1 f (xk ) 0
f new (x) =

f (x),

4.2.2

<z,x>+b=0

. 4.5. .
z

 4.

41

z b
(. . 4.5):
{x Rd |hz, xi + b = 0}, z Rd , b R
z , hz, xi x
z.
kzk.

z b ,
.
(z, b) x1 , . . . , xn Rd ,
min |hz, xi i + b| = 1

i=1,...,n

()

(*) ,
1/kzk:
xi : |hz, xi i + b| = 1, x : hz, xi + b = 0

1
|hz, xi xi| = 1
, xi x =
kzk
kzk
z b

(. . 4.6):
t(x) = sign(y(x)) = sign(hz, xi + b)
, , ..
(z, b) : t(xi ) = ti i = 1, . . . , n

(x, t) :
(z,b) (x, t) =

t(hz, xi + b)
kzk

:
(z,b) = min (z,b) (xi , ti )
i=1,...,n

.
.

 4.

42

. 4.6. ,
. .

g
g+Dg
r

(a)

(b)

. 4.7. .

.
, , .. (x, t)
(x + x, t), kxk r.
> r (. . 4.7, a).
,
(z, b) (. . 4.7, b).
. . -
p(x, t).
... .
{f (x, w) : Rd T |w } L : T T
R+ .
. :
Z
R(w) = Ep(x,t) L() = L(t, f (x, w))p(x, t)dxdt
:
n

Remp (w) =

1X
L(ti , f (xi , w))
n i=1

 4.

43

2 (Vapnik, 1995). 0 < 1 :

r
h(ln(2n/h) + 1) ln(/4)
R(w) Remp (w) +
n

()

h , -.
3 (Vapnik, 1995). , x Rd R.
:
2
R
h min
,d + 1
()
2
, , . (*) h.
-, (**) .

, , .. i, j : ti = 1, tj = 1.
, , :
1
kzk2 min
z,b
2
ti (hz, xi i + b) 1, i = 1, . . . , n


L(z, b, w) =

n
X
1
kzk2
wi (ti (hz, xi i + b) 1) min max
w
z,b
2
i=1

wi 0, i = 1, . . . , n.
n

L(z, b, w) = 0 z =
wi ti xi
z
i=1
n

L(z, b, w) = 0
wi ti = 0
b
i=1

L(z , b , w) =

n
n
n X
n
X
1 XX
wi wj ti tj hxi , xj i
wi wj ti tj hxi , xj i+
2 i=1 j=1
i=1 j=1
n
X

n
X

1 XX
+
wi =
wi
wi wj ti tj hxi , xj i max
w
2 i=1 j=1
i=1
i=1

 4.

44

n
X

wi

i=1
n
X

1 XX
wi wj ti tj hxi , xj i max
w
2 i=1 j=1

ti wi = 0

i=1

wi 0, i = 1, . . . , n

t(x) = sign (hz , xi + b ) = sign

n
X

!
wi ti hxi , xi

+ b

i=1

:
wi (ti (hz , xi i + b ) 1) = 0
, , wi > 0,
. (. . 4.8). b

. 4.8. . , , ( ). .

.
.

t(x) = sign

n
X

!
wi ti hxi , xi + b

i=1

, wi > 0 ( ).
(sparse model).
, , .

 4.

4.2.3

45

, , . ,
, .
.
, (z, b) xi : (z,b) (xi , ti ) < 0.
:

ti (hz, xi i + b) 1 i
ti (hz, xi i + b) 1 i = 1, . . . , n
i 0, i = 1, . . . , n
, ( )
:
n
X
1
kzk2 + C
i min
z,b,
2
i=1

n

X
1
kzk2 + C
i min
z,b
2
i=1
ti (hz, xi i + b) 1 i i = 1, . . . , n
i 0
C 0 ,


L(z, b, , w, v) =

n
n
n
X
X
X
1
kzk2 + C
i
wi [ti (hz, xi i + b) 1 + i ]
vi i min max
z,b, w,v
2
i=1
i=1
i=1

wi 0, vi 0.
n
X

L(z, b, , w, v) = 0
z

z =

L(z, b, , w, v) = 0
b

L(z, b, , w, v) = 0
i

wi + vi = C

wi ti xi

i=1
n
X

wi ti = 0

i=1

 4.

46

n
X

wi

i=1
n
X

1 XX
wi wj ti tj hxi , xj i max
w
2 i=1 j=1

wi ti = 0

i=1

0 wi C
:
t(x) = sign (hz , xi + b ) = sign

n
X

!
wi ti hxi , xi

+b

i=1

4.2.4

, .
,
hxi , xj i.

0.8

0.9

0.6

0.8

0.4

0.7

0.2

0.6
X22

X2

,
(. . 4.9):
: Rd H

0
0.2

0.4

0.4

0.3

0.6
0.8
1
1

0.5

0.2
0.1

0.8

0.6

0.4

0.2

0
X

0.2

0.4

0.6

0.8

0.1

0.2

0.3

0.4

0.5

X21

(a)

(b)

0.6

0.7

0.8

0.9

. 4.9. . . (a)
. (x1 , x2 ) (x21 , x22 ) .

 4.

47

, H h(xi ), (xj )iH .
, K : Rd Rd R,
K(x, y) = h(x), (y)iH
H , K!
,

t(x) = sign

n
X

!
wi ti h(xi ), (x)iH + b

= sign

i=1

n
X

!
wi ti K(xi , x) + b

i=1

, K (H, ), K
. :

K(x, y) = K(y, x)
( )
Z
g(x) :
g 2 (x)dx <
Z
K(x, y)g(x)g(y)dxdy 0
K H .


K(x, y) = hx, yi + , 0

K(x, y) = (hx, yi + )d , 0, d N

kx yk2
K(x, y) = exp
, >0
2 2

K(x, y) = tanh(hx, yi + r), r R
!

 4.

4.2.5

48

(. . 4.10)
2.5

2.5

2.5
2

1.5

1.5

1.5

0.5

0.5

0.5

0.5

0.5

0.5

1.5

1.5

1.5

2
3

2
3

(a)

2
3

(b)

(c)

. 4.10. . (a)
C =
1, 2 = 0.1, (b) C = 1, 2 = 2, (c) C = 1, 2 = 1000

(. . 4.11)

(a)

(b)

(c)

. 4.11. . (a)
C = 102 , (b) C = 1, (c) C = 105

SVM
, ( , ) . , ,
( ).
SVM , ,
SVM (, SM O SV M light ).
. http://www.kernel-machines.org
SVM

 4.

49

X
1
kzk2 + C
i min
z,b,
2
i=1
ti (hz, xi i + b) 1 i
i 0
ti y(xi ) 1, i = 0. i = 1 ti y(xi ). ,

n
X
ESV (ti y(xi )) + kzk2
i=1
1

= (2C)

, ESV () ,
ESV (s) = [1 s]+

SVM vs.
4
3.5
3
2.5
2
1.5
1
0.5
0
2.5

1.5

0.5

0.5

1.5

2.5

. 4.12. . , ,

:
n
X
i=1

ELR (s) = log(1 + exp(s)).

SVM:

n
X
i=1

ELR (ti y(xi )) + kwk2 min

w

ESV (ti y(xi )) + kzk2 min

z

SVM
+ ,
+

C

 4.

4.3

50

vs. SVR
3

2.5

1.5

0.5

0
3

. 4.13. . ,

1X
1
(ti y(xi ))2 + kwk2 min
w
2 i=1
2
, , :

0,
|t y(x)| <
E (t y(x)) =
|t y(x)| ,
:
C

n
X

1
E (y(xi ) ti ) + kzk2 min
z
2
i=1

e
x
x*

. 4.14. . , -,
i , , -, i

 4.

51

n
X

1
(i + i ) + kzk2 min
2
z,b,,
i=1

ti y(xi ) + + i
ti y(xi ) i
i , i 0

X .

n
n
n
X
X
1 X
(wi wi )(wj wj )K(xi , xj )
(wi + wi ) +
ti (wi wi ) max
w,w
2 i,j=1
i=1
i=1

n
X
(wi wi ) = 0
i=1

0 wi , wi C

y(x) =

n
X

(wi wi )K(x, xi ) + b

i=1

wi=C
w*i=0

0< wi<C
w*i=0
e
e

wi=0
w*i=0

wi =0
0< w*i <C

wi=0
w*i=C

. 4.15.

wi ( + i ti + hz, xi i + b) = 0
wi ( + i + ti hz, xi i b) = 0
(C wi )i = 0,

(C wi )i = 0

, -, hz, xi + b

 4.

4.4

52

4.4.1

, ,
, . :



:
, 1 , . . . , n
, ..

1 , 2 = 1 + 2
1 , c R = c1 ,
:
1.
2.
3.
4.
5.
6.
7.
8.

1 + 2 = 2 + 1
1 + (2 + 3 ) = (1 + 2 ) + 3
: + = + =
() : + () =
c(1 + 2 ) = c1 + c2
(c + d) = c + d
(cd) = c(d)
1 =

K : R, :
1. K(1 , 2 ) = K(2 , 1 )
2. K(, ) 0, = 0 =
3. K(1 + 2 , ) = K(1 , ) + K(2 , )
4. K(c1 , 2 ) = cK(1 , 2 )

 4.

t() = sign(K(, ) + b)

Pn
+ C i=1 i min
,b,
ti (K(, i ) + b) 1 i
i 0

Pn
Pn
1
i=1 wi 2
i,j=1 ti tj wi wj K(i , j ) max
w
Pn
i=1 ti wi = 0
0 wi C

Pn
t() = sign ( i=1 ti wi K(i , ) + b)

53

1
2 K(, )

, , K

K. ,

4.4.2

K. . [, 2001]

. 4.16. . ,

.
: y 0 = y( 0 ) = (yt0 , t T )
y 00 = y( 00 ) = (yt00 , t T ), T = {t = (t1 , t2 ), t1 = 1, n1 , t2 = 1, n2 }
:
P
K(y 0 , y 00 ) = hy 0 , y 00 i = tT yt0 yt00
K(y 0 , y 00 ) = [hy 0 , y 00 i + 1]

K(y 0 , y 00 ) = exp ky 0 y 00 k2 = exp ([hy 0 , y 0 i + hy 00 , y 00 i 2hy 0 , y 00 i])

 4.

54

t t + xt (. . 4.16).
:
X
0
K(y 0 , y 00 ) =
yt0 yt+x
t
tT

K
A : A A R:
1. (1 , 2 ) 0, = 0 1 = 2
2. (1 , 2 ) = (2 , 1 )
3. (1 , 3 ) (1 , 2 ) + (2 , 3 )
A A :

1 2
(1 , ) + 2 (2 , ) 2 (1 , 2 )
(1 , 2 ) =
2

1. (1 , 2 ) = (2 , 1 )
2. (, ) 0 = 0 =
3. A (, ) = 0
4. (, ) = 2 (, )
5. (1 , 2 ) = ( (1 , 1 ) + (2 , 2 ) 2 (1 , 2 ))1/2
6. (1 , 2 ) 12 [ (1 , 1 ) + (2 , 2 )]
p
p
7. | (1 , 2 )| (1 , 1 ) (2 , 2 )
(2 , )
+ (, )

8. (1 , 2 ) = (1 , 2 ) (1 , )

!

{1 , . . . , q } A
A. C :
M = ( (i , j ), i, j = 1, . . . , q)
, .
, ,
4. M , M .

 4.

55

c>1
a
a2

0<c<1
a

c<0

a1
a

. 4.17.

A [1 , 2 ]
c R = coax([1 , 2 ]; , c),
(1 , ) = |c|(1 , 2 ),

(2 , ) = |c 1|(1 , 2 )

, 1 = coax([1 , 2 ]; 0), 2 = coax([1 , 2 ]; 1). = coax([1 , 2 ]; c), =

coax([2 , 1 ]; 1 c). [1 , 2 ] = coax([1 , 2 ]; c)
:
(1 , 2 ) + (2 , ) = (1 , ), c > 1
(1 , ) + (, 2 ) = (1 , 2 ), 0 < c 1
(, 1 ) + (1 , 2 ) = (, 2 ), c 0

A ,
[1 , 2 ], 1 , 2 A c R A : = coax([1 , 2 ]; c) A .
:

1
c , coax([, ]; c) 1 + 2 = 2 coax [1 , 2 ];
2

ca1
a1+a2

a1

a=coax([a1,a2];1/2)
a2
f
. 4.18.

 4.

56

5.
, h1 , 2 i = (1 , 2 ) :

1 + 2 = 2 + 1 , (1 + 2 ) + 3 = 1 + (2 + 3 )
+ = , c =
() : () + =
c1 (c2 ) = (c1 c2 )
1 =
(c1 + c2 ) = c1 + c2 , c(1 + 2 ) = c1 + c2
h1 , 2 i = h2 , 1 i, hc1 1 + c2 2 , 3 i = c1 h1 , 3 i + c2 h2 , 3 i
h, i 0, =p
0=
k1 2 k = h1 2 , 1 2 i = (1 , 2 )

.
. , . (
) , .

57

 5.

5.1

58

A B,

a A p(a). l(a)
B
,
, ..
X
EA l(a) =
p(a)l(a) min
aA

. a p(a),

a l(a) = logB p(a)
B . B
(, ), ,
, ,
2.7182... :)

: , ,
( ) - ,

5.2
5.2.1

A(w) , w
,
A()
,
P (w) A(.)

,
P (w) (, ) A
, , ,

 5.

59

, (,
). , ,

,




,
,
: ( II), ( , , 20
)
, , . ,
, ,
- ...

5.2.2

y(x) = sign

n
X

!
wi K(x, xi ) + b

i=1

n
X
i=1
n
X

wi

1 XX
ti tj wi wj K(xi , xj ) max
2 i=1 j=1

ti wi = 0

0 wi C

i=1

C K(x0 , x00 )
(. . 5.1, 4.10, 4.11)

 5.

60

. 5.1. SVM

y(x) =

m
X

wj j (x)

j=1

1 T
w = T + I
t
0, {j (x)}m
j=1 m

(a)

(b)

(c)

. 5.2.

,
(. . 5.3)

 5.

61

(a)

(b)

. 5.3. K (a) (b)

( )

xD

zM
(1)
wMD
(2)
wKM
yK

y1

x1
(2)

x0

z1

w10

z0

. 5.4. .

5.3
5.3.1

?
,
(, )

 5.

62

(-)
,
, ,



k-fold cross validation
k ( ) .
k 1 ,

. 5.5. 4-fold cross validation. ,

,

5.5 4-fold cross validation

k = n leave-one-out
5 2-fold cross validation.

( )

- ,
..

, ..

 5.

5.3.2

63

- ( )
.. ( -, VC-dimension)
( ) : ,
, ,



( ,) ,
,

, , . h = d+1, d

: n d + 1

, , .

( ) Ptrain (w), h() ( ) Ptest (w)
r
h()(log(2n/h()) + 1) log(/4)
Ptest (w) Ptrain (w) +
n
1 w
, ,



,
( , , ..)

 5.

64

( )
, -
,
,
, ,

,

5.3.3

.

1: {(xi , ti )}ni=1
2: ,
Descr(A)
3: Descr(A0 ) , , {xik , tik }pk=1 , p < n

, , ...
... (. . 5.6)
(minimum decription length MDL, Rissanen, 1978)

. 5.6.

 5.

65

, , ..
, ,
MDL
p(w)
l(w) = log p(w)
, , .. p(t|X, w)
l(t|w) = log p(t|X, w)
,
l(t, w) = log p(t|X, w) log p(w)
arg min l(t, w) = arg max p(t|X, w)p(w)
w

, MDL
MDL
MDL
MDL , , .. MDL
,
MDL , , . ( , boosting) ,

5.3.4

1973. ( ) - ( )
(.. , c )
AIC = log p(t|X, wM L ) M,
M
: k

Pn
2
i=1 (ti yk (xi ))
+
k
+
1
k = arg min
2 2

 5.

66

( ) ,
Z
BIC p(t|X, w)p(w)dw
,
1
BIC = log p(t|X, wM P ) M log n
2
: k
Pn

i=1 (ti yk (xi ))

k = arg min
+ (k + 1) log n
2 2

( ) , .
M , , .
( )

wM P ,
wM L