..

2013

 005.519.8(075.8)
65.290-2673

.. .: 2013. - 387 .
(Machine Learning) .
,

, .
- :
(Prediction with Expert Advice).

, , .
. 48.
c .., 2013

17
18

1.1. . . . . . . . . . . . . . . . . . .

19

1.1.1. . . . . . .
1.1.2. . . . . . . . . . .
1.1.3. : . . . .

19
23
26

1.2. . . . . . . . . . . . . . . . . . . . .

33

1.2.1.
. . . . . . . . . . . . . . . . .
1.2.2. VC- . . . . . . . . . . . . . . . . . .

33
44

1.3. . . . . . . . . . .

55

1.3.1. . . .
1.3.2. . . . . . . . . . . . . . .

55
62

1.4. . . . . . . . . . . . . . . . . .

70

1.5. . . . . . . . . . . . . . . . . . . . . . . . .

79

1.6. . . . . . . . . . . . . . . . . . .

84

88

2.1. . . . . . . . . . . . . . .

88

2.2.
. . . . . . . . . . . . . . . . . . . . .

93

2.3.
. . . . . . . . . . . . . . . . . . . .

96

2.4. SVM- . . . . . . . . .

97

2.5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
2.5.1. . . . . . . . 105
2.6. . . . . . . . . . . . . . 113
2.6.1. . . . . . 113
2.6.2.
. . . . . . . . . . . . . . . . . . . . 117
2.7. . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
2.8. . . . . . . . . . . . . . 131
2.8.1. . . . . . . . . . . 131
2.8.2. . . . . . . . . . . . . . . . 134
2.9.

. . . . . . . . . . . 137

2.9.1. SVM . 137

2.9.2. . 143
2.10. . . . . . . . . . . . . . . . . 147
2.11. . . . . . . . . . . . . . . . 152
2.12. . . . . . . . . . . . . . . . . . . 156
2.13. SVM . . . . . . . . . . 158

II

162
163

3.1. . . 163

3.2. . . . . . . . . . . . . . . . . 167
3.3. . . . . 173
3.4. . . . . . . . 177
3.5. 184
3.5.1. 190
3.5.2. 3.4 . . . . . . . . . . 202
3.6. . . . . . . . . . . . . . . . . . . 207
3.7. . . . . . . . . . . . . . . . . . . 207
4
209
4.1. . . . . . . . . . . 210
4.2. . . . . . . . . . . . . . . . . . . . . . . . 215
4.3. . . . 220
4.4. . . . . . . . . . . . . . . . . . . . . . . . 232
4.5. . . . . . . . . . . . . . 238
4.6. . . . . . . . . . . . . . 240
4.7. . . . . . . . . 247
4.8. . . . . 253
4.9. . . . . . . . . . . . . . . . . . . 261
4.10. . . . . . . . . . . . . . . . . . . 261
5

264

5.1. . . . . . . . . . . . . . 264
5.2. . . . . . . . . . . . . . 271
5.3. . . . . . . . . . . . 277

5.4. . . . . . . . . . . . . . 280
5.5. . . . . . . . . . . . 281
5.6. . . . . . . . . . . . . . 285
5.7. . . . . . . . . 287
5.8. . . . . . . . . . . . . . . . . 291
5.9. . . . . . . . . . . . . . 295
5.9.1.
. . . . . . . . . . . 295
5.9.2. . . 302
5.9.3. . . . . . 305
5.10. . . . . . . . . . . . . . . . . . . 306
5.11. . . . . . . . . . . . . . . . . . . 307

III

309

310

6.1. . . . . . . . . 310
6.2.
. . . . . . . . . . . . . . . . . . . . . . 313
6.3. . . . . . . . 316
6.3.1.
6.3.2.
6.3.3.
6.3.4.
6.3.5.

. . . . . . . . . .
. . . . . . . . . . . .
(2 M )
(N M ) . . . . . .
K . .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

316
318
321
324
326

6.4. . . . . . . . . . . . . . . . . . . 332
7 -
333
7.1. - . . . . . . . 333

7.2. - . . . . . . . . . . . . . 339
7.3. . . . . . . . . . 346
7.4. . . . 351
7.5. . . . . . . . . . . . . . . . . . . 357
8

360

8.1. . . . . . . . . . . . . . . . . . . . . . . . 361
8.2. . . . . . . . . 366
8.3. . . . . . . . . . . . . . . . 373
8.4. . . . . . . . . . . . . . . . . . . . . . . . . . . 378
8.5. . . . . . . . . . . . . . . . . . . 385

385

.
, ,
. ,

, . , . ,
. ,
, (Machine Learning).

,
.
- .
,
.

 . , ,
, . , ,
.

. ,
, , .
.

.
. .
,
.
,
, , ..

.

, ,
.
, , ..
. , . , -

 ( )
.
,

.
,
. , .
.
, , , . .
, , ,
,

. ,

;
,
.
. ,
, ,
.
(. [2]).
.
,

10

 () .
() ().

, ,
.

1.
. (VC-) 60- 20- . (fat),
,
. .
VC-
.
2
. , , .
,
.
, (Reproducing Kernel Hilbert Space RKHS),
.
. , . -

11

, , , (..,
).

.

.
( 3) , .
,
.
.
, .

.
3.5

. . ,

.
( )
4.
,
. ,
.
, . , , ..
, . -

12

 ,
. .
. ( regret)
.
4 .
, .
, 4, . 4.3 ,
.
4.8
. , .
4.8 (Boosting). AdaBoost, . AdaBoost
,
.
5 . , , , ,
4. , .
.
.
6.

13

 . . .
.
7 -
, [31].
,
, ,
.
, ,
.. -
, .

,
3. . ,
.
, ;
. . . , ,
.
. ,
, , . ,
, ,
, .
-

14

 .
8 . (Blackwell approachability theorem).
. .
, , ,
.


, - , - . ,
,
.
, 20082013 - ()
.
[3].
1 2 . 3, 4 5
, , 3, 6, 7 8
.
,
, . -

15

 : , , , ,
, ,
.
.

16

17

,
, ,
.
:
(supervised learning), (unsupervised learning) ( , ), (reinforcement learning).

.
,
.

.

.
.
,
.
,
18

 .
: .
, ,
.

1.1.
1.1.1.
, ,
, .
PAC- (Probably Approximately Correct-learning),
[36].
PAC- .
.
, x,
, X ( ) P X .
, P .
, x
.
D
c C c : X D, : c(x)
x.
,
S = ((x1 , c(x1 )), . . . , (xl , c(xl )),
P , h = hS ,
x ( X ), c.
19

 h
errP (h) = P {h(x) 6= c(x)}.
errP (h) ,
h = hS S .
: h, , errP (h)
, . , , h (probably
approximately correct).
: errP (h) 6 .

.
c C . , ,
errP (h) 6 
1 .
PAC- .
A C
H, c C,
P x,
 (0, 1/2) (0, 1/2), :
A , (x, c(x))
P ,
1/ 1/;
A h,
errP (h) 6  1 .
, C PAC (PAC-learnable).

PAC20

 . , , , (x, y) x y P
X D.
.
PAC- .
, S = ((x1 , y1 ), . . . , (xl , yl )) () . , S,
, (x, y), .. X D P , (xi , yi ),
S, .
(X D)l
P l = P P P .
() h : X D, xi X
. h , .
D = {1, 1}, h : X D
.
S :
S + = ((xi , yi ) : yi = 1) (
) S = ((xi , yi ) : yi = 1) (
).

h f
rR :

1, f (x) > r,
h(x) =
1 .
, (x, y) (X, Y ),
P . P
P (x, y).
21

 h ,

errP (h) = P {h(X) 6= Y } = P {(x, y) : h(x) 6= y}.
h(X) X,
,
{h(X) 6= Y }.
errP (h) -.
H , .. h H,
errP (h)
H.
n- Rn ,
R . D , D
. n Rn
.
Rn
: x
, y, . . . Rn ; x
= (x1 , . . . , xn ).

x1 + y1
x2 + y2

x
+ y =

...
xn + yn

x1
x2

x=
...
xn

x
= (x1 , . . . , xn )0 y = (y1 , . . . , yn )0 .
1

,
.

22

 Rn (
x y) = x1 y1 + + xn yn . s
() x

n
p
P
x2i . k
xk =
(
xx
) =
i=1

S =
((
x1 , y1 ), . . . , (
xl , yl )), x
i X n
R n (, - ), yi D ( ),
, yi {1, 1}. yi D x
i .
S = ((
x1 , y1 ), . . . , (
xl , yl )),
yi , ..
D = R. 2.8, 2.9,
5.9.

1.1.2.
. , P , (xi , yi ), .
(X, Y ), X {1, 1}. ,
P P (x, y). , P (x|Y = 1) , P (x|Y = 1)
. P {Y = 1}
P {Y = 1}
.
R P (x, y). , P {Y = 1} = P (x, 1)dx, P (x|Y = 1) =
X

P (x, 1)/P {Y = 1}.

2

, P {Y = 1} > 0 P {Y = 1} > 0,

23

 , x

P {Y = 1|X = x} = cP (x|Y = 1)P {Y = 1),
P {Y = 1|X = x} = cP (x|Y = 1)P {Y = 1),

c=

1
,
P (x|Y = 1)P {Y = 1} + P (x|Y = 1)P {Y = 1}

, x
(x) = P {Y = 1|X = x}.
g : X {1, 1}

errP (g) = P {g(X) 6= Y }.


1, (x) > 12 ,
h(x) =
1 ,
,
errP (h), , .
1.1. g : X {1, 1}
P {h(X) 6= Y } 6 P {g(X) 6= Y }.

(1.1)

. g X = x
P (x|Y = 1) > 0 P (x|Y = 1) > 0.

24


P {g(X) 6= Y |X = x} =
= 1 P {g(X) = Y |X = x} =
= 1 (P {Y = 1, g(X) = 1|X = x} +
+P {Y = 1, g(X) = 1|X = x}) =
= 1 (1g(x)=1 P {Y = 1|X = x} +
+1g(x)=1 P {Y = 1|X = x}) =
= 1 (1g(x)=1 (x) + 1g(x)=1 (1 (x))}),
R(x) 1R(x) (x) = 1, R(x) , 1R(x) (x) = 0, .
h(x).
, 1g(x)=1 = 1 1g(x)=1 g. , x X
P {g(X) 6= Y |X = x} P {h(X) 6= Y |X = x} =
= (x)(1h(x)=1 1g(x)=1 ) +
+(1 (x))(1h(x)=1 1g(x)=1 ) =
= (2(x) 1)(1h(x)=1 1g(x)=1 ) > 0
h.
x. . 4
.
D g : X {1, 1}. (1.1)

errP (h) = P {h(X) 6= Y } = inf P {g(X) 6= Y }.
gD

gl D A S = ((x1 , y1 ), . . . , (xl , yl )) P . A P , errP (gl ) errP (h)

25

P , ..  > 0
P {|errP (gl ) errP (h)| > } 0

(1.2)

l .
A , (1.2) P .
,
h(x) P , (x, y). , P . . (X, Y ).

, h(x)
,
(x, y).
, , , .
.

1.1.3. :

.
. .
.
. X 26

. y = (
x), yi = i (
x), i = 1, . . . , n,
x
= (x1 , . . . , xm ) X
y = (y1 , . . . , yn ) Y. , X Rm Y Rn
m, n.

L(
x) = ( (
x)) =

n
X

i i (
x) =

n
X

i=1

i yi ,

i=1

i , yi . ( (
x)) = (1 , . . . , n )
(
x) = (1 (
x), . . . , n (
x)) Rn .
:
!
n
X
f (
x) =
i i (
x) .
i=1

:
(t) = sign(t),
1
(t) =
,
1 + et
et et
(t) = t
,
e + et


sign(t) =

1, t > 0,
1, t 6 0.

(t) = sign(t), : , x
,
n
X
i i (
x) > 0,
i=1

, x
.
27

 , X

n
X

i i (
x) = 0,

(1.3)

i=1

X .
,
.
.
(1.3)

n
X

i yi = 0

i=1

Y. Y
.

S = ((
y1 , 1 ), (
y2 , 2 ), . . . ),
i yi = (
xi )
i {1, 1}, i = 1, 2, . . .
, , S. = (1 , . . . , n )
. ,
inf i ( yi ) > 0.
i

(1.4)

. y1 , y2 , . . . ,

yi , i = 1,
yi =

yi , i = 1,

28

 i. (1.4)

inf ( yi ) > 0.
i

( yi )
,
kk
0 = sup (),

() = inf
i

(1.5)

6=
0

s
kk =

n
P

i=1

2i Rn .

S 0 > 0.
.

(
y1 , 1 ), (
y2 , 2 ), . . .
, ( y) = 0, , .. ,

inf ( yi ) > 0.
i

, k k = 1.
0 > 0 ,
( yi ) > 0

(1.6)

i. , yi
sup |
yi | = D < .
i

.
29


.
t = (1,t , . . . , n,t )
, t , t = 1, 2, . . . .
y1 , y2 , . . . .
0 = (0, . . . , 0).
FOR t = 1, 2, . . .
(t1 yt ) > 0, t = t1 .
(.. , ).
(t1 yt ) 6 0 ( ),
t = t1 + yt , .
ENDFOR
, .. , , , , , ,
, .
1.1. ,

(
y1 , 1 ), (
y1 , 1 ), . . .
,
 2
D
20
.3 j,
t 6= t1
k
D2
2 t.
0

, r, brc 6 r, dre > r.

30

 ,

.
. t t ,
kt k2 = kt1 k2 + 2(t1 yt ) + k
yt k2 .
(t1 yt ) 6 0 ( t- )
k
yt k 6 D,
kt k2 6 kt1 k2 + D2 .
T k ,

kt k2 6 kD2 .

(1.7)

(1.6)
,
( yi ) > 0
i.
(t ). (0 ) = 0.
t ,
(t ) = (t1 ) + ( yt ) > (t1 ) + 0 .
t ,
(t ) = (t1 ).
, t k ,

(t ) > k0 .

(t ) 6 kt k k k = kt k.
31


kt k > k0 .

(1.8)

(1.7) (1.8),
k6

D2
.
20

,
 2
D
k6
.
20
. 4
1.1,
, ,
h 2i
D
, 2 ,
0
- .

, .. Y = {1, 1}n .
, D2 6 n. 1.1

 
n
k6 2 ,
0
.. .
x)
. y = (
X
Y. Y
.
, , y,
.

32

 ,
x), y = (
X Y .
. .
f (
x) = ((w
x
) + b ),
;
.
, l . n1 , . . . , nl ( ),
, : nl = 1.
j- i-
fi,j (
x) = ((w
i,j x
) + bi,j ), w
i,j , x
Rni1 bi,j R.

fi : Rni1 Rni ,
i = 1, . . . , l, fi = (fi,1 , . . . , fi,ni1 ).

fl fl1 f2 f1 .
w
i,j ,
(i, j) .

1.2.
1.2.1.

, , ..
.
33

 .
, (xi , yi ) , , ,
. ,
.
, , .
n-
,
, m > n. ,
.
.

.
(, , ,
) .

, .

.

.
.
-

34


.
.
S = ((x1 , y1 ), . . . , xl , yl )) .
xi X yi {1, 1} 1 6 i 6 l. X
, D .
X Rn n- .
, X D P .
, S ,
(xi , yi ), i = 1, . . . l. ( 1.1.2) (xi , yi )
.
( ) h : X {1, 1}.
( )
errP (h) = P {(x, y) : h(x) 6= y}.
.
h
S = ((x1 , y1 ), . . . , (xl , yl )),
h(xi ) = yi 1 6 i 6 l.
1
errS (h) = |{i : h(xi ) 6= yi , 1 6 i 6 l}|
l
h S.
|A| A. h S, errS (h) = 0.

35

 h  > 0
P l {S : errS (h) = 0&errP (h) > } =
=

l
Y

P {h(xi ) = yi } =

i=1

l
Y
(1 P {h(xi ) 6= yi }) =
i=1

= (1 errP (h))l 6 el .

(1.9)

.
H .
H , (1.9)
P l {S : (h H)(errS (h) = 0&errP (h) > )} 6 |H|el .

(1.10)

(1.10) .
> 0 h H,
S. (1.10) , > 1 hS H,
S ,
4
errP (h) 6  = 1l ln |H|
.
, h, errP (h) > , > 1 |H|el
l.
H

. H
BH (l) =

max
(x1 ,x2 , ..., xl )

|{(h(x1 ), h(x2 ), . . . , h(xl )) : h H}|.

.
4
ln r
r, log r r 2.

36

 (1.10) H.
1.2. l > 2/
P l {S : ( h H)(errS (h) = 0&errP (h) > )} 6 2BH (2l)el/4 .
. 1A (x) = 1, x A,
1A (x) = 0, x 6 A. 1h(x)6=y (x, y)
, 1, h(x) 6= y 0, .

E1h(x)6=y = errP (h),

E P .
l

errS (h) =

1X
1h(xi )6=yi
l
i=1

S.

.
1.2. H . S, S 0 l.
 > 0 l > 2/
P l {S : ( h H)(errS (h) = 0&errP (h) > )} 6
1
6 2P 2l {SS 0 : (h H)(errS (h) = 0&errS 0 (h) > )}.
2

(1.11)

. , (1.11)
P l {S :

sup

errP (h) > } 6

h:errS (h)=0

6 2P 2l {SS 0 :

1
errS 0 (h) > }.
2
h:errS (h)=0
sup

(1.12)

(1.12). S
(1.12) hS - H, errS (hS ) =
37

0 errP (hS ) > . ,

.
5
1errS (hS )

= 0&errP (hS ) >  1errP (hS ) errS 0 (hS ) 6

6 1errS (hS )

1

2

= 0&errS 0 (hS )> 21  .

(1.13)

S 0
(1.13).
, S :
1
{S 0 : errP (hS ) errS 0 (hS ) 6 } 6
2
1
l
0
(1.14)
6 P {S : errS (hS ) = 0&errS 0 (hS ) > }.
2

1errS (hS )

= 0&errP (hS ) >  P

1
P l {S 0 : errP (hS ) errS 0 (hS ) 6 } =
2
1
= P l {S 0 : errS 0 (hS ) > errP (hS ) } =
2
 
X
l k
1
=
p (1 p)lk >
2
k

(1.15)

{k:k/l>p/2}

l > 2/. p = errP (hS ).

, l > 2/ p /2 < p 1/l.
,
 
X l
X
1
l k
pk (1 p)nk > .
p (1 p)nk =
k
k
2
{k:k/l>p1/l}

{k:k>lp1}

X l
1
pk (1 p)nk < .
k
2
{k:k<lp1}

5
1errS (hS )=0&errP (hS )> (S) = 0, S
(1.12). 1errS (hS )=0&err 0 (hS )> 1  (SS 0 ) = 0, SS 0
S
2
(1.12).

38

 :
X l
pk (1 p)nk =
k
{k:k<lp1}
 
X
l k
p (1 p)nk =
=
k
{k:lk>l(1p)+1}
X l
pk (1 p)nk .
=
k

(1.16)

{k:k>lp+1}

(1.16) 1.
1/2.
(1.15) (1.14),
1errS (hS )=0&errP (hS )> 6
1
6 2P l {S 0 : errS (hS ) = 0&errS 0 (hS ) > }.
2

(1.17)

S
P l {S : errS (hS ) = 0&errP (hS ) > } 6
1
6 2P 2l {SS 0 : errS (hS ) = 0&errS 0 (hS ) > } 6
2
1
2l
0
6 2P {SS :
sup errS 0 (h) > }.
2
h:errS (h)=0

(1.18)

(1.12). . 4
1.3. , S S 0 l h H l

P 2l {SS 0 : (h H)(errS (h) = 0&errS 0 (h) > )} 6 BH (2l)el/2 .
. , SS 0 = ((x1 , y1 ), . . . , (x2l , y2l )) 2l

39

 , ..
(SS 0 ) = = {((x1 , y1 ), k1 ), . . . , ((xL , yL ), kL )},
ki (xi , yi ) SS 0 , i = 1, . . . , L,
L (xi , yi ) SS 0 ;
k1 + + kL = 2l.
.
P 2l 2l P :
P () = P 2l {SS 0 : (SS 0 ) },
, .
:
2l.
h;
h m
.
SS 0 = ((x1 , y1 ), . . . , (x2l , y2l ))
1 , . . . , 2l
,

1, h(xi ) 6= yi ,
i =
1, h(xi ) = yi ,
i = 1, . . . , 2l.
p = P {h(x) 6= y},
1 , . . . , 2l 01 , . . . , 02l , m
.
, SS 0 , ,

40

 , :


l
l!
(2l m)!m!
m

=

=
2l
(l

m)!m!
(2l)!
m
(2l m) . . . (l m + 1)
6
2l . . . (l + 1)

m l 
 l
6 1
6 1
< el/2 ,
2l
2
=

(1.19)

m > l.
h H,
m > l , H
{x1 , . . . , x2l } SS 0 (SS 0 ) = ,
{(h(x1 ), h(x2 ), . . . , h(x2l )) : h H}, 2l.

H :
BH (l) =

max
(x1 ,x2 ,...,xl )

|{(h(x1 ), h(x2 ), . . . , h(xl )) : h H}|.

, BH (l) 6 2l .
.
, H 2l
BH (2l).
,
H l

,
P 2l {SS 0 : ( h H)(errS (h) = 0&errS 0 (h) > |(SS 0 ) = } 6
6 BH (2l)el/2 .
( ).
.
41

 P ,

P 2l {SS 0 : ( h H)(errS (h) = 0&errS 0 (h) > } 6
6 BH (2l)el/2 .
1.3 . 4
1.2 1.2 1.3.
1.2 , > 12BH (2l)el/4
h, errP (h) > ,
l > 2/, ..
.
= 2BH (2l)el/4 . 0 < < 1 l > 2, .. 1.2 .
1.1. , H VC- d. 6
0 < < 1 h H, S.
l > d > 1 hS H, S
,


4
2el
2
errP (hS ) 6
d ln
+ ln
.
l
d

. 1.4 1.5, 1.3.

1.4. H . S, S 0 l.
 > 0 l > 2/2
P l {S : ( h H)(errP (h) errS (h) > )} 6
1
6 2P 2l {SS 0 : ( h H)(errS 0 (h) errS (h) > )}.
2
6

VC- 1.2.2.

d
BH (l) 6 el
l > d.
d

42

 1.3.
1.5. , S S 0 l h H  > 0,

P 2l {SS 0 : (h H)(errS 0 (h) errS (h) > )} 6 2BH (2l)e2 l .

1.5.

.
1.3.
P l {S : ( h H)(errP (h) errS (h) > )} 6 4BH (2l)e

2 l/2

l > 2/2 .
,
.
1.2. , H VC- d, 0 < < 1 l > d.
> 1 h H
s 

2el
4
2
errP (h) 6 errS (h) +
d ln
+ ln
.
l
d

, 1.2 1.3, 1.1 1.2, ,

VC- d
l.
(. 1.9).

43

1.2.2. VC-
VC-, .
X H
X . h H
(x1 , . . . , xl ) X .
(h(x1 ), . . . , h(xl )),
{1, 1}, {x1 , . . . , xl }
{xi : h(xi ) = 1}
{xi : h(xi ) = 1} .
{x1 , . . . , xl } H
(shattered by H),
{(h(x1 ), . . . , h(xl )) : h H} = {1, 1}l .
H
l
, H
BH (l) =

max
(x1 ,x2 , ..., xl )

|{(h(x1 ), h(x2 ), . . . , h(xl ) : h H}|.

, BH (l) 6 2l ,
( H) l , BH (l) = 2l .
VC- ( ) 7 :
1.4. H :
1) BH (l) = 2l l, .. l l.
7

[2].

44

2) d; BH (l) = 2l l 6 d
 d
d  
X
l
el
BH (l) 6
6
(1.20)
d
i
i=0

l > d.
, GH (l) = ln BH (l) ,
,

O(d ln l) (, O( l)).
d
VC- H. d = V Cdim(H).
, VC- H
.
. , VC-
H d.
BH (l) = 2l l 6 d.
(1.20)
l + d. l = d = 1 ,
2.
, < l+d,
, l 1 d, l 1 d 1.
, l, VC d.

d  
X
l
h(l, d) =
.
i
i=0

, H
VC- 6 d BH (l) 6 h(l, d) l.
:
 
 

l
l1
l1
=
+
,
i
i
i1
, h :
h(l, d) = h(l 1, d) + h(l 1, d 1).
45

 H VC- d X1 = (x1 , x2 , . . . , xl ) l,
X2 = (x2 , . . . , xl ) , .
H1 = H|X1 H
X1 H2 = H|X2
H X2 . ,
|H1 | = |{(h(x1 ), h(x2 ), . . . , h(xl )) : h H}|
BH (l) =

max
(x1 ,x2 ,..., xl )

|{(h(x1 ), h(x2 ), . . . , h(xl )) : h H}|.

X1 = (x1 , x2 , . . . , xl ) ,
. |H1 | = BH (l).
H1 X2 ,
x1 , H2 .
H3 f
H2 , f 0 H ,
f 0 (x1 ) = f (x1 ) f 0 (xi ) = f (xi ) i = 2, . . . , l. |H3 |
f 0 .
, |H1 | = |H2 | + |H3 |, H2 H1 , f
f 0 H1 , x1 ( ),
H2 .
, VC- H2
d, H1 .
VC- H3 d 1, ,
d,
H1 x1 , x1 (
x1 H3 H1 ,
). , H1
,
d + 1. .
46


|H2 | 6 BH2 (l 1) 6 h(l 1, d)
|H3 | 6 BH3 (l 1) 6 h(l 1, d 1).

|H1 | = |H2 | + |H3 | 6 h(l 1, d) + h(l 1, d 1) = h(l, d).

X
BH (l) = |H1 | 6 h(l, d) =

d  
X
l
.
i
i=0

(1.20) .
BH (l) 6

d  
X
l
i=0


6

el
d

d

l > d, :
 d X
d    i
l
l
d
6
6
i
i
d
l
i=0
i=0
 d X
l    i
l
l
d
6
=
d
l
i
i=0
 d 
l  d
 d
l
d
l
el
=
1+
<
ed =
.
n
l
d
d
d  
X
l

(1.21)

. 4
VC- L Rn , .. h(
x) = sign(L(
x)), L(
x) .
sign(r) = 1, r > 0, sign(r) = 1, . L(
x).

47

 X = Rn n- .
L(
x) = (
ax
) + b,
x
Rn , a
Rn , b .
b = 0, sign(L(
x)) = sign(
ax
)
.
, ( ) ,
.
1.5. 1) VC-
Rn n + 1.
2) VC- Rn n.
3) Rn
!
n1
X l 1
GL (l) = ln HL (l) = ln 2
<
i
i=0

< (n 1)(ln(l 1) ln(n 1) + 1) + ln 2

(1.22)

l > n.
. . n e1 = (1, 0, . . . , 0), . . . , en = (0, 0, . . . , 1)
, ei1 , . . . , eik h(
x) = sign(L(
x)), L(
x) = a1 x1 + +
an xn , .
L(
x), , :
aij = 1 1 6 j 6 k ai = 1 i.
L(
eij ) = 1 1 6 j 6 k L(
eij ) = 1 j.
, n+1 u
1 , . . . , u
n , u
n+1
.
2n+1 a
1 , . . . , a
2n+1 , Z, zi,j = (
aj u
i ),

48

i = 1, . . . , n + 1, j = 1, . . . , 2n+1 ,

z1,1 , . . . , z1,j , . . . , z1,2n+1

...

zi,1 , . . . , zi,j , . . . , zi,2n+1

Z=

...
zn+1,1 , . . . , zn+1,j , . . . , zn+1,2n+1

j- ( ) j- , 2n+1 n + 1.
u
1 , . . . , u
n , u
n+1 n- , .. (i 6= 0 i)
1 u
1 + + n u
n + n+1 u
n+1 = 0.
a
j , j = 1, . . . , 2n+1 ,
1 , . . . , n+1 j- ,
1 (
aj u
1 ) + + n+1 (
aj u
n+1 ) = 0.
, L(
x) L(
ui ) 6= 0 i.
, 1 , . . . , n+1 .
i .
. .
. ,
n + 1
e0 = (0, 0, . . . , 0)0 , e1 = (1, 0, . . . , 0)0 , . . . , en = (0, 0, . . . , 1)0

. ,
49

 {
ei1 , . . . , eik }
h(
x) = sign(L(
x)),
L(
x) = a1 x1 + + an xn + b, x
= (x1 , . . . , xn ).
,
: aij = 1 1 6 j 6 k, ai = 1 i,
b = 21 , e0 b = 12 . L(
eij ) > 0 1 6 j 6 k L(
eij ) < 0
j.
, n+2 n-
, .
x
1 = (x1,1 , . . . , x1,n )0 , . . . , x
n+2 = (xn+2,1 , . . . , xn+2,n )0 .
, , n + 2
x
01 = (x1,1 , . . . , x1,n , 1)0 , . . . , x
0n+2 = (xn+2,1 , . . . , xn+2,n , 1)0 ,
n+1- , . x
0i1 , . . . , x
0ik , x
i1 , . . . , x
ik .
L(
x) = a1 x1 + + an xn + b
x
i1 , . . . , x
ik , .. L(
xij ) > 0 j = 1, . . . , k L(
xi ) < 0
i.
n + 1-
L0 (
x) = a1 x1 + + an xn + bxn+1 .
L0 (
x0i ) = L(
xi ) i = 1, . . . , n + 2.
L0 (
x0 ) 0
0
x
i1 , . . . , x
ik
x
01 = (x1,1 , . . . , x1,n , 1)0 , . . . , x
0n+2 = (xn+2,1 , . . . , xn+2,n , 1)0 .
50

 , n + 1- ,
n+2 , . . .
. l
x
1 , . . . , x
l . .
L(
u) = (
ux
), u
,
, x
.
Rn (u) = Rn (x) = Rn . , .
u
Rn (u)
n
L(
x) = (
ux
) R (x).
, . x
Rn (x) L(
u) = (
xu
)
n
R (u), l x
1 , . . . , x
l Rn (x) l X1 , . . . , Xl Rn (u),
.
u
Rn (u) , L(
u) = (
ux
) Rn (x), x
1 , . . . , x
l .
Rn (x),
x
1 , . . . , x
l ,
u
Rn (u).
() Rn (u), X1 , . . . , Xl , Rn (u) x
1 , . . . , x
l ,
. ,
x
1 , . . . , x
l
.

x
1 , . . . , x
l ,
Rn (x), , l X1 , . . . , Xl n-
Rn (u).
(n, l) ,

51

l X1 , . . . , Xl n-
Rn (u).
(1, l) = 2, L(x) = ux l . (n, 1) = 2,
Rn (u) .
l 1 x
1 , . . . , x
l1 n
R (x). l 1 X1 , . . . , Xl1
Rn (u),
(n, l 1) .
x
l
x
1 , . . . , x
l1 Rn (x). Xl Rn (u).
Xl ,
. .
Xl Xl .
, , Xl , , X1 , . . . , Xl1 Xl .
Xl n 1,
(n 1, l 1).

(n, l) = (n, l 1) + (n 1, l 1)

(1.23)

(1, l) = 2, (n, 1) = 2.
, (1.23)
:
l
2 l 6 n
n1
P l1

(n, l) =
l > n.
2
i
i=1

(1.22) (
n




P
l
el n
(1.20))
,
i 6 n
i=0

52

 n 6 l.
(1.21). . 4
VC-
(t) = sign(t).
F , Rn . .
F
BF (m) =

max

XRn ,|X|=m

|F|X |,

F|X , F X.
.
1.1. F 1 F 2 F =
F 1 F 2 , G = F 1 F 2 , .
m
1. BF (m) 6 BF1 (m) BF2 (m);
2. BG (m) 6 BF1 (m) BF2 (m)
. (1) ,
X , |X| = m
1
2
|F|X | 6 |F|X
| |F|X
| 6 BF 1 BF 2 .

(2) . 4
1.1.3

fi : Rni1 Rni ,
ni , i = 1, . . . , l, fi = (fi,1 , . . . , fi,ni1 )
Rni1 R.

53


f = fl fl1 f2 f1 .
F f ,
, F i
fi : Rni1 Rni F i,j , j-
.
, ,
i- , .
VC- F i,j ni1 + 1 j.
1.1, , :
BF (m) 6

l
Y

BF i (m) 6

i=1
n
l
i
YY

BF i,j (m) 6

i=1 j=1

ni 
l Y
Y
i=1 j=1

l 
Y
i=1

me
ni1 + 1

me
ni1 + 1

ni1 +1

ni (ni1 +1)

6 (me)N ,

N=

l
X

ni (ni1 + 1)

i=1

.
VC- F. m
, F. 2m 6 (me)N .
m = O(N log N ). , VC F O(N log N ).

54

1.3.

, VC-
.
, 1.2 1.1, VC-,
.
VC- ,

, , . , .
,
. .
.
,
.

1.3.1.
F
X . f F


1, f (x) > 0,
hf (x) =
1 .
S = ((x1 , y1 ), . . . , (xl , yl )) l, xi X ,
yi {1, 1}.
(xi , yi ) f i = yi f (xi ).
, i > 0 , f
55

 : hf (xi ) = yi . ,
mS (f ) = min i
i=1,..., l

f S. mS (f ) > 0 ,
f S .
 > 0. - F X = {x1 , . . . , xl } B , f F g B
, |f (xi ) g(xi )| <  i = 1, . . . , l.
N (, F, X) F X ( ).
X l
N (, F, l) = sup N (, F, X)
|X|=l

F.

1
errS (f ) = |{(xi , yi ) : yi f (xi ) 6 0}|
l
f
S = ((x1 , y1 ), . . . , (xl , yl )).
P X {1, 1} S. f :
errP (f ) = P {yf (x) 6 0}.
1.2.
1.6.  > 0 > 0
P l {S : ( f F)(errS (f ) = 0&mS (f ) > &errP (f ) > )} 6
6 2N (/2, F, 2l)el/4
l > 2/.
56

 1.6 1.2. errS (f ) = 0

mS (f ) > ( (1.11) 1.4). ,
1.6. l > 2/
P l {S : ( f F)(errS (f ) = 0&mS (f ) > &errP (f ) > )} 6

6 2P 2l {S S : ( f F)(errS (f ) = 0&mS (f ) > &errS (f ) > )}.
2
1.4.
1.5
1.7. l > 2/

P 2l {S S : ( f F)(errS (f ) = 0&mS (f ) > &errS (f ) > )} 6
2
6 N (/2, F, 2l)el/4 .
/2- B g B F S S.
f F /2. mS (f ) > ,
mS (g) > /2. , errS (f ) = 0 mS (f ) > ,
errS (g) = 0.
f xi , ..
yi f (xi ) 6 0, yi g(xi ) < /2. errS (/2, g) i, yi g(xi ) < /2, xi
S .

P 2l {S S : ( f F)(errS (f ) = 0&mS (f ) > &errS (f ) > )} 6
2


2l

6 P {S S : ( g B)(errS (g) = 0&mS (g) > &errS (/2, g) > )}.

2
2
1.5. ,
g B S
: errS (g) = 0, , mS (g) > /2,
57

 ,
6 /2 errS (/2, g) > 2 .
1.5, , (1.19).


&errS (/2, g) > )} 6
2
2
6 |B|el/4 6 N (/2, F, 2l)el/4 .

P 2l {S S : ( g B)(errS (g) = 0&mS (g) >

1.6 1.7 1.6. 4

= 2N (/2, F, 2l)2l/4 . 1.6 el/4 < 2l/4
1.3. F
> 0. P X
{1, 1}, l 0 < < 1 1
f F, S
mS (f ) > ,


4
2
errP (f ) 6
log N (/2, F, 2l) + log
.
l

. F fat-
(fat-shattered dimension). > 0. X = {x1 , . . . , xl }
-, r1 , . . . , rl () ,
E X fE F , fE (xi ) > ri + ,
xi E, fE (xi ) 6 ri , x
i 6 E.
X - ,
ri = r i.
fat (F) F -
X. F
> 0. F , - .
1.10,
1.3.2.
58

 1.7. F Rn [a, b],

a < b. 0 < < 1 d = fat/4 (F).
log N (, F, l) 6 1 + d log

2el(b a)
4l(b a)2
log
.
d
2

1.7 1.3 .
1.4. F [1, 1], > 0, > 0 P , S. 1
f F, mS (f ) > ,



4
16el
128l
2
errP (f ) 6
d log
log 2 + log
,
l
d

d = fat/8 (F).
() .
1.8. X R n- : X = {
x : k
xk 6 R}, F
f (
x) = (w
x
), kwk
61x
X.
 2
R
fat (F) 6
.

. , Y = {
x1 , . . . , x
l }
- F r1 , . . . , rl .
Y Y . ,

X
X
ri >
ri .
x
i Y

x
i Y \Y

59

 w,
kwk
6 1,

(w
x
i ) > ri + x
i Y (w
x
i ) 6 ri x
i 6 Y .
X
X
ri + |Y |
(w
x
i ) >
x
i Y

x
i Y

(w
x
i ) 6

ri |Y \ Y |.

x
i Y \Y

x
i Y \Y

X
X
(w
x
i )
(w
x
i ) > l.
xi Y

(1.24)

xi Y \Y

X
X
(w
x
i )
(w
x
i ) =
xi Y

xi Y \Y

= w

x
i

xi Y

x
i 6

xi Y \Y

2


X
X


6
x
i
x
i
2.
kwk
x Y

x Y \Y
i

(1.25)

(1.24), (1.25) |w
6 1


X

X



> l.
x

(1.26)
i
i


x Y

x Y \Y
i

,
X
x
i Y

ri <

x
i Y \Y

60

ri .

 Y Y \ Y

X
X


> l.

x

i
i



x Y \Y
x Y
i

(1.26) .
.
= (1 , . . . , l ) l; i {1, 1} i = 1 , . . . , l.
Y Y Y \ Y .
(1.26)
Y .

2

2
X

l
X

X






E
x
i
x
i = E
i x
i =


x Y

i=1
x Y \Y
i

=E

l
X

i2 k
xi k2 + 2E

i=1

l
X

i j (
xi x
j ) =

i,j=1,i6=j

=E

l
X

k
xi k2 6 R2 l.

i=1

Y ,
:


X

X

6 R l.
x

i
i


x Y

x Y \Y
i

(1.26) R l > l. l 6 (R/)2 . , fat (F) 6 (R/)2 . 4

1.8 .
,
61

S 0 = ((
x01 , y1 ), . . . , (
x0l , yl )), x
0i = (
xi , 1) 1 6 i 6 l.
0
0
f (
x ) = (w
0 x
0 ), w
0 = (w,
b),
0
S ,
f (
x) = ( w
x
) + b S.
1.8 1.4 .
1.9. f (
x) = (w
x)+b L, x
Rn , kwk
= 1.
> 0 0 < < 1.
P , R
S = ((
x1 , y1 ), . . . , (
xl , yl )), 1 f L
mS (f ) >

4
l

errP (f ) = P {yf (
x) 6 0} 6

el
128l
2
log
log 2 + log
.
2
4R2

64R2

(1.27)

, (1.36) 1.8 d
d0 = Sdim(F /2 ).
2
d = 64R
.
2
1.8 1.9

2.4
2.6.1.

1.3.2.
.
(X , d) , d(x, y)
x, y X .
A X B A > 0. , B - A, a A b B , d(a, b) < .
62

A
Nd (, A) = min{|B| : B - A}.

(1.28)

, B X ,
a, b B , a 6= b, d(a, b) > .
A
Md (, A) = max{|B| : B A -}.

(1.29)

.
1.8. A X > 0
Md (2, A) 6 Nd (, A) 6 Md (, A).
. M 2- A
N - A. N
a M b N , d(a, b) < . a, a0 M
b, b0 N , b
b0 , b = b0 d(a, a0 ) 6
d(a, b) + d(b, a0 ) < 2. , M . , |M | 6 |N |. .
M - A. , M - A.
, . x A ,
M x .
x M
M {x} A, -. M .
. 4
1.7.
.
X B = {0, 1, . . . , b} . F B X ,
63

 X
B. F
l(f, g) = sup |f (x) g(x)|.
xX

f, g F (2-), l(f, g) > 2.

x X , |f (x) g(x)| > 2.
F , f, g F .
X = {x1 , . . . , xn } X () F B X .
F X,
s = (s1 , . . . , sn } B ,
E X fE F ,
xi E = fE (xi ) > si + 1
xi 6 E = fE (xi ) 6 si 1
i.
, F X s. X
F
F Sdim(F).
, f : X [0, 1] ,
. > 0


f (x)

f (x) =

x, [r] r, ,
; , |r [r]| 6 0.5.8
F = {f : f F}.
, f
{0, 1, . . . , [1/]}.
8

r , .

64

, F F .
Nd (, A) Md (, A)
(1.28) (1.29).
F, X = {x1 , . . . , xn }, :
lX (f, g) = max |f (xi ) g(xi )|.
16i6n

:
N (, F, X) = NlX (, F),
M(, F, X) = MlX (, F).

N (, F, n) = sup N (, F, X),
|X|=n

M(, F, n) = sup M(, F, X).

|X|=n

.
1.9. F [0, 1]X > 0.
Sdim(F ) 6 f at/2 (F),
M(, F, X) 6 M(2, F

/2

, X)

(1.30)
(1.31)

.

.
1.10. X , |X | = n B =
{0, 1, . . . , b}. F B X d = Sdim(F). 9
MlX (2, F, n) 6 2(n(b + 1)2 )dlog ye ,
y =

d
P
i=1

n
i

bi , lX X F.

dre r.

65

. t(h, n), h n . , n > 1 h > 2. t(h, n)

.
F F, h .
.
F
X X s. , F (X, s),
, F X s.
t(h, n) :
t(h, n) = max{k : F F, |F | = h, F
F k (X, s)}.
F
h, t(h, n) = .
1.11. t(h, n) > y Sdim(F) 6 d, Ml (2, F) < h,
d


P
n i
y =
i b .
i=0

. , Ml (2, F) > h. ,
F F > h.
t(h, n) > y, F - y (X, s).
Sdim(F) 6 d, F (X,

s),
|X| 6 d. X i ni , < bi
s i (- X s
0 b). , F
d  
X
n i
b =y
i
i=0

(X, s) . . 4
1.11 , , 1.10, ,


t 2(n(b + 1)2 )dlog ye , n > y,
(1.32)
66

 y =

d
P
i=0

n
i

bi .

, (1.32)
.
1.12.
t(2, n) > 1 n > 1,

(1.33)

t(2mn(b + 1) , n) > 2t(2m, n 1) n > 2, m > 1.

(1.34)

. f g,
|f (x)g(x)| > 2 x, ..,
{x}. , t(2, n) > 1, ..,
(1.33) .
(1.34) F , - 2mn(b + 1)2 .
, t(2mn(b + 1)2 , n) =
(1.34) . F {f, g}.
mn(b + 1)2 . P .
{f, g} P (f, g)
x, |f (x) g(x)| > 2.
x X , i, j B, j > i + 2,
bin(x, i, j) = {{f, g} P : (f, g) = x, {f (x), g(x)} = {i, j}}.



b+1
n
< n(b + 1)2 /2.
2
, 1.10 |X | = n.
(f, g) mn(b + 1)2 , x , i j > i + 1 ,
|bin(x , i , j )| > 2m.

F1 = {f bin(x , i , j ) : f (x ) = i } ,
F2 = {g bin(x , i , j ) : g(x ) = j } .
67

 A, A , .
, |F1 | = |F2 | > 2m. F1 ,
X \ {x }. , F , F1 X ,
f, f 0 F1 |f (x0 ) f 0 (x0 )| > 2
x0 , x0 X \{x }, x . , F2
X \ {x }.
, U V
> t(2m, n 1), (X, s) ,
F1 U F2 V .
, U V
F.

(X, s) U V . ({x } X, (b i +j
2 c, s)) F . ,
f F1 g F2 , X,
f (x ) = i g(x ) = j , j > i + 2.

g(x ) = j > i +j
+ 1 f (x ) = i 6 i +j
1.
2
2
.
, E {x } X = {x , x1 , . . . , x|X| }
f (xi ) > si + 1 xi E, f (xi ) 6 si 1 xi 6 E s = (s1 , . . . , sn1 ). , g(xi ) > s0i + 1
xi E, g(xi ) 6 s0i 1 xi 6 E
s0 = (s01 , . . . , s0n1 ). f F1 g F2 . f E

s1 = (b i +j
), x 6 E,
2 c, s1 , . . . , sn1

0
0
g E s2 = (b i +j
2 c, s1 , . . . , sn1 ),

x E.
, F
|U V | + |U V | = |U | + |V | > 2t(2m, n 1)
(X, s). (1.34) 1.12 . 4
1.10.
(1.33) (1.34) ,
t(2(n(b + 1)2 )r , n) > 2r t(2, n r) > 2r
68

(1.35)

 n > r > 1.
dlog ye < n, r = dlog ye (1.35)
(1.10).
dlog ye > n,
2(n(b + 1)2 )dlog ye > (b + 1)n
B
X , |X | = n.
F 2(n(b + 1)2 )dlog ye
t(2(n(b + 1)2 )dlog ye , n) = .
, (1.10) . 4
, -, -
[6].
1.10. F [0, 1]X [0, 1]. d =
fat/4 (F).

N (, F, n) 6 2 n

2
+1

)e
2 !dd log( 2en
d

. , , (1.31) 1.9,
1.10,
N (, F, n) = sup N (, F, X) 6
|X|=n

6 sup M(, F, X) 6
|X|=n

6 sup M(2, F

/2

, X) = M(2, F /2 , n) 6

|X|=n

6 2(n(b + 1)2 )dlog ye ,

b = [ 1 ], y =

d0
P
i=1

n
i

bi , d0 = Sdim(F /2 ).
69

, F /2 1.10 b = [ 1 ].
(1.30) 1.9 d0 6 f at/4 (F) = d.

d  
d  
d  
 en d
X
X
n i X n i
n
d
y=
b 6
b 6b
6 bd
.
i
i
i
d
i=1

i=1

(1.36)

i=1

, log y 6 (ben/d).
. 4
1.7 1.3
.

1.4.

.
.10
X ,
P . F ,
X .
z l = (z1 , . . . , zl ) , X . ,
P .
1 , . . . , l ,
+1 1 :
B1/2 (i = 1) = B1/2 (i = 1) = 1/2 1 6 i 6 l.
.
l
= B1/2

1 , . . . , l l.
F 10

. : [9], [12], [22].

70


l (F) = E
R

11

!
l
1X
sup
i f (zi ) .
f F l
i=1

P P l z l = (z1 , . . . , zl )
l.
F
!
l
1X

Rl (F) = EP l (Rl (F)) = EP l E sup

i f (zi ) .
f F l
i=1

,
P l . ,
, F.
,

2.7.
f z l

l
1X

Ez l (f ) =
f (zi ).
l
i=1

, z l = (z1 , . . . , zl )
P .
R
f EP (f ) = f (z)dP .
F.
1.11.
z l (f ))) 6 2Rl (F).
Ez l P l (sup (EP (f ) E

(1.37)

f F

11
, z l
(z1 , . . . , zl ).

71

. zl = (
z1 , . . . , zl ) , z l =
(z1 , . . . , zl ). , ,
z1 , . . . , zl , z1 , . . . , zl
.
:
!!
l
1X
Ez l P l sup EP (f )
f (zi )
=
l
f F
i=1
!!
l
X
1
Ezi P (f (
zi )) f (zi ))
6
= Ez l P l sup
l
f F
i=1
!!!
l
1X
6 Ez l P l Ezl P l sup
(f (
zi ) f (zi ))
=
l
f F
i=1
!!
l
1X
=
= Ez l zl P 2l sup
(f (
zi ) f (zi ))
l
f F
i=1
!!
l
1X
6
= Ez l zl P 2l EB1/2 sup
i (f (
zi ) f (zi ))
l
f F
i=1
!!
l
1X
= Ezl P l EB1/2 sup
i f (
zi )
+
l
f F
i=1
!!
l
X
1
+Ez l P l EB1/2 sup
i f (zi )
=
l
f F
i=1

= 2Rl (F).

(1.38)

2- 3- :
Z
Z
sup f (
z )dP 6 sup f (
z )dP,
f F

f F

:
. 5- i ,
72

 zi zi ;
6- EB1/2 .
(1.37) . 4
1.11.
-, (1.37) :
1.5.
z l (f ) EP (f ))) 6 2Rl (F).
EP l (sup (E

(1.39)

f F

(1.39) (1.37) Rl (F) = Rl (F), F = {f : f F}.

, .
.
1.13. f : Z l R ,

|f (z1 , . . . , zi1 , zi , zi+1 , . . . , zl )

f (z1 , . . . , zi1 , zi0 , zi+1 , . . . , zl )| 6 ci
i z1 , . . . , zl , zi0 Z, c1 , . . . , cl .
z1 , . . . , zl ( P )
, Z.
:
P l {f (
z1 , . . . , zl ) EP l (f (
z1 , . . . , zl )) > t} 6

2t2

6 exp l
,
P 2
ci

(1.40)

i=1

EP l
P l l.
73

 [30]
[33].
f f , :
P l {EP l (f (z1 , . . . , zl )) f (z1 , . . . , zl ) > t} 6

2t2

6 exp l
.
P 2
ci

(1.41)

i=1

F . .
1.6. , F
[0, 1]. > 0, 1 , :
s
ln 2

EP (f ) 6 Ez l (f ) + 2Rl (F) +
6
2l
s
2
z l (f ) + 2R
l (F) + 3 ln ,
(1.42)
6E
2l
f F, z l = z1 , . . . , zl .
. f
z l (f ) + sup (EP (h) E
z l (h)).
EP (f ) 6 E

(1.43)

hF

(1.40) 1.13 (1.43).

f ,
ci = 1/l 1 6 i 6 l.

74

 (1.40) /2.

2t2
2

exp l
= e2t l = /2.
P 2
ci
i=1

q 2
ln
t =
2l . (1.40) , 1 /2
z l (h)) 6 EP l (sup (EP (h) E
z l (h))) +
sup (EP (h) E
hF
s
ln 2
+
.
2l

hF

(1.44)

(1.37) ,
z l (f ))) 6 2Rl (F).
EP l (sup (EP (f ) E
f F

(1.44)
s
z l (h)) 6 2Rl (F) +
sup (EP (h) E

hF

ln 2
.
2l

, 1 /2
s
2
z l (f ) + 2Rl (F) + ln
EP (f ) 6 E
2l

(1.45)

(1.46)

f F.
(1.42) .
, (1.41) 1.13 , 1 /2
s
2
l (F) + ln .
Rl (F) 6 R
(1.47)
2l
75

 (1.46) (1.47) , 1
(1.42) . 4

F = (F) = {(f ) : f F } F
.
1.12. L:
|(x) (y)| 6 L|x y|
x, y.
F F :
l ((F)) 6 LR
l (F),
R

(1.48)

Rl ((F)) 6 LRl (F).

(1.49)

. z l = (z1 , . . . , zl ) F, P , 1 , . . . , l

{+1, 1},
.
E = E , E =
EP l E . ,
(1.48) (1.49).
()
(F)
!
l
1X
Rl ((F)) = E sup
(1.50)
i (f (zi )) .
f F l
i=1

, L = 1. 12
!
l
1X
Rl ((F)) 6 Rl (F) = E sup
i f (zi ) .
(1.51)
f F l
i=1

12

/L.

76

 (1.50) (1.51)
. (1 , . . . , l ), i I.
i = ,
: i = I. ,
, , 1 = . i =
: 0i = I. -

77

 :
l

E(sup
f F

1X
i i (f (zi ))) =
l

1
E(sup ((f (z1 )) +
2l f F

i=1
l
X

i i (f (zi ))) +

i=2
l
X

+ sup ((f (z1 )) +

i i (f (zi )))) =

f F

i=2
l

X
1
E( sup ((f (z1 )) +
i i (f (zi ))
2l f,f 0 F
(f 0 (z1 )) +

i=2
l
X

i i (f 0 (zi )))) 6

i=2

1
6 E( sup (|f (z1 ) f 0 (z1 )| +
2l f,f 0 F
+

l
X

i i (f (zi )) +

i=2

l
X

i i (f 0 (zi )))) =

i=2

1
= E( sup (f (z1 ) f 0 (z1 ) +
2l f,f 0 F
+

l
X

i i (f (zi )) +

i=2

l
X

i i (f 0 (zi )))) =

i=2
l

X
1
= E(sup (f (z1 ) +
i i (f (zi )) +
2l f F
sup (f 0 (z1 ) +

f 0 F

i=2
l
X

i i (f 0 (zi ))) =

i=2
l

= E(sup
f F

1X
i 0i (f (zi ))),
l

(1.52)

i=1

01 , . . . , 0l
1 , . . . , l .
78

 (1.52) 1- 2- 3- 1 ; E , 1 . 4- 5- 6- 7- ,
(f (z1 ))(f 0 (z1 )), , ,
L = 1.
6- 7- 8- 9-. 8- 9-
10- ,
1- 2- 3-.
(1.52)

l

1X
E(sup
i 0i (f (zi ))),
f F l

(1.53)

i=1

0i , .. (1.53) Rl (F).
(1.52) Rl ((F)) E = EP l E
l ((F)) E = E . , (1.48)
R
(1.49) . 4

1.5.

BF (l)
N (, F, l).
. :
1.14. A Rl 1 , . . . , l

79

 .
!
p
l
2 ln |A|
1X
E sup
i ai 6 sup kak
,
l
aA l
aA
i=1

a = (a1 , . . . , al ).
. . E = E .
. 7- 8- ex + ex 6
2
2ex /2 . :
!!
l
X
exp E sup
i ai
6
aA i=1
l
X

6E

exp sup

aA i=1
l
X

sup exp

=E

6E

aA

i=1

l
X

exp

exp

=
!!

i ai

6
!!

i ai

l
X

!!
i ai

i=1

aA

i ai

i=1

aA

!!

l
XY

E(exp(i ai )) =

aA i=1

l
XY
eai + eai
6
2

aA i=1

l
XY

2 kak2 /2

aA i=1

6 |A|e
80

2 r 2 /2

 r = sup kak.
aA

!
l
X
ln |A| r2
+
.
(1.54)
E sup
i ai 6

2
aA
i=1

,
p (1.54)
= 2 ln |A|/r2 .
(1.54)
!
l
X
p
E sup
i ai 6 r 2 ln |A|.
aA i=1

. 4
.
1.13. F , ..
{1, +1}.

r
2 ln BF (l)
Rl (F) 6
l
l.
. E = EP l a =
(a1 , . . . , al ) (f (z1 ), . . . , f (zl )), z l = (z1 , . . . , zl ).
:
!
l
1X
Rl (F) = EE sup
i ai 6
a l
i=1
!
p
2 ln |Fz l |
6 E sup kak
6
l
a
p
2 ln BF (l)
6 l
=
r l
2 ln BF (l)
=
.
l
81

 1- 2- 1.14,
2- 3-

kak = l. . . 4
. F
X [1, 1]. X . xl = (x1 , . . . , xl )
X .
lxl (f, g) = sup16i6l |f (xi ) g(xi )| xl N (, F, xl ) ,
B F , f F g B
lxl (f, g) < .
1.14.

!
r
l)
2
ln
N
(,
F,
x
l (F) 6 inf
R
+ .
(1.55)

l
. B F
xl . , B {x1 , . . . , xl }.

B (g) = {f F : lxl (f, g) < }.

82

 gB B (g) = F.
l (F) = E
R

sup
f F

= E

sup sup
gB f B (g)

= E

sup sup
gB f B (g)

!!
l
1X
i f (xi )
=
l
i=1
!!
l
1X
i f (xi )
=
l
i=1
!!

1X
1X
i g(xi ) +
i (f (xi ) g(xi ))
l
l
i=1

i=1

6 E
l

+E

1X
sup
i g(xi )
gB l i=1
!

!
+

1X
i (f (xi ) g(xi )) . (1.56)
gB f B (g) l
sup sup

i=1

(1.56)
!
l
1X
E sup sup
i (f (xi ) g(xi )) =
gB f B (g) l i=1
!

l

1 X


i (f (xi ) g(xi )) 6
= E sup sup 


l
gB f B (g)
i=1
!
l
1X
6 E sup sup
i |f (xi ) g(xi )| 6 .
(1.57)
gB f B (g) l
i=1

1.14
!
l
1X
E sup
i g(xi ) 6
gB l i=1
p
2 ln |B|
6 sup kgk
6
l
gB
r
r
2 ln |B|
2 ln N (, F, xl )
6
=
.
l
l
83

(1.58)

s
kgk =

l
P

g 2 (xi ) 6

l, -

i=1

g l,
.
(1.57) (1.58)
!
r
l)
2
ln
N
(,
F,
x
l (F) 6
R
+ .
(1.59)
l
(1.59) > 0, > 0. (1.55). . 4
1.14
.
1.7.
r
Rl (F) 6 inf

!
2 ln N (, F, l)
+ .
l

. [9], [10], [26].

1.6.
1. 1.4 1.5.
2. Z ,
Pk (Z) = {A : A Z&|A| 6 k}
, k , fA A, ..
, 1 A, 0 . HZ . , BHZ (l)
BHZ (l) = 2l
84

 l 6 k,
BHZ (l) =

k  
X
l
i=0

l > k.
3. .
BH (2), BH (3), BH (4), . . . ,
a) H , ;
b) H , ;
c) H ,
2- , 3- ..
4. ,
V C- . (:
F = {sign(sin(tx)) : t R}. l
xi = 210i , i = 1, . . . , l, 1 , . . . , l
{0, 1}, xi
. ,
!
l
X
i
t=
(1 i )10 + 1
i=1

sign(sin(txi )) = i i (. [37])).
5. .
a) , VC- F
;
b) , VC- F
, , 4;
c) VC- F
d
, d > 3.
6. 3) 1.5 .

85

7. , (1.23) :
l
2 l 6 n
n1
P l1

(n, l) =
l > n.
2
i
i=1

8. G k- , Rn :
F = {f : f (
x) = sign(g(
x)) : g G}.
, VC- F k (: F Rn ).
9. L p f (
x) =
(w
x
) , kwk
2 = (w
w)
6 A k
xk2 6 R. ,
:
AR
Rl (L) 6 .
l
10. F X R , X =
{x1 , . . . xl }, xi X . X -
F,
r1 , . . . , rl , E X
fE F, fE (xi ) > ri xi E fE (xi ) < ri
xi
/ E, 1 6 i 6 l. , r1 , . . . , rl . - P dim(F) F
X - F.
-:
(a) Bf (x, y) = sign(f (x) y), f F, x X
y R. BF = {Bf : f F}. , V Cdim(BF ) =
P dim(F).
(b) : R R
(F) = { f : f F}. P dim((F)) 6 P dim(F).
(c) > 0, fatF () 6 P dim(F).
86

(d) X -, 0 , < 0 , X -
(: , X = {x1 , . . . , xl }
- F r1 , . . . , rl .

0 =

1
min{ri fE (xi ); 1 6 i 6 l, E X, fE (xi ) < ri }
2

< 0 , X -,
. [7]).
(e) P dim(F) = lim fatF ().
0

(f) F f (w)
=
(w
x
), x
, w
Rn . P dim(F) = n.
11. 1.9:
Sdim(F ) 6 f at/2 (F),
M(, F, X) 6 M(2, F /2 , X),
F [0, 1]X > 0.
12. , 1.13
1.6 1.4 ci = 1/l i.

87

Support Vector Machines (SVM),
. .

2.1.
, .. , .
S = ((
x1 , y1 ), (
x2 , y2 ), . . . , (
xl , yl )), x
i Rn
yi {1, 1}, i = 1, . . . , l, ()
(w
x
) c = 0,
w
(|w|
= 1) c ,
(w
x
i ) c > 0 yi = 1,
(w
x
i ) c < 0 yi = 1.

(2.1)

, (w
x
i ) c = 0
,
c1 (w)
= min(w
x
i ),
yi =1

c2 (w)
= max (w
x
i ).
yi =1

88

(2.2)

 c1 (w)
> c2 (w).
, c1 (w)
> c > c2 (w),

(w
x
i ) c = 0 .

c1 (w)
c2 (w)

(w)
=
.
2
(w)
= 12 ((c1 (w)
c) + (c c2 (w))


(w
x
) c = 0 (. (2.1)).
, S , .. c , (2.1).
(w)
{w
: |w|
6
1} . w
=w
0 .
2.1. (w)
w
= w
0 . (w
0 x
) c0 = 0,
c0 = 21 (c1 (w
0 ) + c2 (w
0 )), S
.
. , yi = 1
c1 (w
0 ) + c2 (w
0 )
=
2
c1 (w
0 ) c2 (w
0 )
=
> 0.
2

(w
0 x
i ) c0 > c1 (w
0 )

(2.3)

yi = 1
c1 (w
0 ) + c2 (w
0 )
=
2
c1 (w
0 ) c2 (w
0 )
=
< 0.
2

(w
0 x
i ) c0 6 c2 (w
0 )

(2.4)

. 4
(w
0 x
) c0 = 0 .
( )
S .
89

 2.2. ,
( ) S ,
.
. w
0 (w)

{w
: kwk
6 1} ,
0
w
= |w
k = 1 (w
) =
w
0 | kw
(w
0 )
kw
0 k

> (w
0 ).
, (w)
; ,
, ,
. 4
, (w)
. ,
(w
+ (1 )
u)) > (w)
+ (1 )(
u)

(2.5)

0 6 6 1 w,
u
, .

min(f (i) + g(i)) > min f (i) + min g(i),
iI

iI

iI

max(f (i) + g(i)) 6 max f (i) + max g(i)

iI

iI

iI

f g I.

1
(w)
= (min(w
x
i ) max (w
x
i )),
yi =1
2 yi =1
(2.6) f (i) = (w
x
i ) g(i) = (
ux
i )
min((w
+ (1 )
u) x
i ) =
yi =1

= min((w
x
i ) + (1 )(
ux
i )) >
yi =1

> min(w
x
i ) + (1 ) min(
ux
i ).
yi =1

yi =1

90

(2.6)

 .
(2.5). 4
. . , , .
w
0 b0 ,
(w
0 x
i ) + b0 > 1 yi = 1,
(w
0 x
i ) + b0 6 1 yi = 1,

(2.7)

i = 1, . . . , l, kw
0 k
.
2.1. w
0 , (2.7)
, w
0
w
0 = kw
0 k .



1
1
(w
0 ) = max
(min(w
x
i ) max (w
x
i )) =
.
yi =1
2 yi =1
kw
0 k
kwk=1

.
 



1
1
w
0
w
0

(w
0 ) =
c1
c2
>
,
2
kw
0 k
kw
0 k
kw
0 k
(2.7):

1 b0
w
0
>
,
c1
kw
0 k
kw
0 k


w
0
1 b0
c2
6
.
kw
0 k
kw
0 k


, (w
0 ) >
.

1
kw
0 k . w

w
1 = (w0 ) . 0

kw
1 k =

kw
0 k
< kw
0 k,
(w
0 )
91

 kw
0 k = 1.
, w
1 (2.7) b0 =
1 )
2 (w
c1 (w1 )+c
.

yi = 1 :
2
c1 (w
1 ) + c2 (w
1 )
=
2
1
c1 (w
0 ) + c2 (w
0 )

=
(
w

>
i
(w0 ) 0
2(w0 )
c1 (w
0 ) + c2 (w
0 )
c1 (w
0 )

> 1
= 1.

0 ) c2 (w
0 )
0 )) c1 (w
0 ) c2 (w
2 (c1 (w
(w
1 x
i )

yi = 1 .
, w
1

, w
0 . (w
0 ) = kw10 k . 4
w
0 (w
0 )
(w
0 ) = max (w)
=
kwk=1

1
.
kw
0 k

2.1 (w
0 ) = kw10 k
( )
(w
0 x
)

c1 (w
0 ) + c2 (w
0 )
= 0,
2

c1 (w
0 ) + c2 (w
0 )
(w
0 x
)
= 1,
2
.
(w
0 ) .

c1 (w
0 ) + c2 (w
0 )
(w
0 x
)
= 0.
2

92

2.2.

.
(2.7)
yi ((w
x
i ) + b) > 1

(2.8)

i = 1, . . . , l.
,

kwk
(2.8).
2.10 () ,

(w
w)
=

l
X

wi2 min

i=1

(2.7) (
(2.8))
l

X
1
L(w,
b,
) = (w
w)

i (yi ((w
x
i ) + b) 1),
2

(2.9)

i=1

i > 0 .
, (2.9),
w
b,
,
i > 0, i = 1, . . . , l.

l

X
L(w,
b,
)
=w

i yi x
i = 0,
w

(2.10)

i=1

L(w,
b,
) X
=
i yi = 0.
b
i=1

93

(2.11)

 (2.10) (2.11) ,
w
=

l
X

i yi x
i ,

(2.12)

i yi = 0.

(2.13)

i=1
l
X
i=1

(2.12) (2.9) W (
) = L(w,
b,
).
(2.13)
W (
) =

l
X

l
1 X
i j yi yj (
xi x
j ).
2

(2.14)

i,j=1

i=1

(2.14) (2.13) i > 0,

i = 1, . . . , l.
i = i0 , i = 1, . . . , l.

w
0 =

l
X

i0 yi x
i .

(2.15)

i=1

min(w
0 x
i ) + max (w
0 x
i )
b0 =

yi =1

yi =1

.
2
w
0 b0

i0 (yi ((w
0 x
i ) + b) 1) = 0

(2.16)

i = 1, . . . , l.
, i0 > 0 i,
yi ((w
0 x
i ) + b0 ) 1 = 0, .. ,
(w
0 x
i ) + b0 = 1. (support vectors). w
0
94

. 1.1.
H1 H2

x
is ,
s = 1, . . . , k, k
w
0 =

k
X

i0s yis x
is .

s=1

k
X

i0s yis (
x is x
) + b0 = 0.

(2.17)

s=1

, , , , , .
95

 .
2

kw
0 k = (w
0 w
0 ) =

k
X

iq ),
x is x
i0s i0q yis yiq (

(2.18)

s,q=1

W (
0) =

k
X
s,q=1

1
i0s kw
0 k2 .
2

(2.16)
k
X

i0s yis (w
0

x
is ) + b0

s=1

k
X

i0s yis

s=1

k
X

i0s .

s=1

(2.11) 0. , (2.18),
k
X
s=1

i0s =

k
X

i0s i0q yis yiq (

xis x
iq ) = k w
0 k2 .

s,q=1

0 k2 .
W (
0 ) = 12 kw
kw
0 k = s

1
k
P

s=1

.
i0s

2.3.

, S, .
S
(S) , S.
96

 S
, S, .. (S)

= (S).
l

, S d. d . hS .
hS , l d , > ,
(1 )ld 6 exp((l d)). ,
- hS , d,
l ,



ld
exp((l d)).
d
,
2.2. (S). P
X {1, 1} 0 < < 1, 1 S l hS ,
d,


 
 
1
el
l
errP (hS ) 6
d ln
+ ln
.
ld
d

, d > 2
l
d ln l
errP (hS ) 6
,
ld
d .

2.4. SVM-
S = ((
x1 , y1 ), (
x2 , y2 ), . . . , (
xl , yl )). SVM
. x
1 , . . . , x
l , Rn ,
97

 (feature space)
, :
x) = (1 (
x
= (x1 , . . . , xn ) (
x), . . . , N (
x)).

(2.19)

x1 ), . . . , (
xl )
(
N
R .
, (2.19) .
x)
Rn x
(
N
R .
RN x1 ), . . . , (
xl ).
, (
. , n 2- n
. .

z1 = x 1 , . . . , z n = x n ,
zn+1 = x21 , . . . , z2n = x2n ,
z2n+1 = x1 x2 , . . . , zN = xn xn1 .
N = 2n+ n(n1)
. ,
2

x) = z = (z1 , . . . , zN )
x
= (x1 , . . . , xn ) (
Rn RN .
Z = RN :
(w
z) + b = 0
x) = z
x
(
Rn :
x)) + b =
(w
(

N
X

wi z i + b =

i=1

n
X
i=1

wi x i +

2n
X
i=n+1

wi x2i +

N
X
i=2n+1

98

wi xji xki + b = 0,

 (ji , ki ) i -
6 n.
.
(2.19)
x) = (1 (
(
x), . . . , N (
x))
Rn
RN = {
z = (z1 , . . . , zN ) : zi R, i = 1, . . . , N }.
zj = j (
x),
j = 1, . . . , N .
x
1 , . . . , x
l Rn x1 ), . . . , (
xl ) RN .
(
,
2.2, RN :
N
X

wj zj + b = 0,

(2.20)

j=1

x1 ), . . . , (
xl ).
(

Rn , ,
N
X

wj j (
x) + b = 0.

(2.21)

j=1

,
xi ) : 0 > 0}:
{(
i
w
=

l
X

xi ).
i0 yi (

i=1

wj =

l
X

i0 yi j (
xi )

i=1

99

(2.22)

 j = 1, . . . , N .
.
(2.22) (2.21) , Rn
, RN :
N
X

wj j (
x) + b =

j=1

N
l
X
X
j=1

!
i0 yi j (
xi ) j (
x) + b =

i=1
l
X

i0 yi

i=1

N
X

j (
x)j (
xi ) + b =

j=1

l
X

x) (
xi )) + b =
i0 yi ((

i=1

l
X

i0 yi K(
x, x
i ) + b = 0,

(2.23)

i=1

xi ) (
x)).
K(
xi , x
) = ((

(2.24)

, SVM-
( ) Rn
,
(
xi x
) (2.17) :
k
X

i0 yi (
xi x
) + b = 0

s=1

K(
xi , x
), (2.24) .

100

,
f (
x) =

l
X

i0 yi K(
xi , x
) + b,

(2.25)

i=1

(2.23) l
N .
, Rn

x), K(
x
(
xi , x
).

RN ,
x1 ), . . . , (
xl )
(
x
1 , . . . , x
l .
W (),
(2.14).
, , :
W () =

l
X

i=1

l
1 X
xi ) (
xj )) =
i j yi yj ((
2
i,j=1

l
X
i=1

l
1 X
i j yi yj K(
xi , x
j )
2

(2.26)

i,j=1

,
(2.25), ((
x1 , y1 ), . . . , (
xl , yl ))
n
R , (2.26)
(2.13) i > 0, i = 1, . . . , l. ,
x1 ), . . . , (
xl ), ,
N - (
K(
xi , x
j )
.
, .
101

2.5.
.
X . K(x, y) X X
R,

K(x, y) = ((x) (y)),

(2.27)

X .
,
.
: K(
x, y) = (
x y)d K(
x, y) = ((
x y) + c)d
.
n(n+1)
. Rn R 2 :

= (1, x21 , . . . , x2n ,

x) = (x
1 , . . . , xn ) =
(

2x1 x2 , . . . , 2xn1 xn ).

x) (
y )) = 1+
K(
x, y) = ((

n
X

x2i yi2 +

i=1

n
X

2xi xj yi yj = (1+ x
y)2 .

i,j=1,i<j

K(
x, y) = (1 + (
x y))2
.
(2.25), ,

f (
x) =

l
X

i0 yi (1 + (
xi x
))2 + b.

(2.28)

i=1

,
K(
x, y) = K(
x
y ).
x
y .
102

 1 K(x)
[0, 2].
:
K(x) =

an cos(nx).

n=0

= a0 +

K(x, y) = K(x y) =

X
an sin(nx) sin(ny) +
an cos(nx) cos(ny).

n=0

n=0

x (1, sin x, cos x, sin 2x, cos 2x, . . . , sin(nx), cos(nx), . . . )
.

K(
x, y) = exp(k
x yk2 / 2 ).
.
K(
u, v) = exp((
u v)/ 2 ),
> 0 .
:
exp((
u v)) =

X
(
u v)k
k=0

k!

:
K(
u, v)
p
=
K(
u, u
)K(
v , v)
exp((
u v)/ 2 )
=
=p
exp((
uu
)/ 2 ) exp((
v v)/ 2 )
= exp(k
u vk2 /2 2 ).
103

 , .
.
. s
s = s1 s2 . . . sn ;
, .
|s| = n s ; 0.
n ()
n
n. =
n=0 .
, st , s t,
s[i : j] = si si+1 . . . sj .
u ()
s, i = (i1 , . . . , i|u| )
, 1 6 i1 < < i|u| 6 |s| uj = sij
j = 1, . . . , |u|; u = s[i]. u s l(i) = i|u| i1 + 1.
, :
, n
. Fn = R
, n.

n (s) = (nu (s) : u n ),

nu (s) =

l(i)

i:u=s[i]

0 < 6 1, n s,
s.

104

 :
Kn (s, t) =

(nu (s) nu (t)) =

un

X X
un i:u=s[i]

l(i)

X X

l(j) =

j:u=s[
j]

l(i)+l(j) .

un i:u=s[i]
j:u=s[
j]

Kn (s, t)
. (. [15]).
.
. [34].

2.5.1.
.
. .
2.4.
x) (
x)) K(
x, y) = ((

Rn RN .
K(
x, y) , ..,
K(
x, y) = K(
y, x
) x
y. , :
x
1 , . . . , x
l

105

 1 , . . . , l
l
X

i j K(
xi , x
j ) =

i,j=1

l
X
i=1

l
X

xi ) (
xj )) =
i j ((

i,j=1

xi )
i (

l
X

!
xi )
i (

i=1

2
l
X


xi )
=
i (
> 0.

(2.29)

i=1

(2.29) . X . K : X X
R , x1 , . . . , xl
1 , . . . , l
l
X

i j K(xi , xj ) > 0.

i,j=1

x) (
x)) (2.29) K(
x, y) = ((
.
(K(xi , xj )li,j=1 .
. H R
(x y) x, y Hp , kxk = (x x). ,
H
.
X F f : X R.
F (RKHS Reproducing Kernel
Hilbert Space), x X
f f (x)
. . RKHS
106

x Kx F , f (x) = (f Kx )
f F. , : X F,
(x) = Kx x.

f F :
f (x) = (f (x))
x.
K(x, y) = ((x) (y)) = (Kx Ky ) = Kx (y) = (Ky Kx ) = Ky (x).
RKHS : Rn RN . F
f (
x) = (w(
x)), x
Rn , w
RN (w
w
0 ) N
R . f kf k = kwk,
f (
x) = (w
(
x)) g(
x) = (w
0 (
x))
0
(f g) = (w
w
).
, f f (
x) = (w
(
x))
f . K(
x, y) = ((
x) (
y )). , f (
y) =
(w
(
y )) Kx (
y ) = K(
x, y) = ((
x), (
y )) (f Kx ) = (w
(
x)) = f (
x).

RKHS

|f (x)| 6
p
(K(x, x))kf k. ,

p
p
|f (x)| = |(f Kx )| 6 (Kx Kx )kf k = (K(x, x))kf k.

|f1 (x) f2 (x)| 6

p
(K(x, x))kf1 f2 k

f1 , f2 F x X. , fn f
RKHS F, :
fn (x) f (x) x X.

107

 ,
K(x, y) F .
: X RX X X R:
(x) = K(x, ) = Kx .
Kx , : Kx (y) = K(x, y)
y. F1 ,
f=

n
X

i Kxi ,

(2.30)

i=1

n, i R xi X . . n
P
f =
i Kxi
i=1

g=

n0
P
j=1

j Kx0j
0

(f g) =

n X
n
X

i j K(xi , x0j ).

(2.31)

i=1 j=1

, (2.31)
n0
n
P
P
(f g) =
j f (x0j ) (f g) =
i g(xi ). ,
j=1

i=1

(2.31) f g .
, (f g) f g.
: (f g) = (g f ) f, g F1 .
.
,
(f f ) =

n X
n
X

i j K(xi , xj ) > 0.

i=1 j=1

108

 ,
f1 , . . . , fn 1 , . . . , n R (f g):

n X
n
n
n
X
X
X
i j (fi fj ) =
i fi
j fj > 0.
i,j=1 j=1

i=1

j=1

(2.31) f (x) = (f Kx ) , , (Kx

Ky ) = K(x, y) x y.
(2.30) ,
|f (x)|2 = |(f Kx )|2 6 K(x, x)(f f )
f F1 . , (f f ) = 0 , f (x) = 0
x.

p
(2.32)
|f1 (x) f2 (x)| 6 K(x, x)kf1 f2 k
f1 , f2 F1 .
p
kf k = (f f ) , .
F1
(2.30) .
F. (2.32) f1 , f2 , F1 x
p
|fn (x) fm (x)| 6 K(x, x)kfn fm k
m, n. x
f1 (x), f2 (x), R f (x) = lim fn (x) x X. n
, F1 F.
F
(f g) = lim (fn gn ),
n

109

 f1 , f2 , . . . g1 , g2 , . . . F1 , f, g F, . , f1 , f2 , . . . g1 , g2 , . . . .
, F ,
. F:
f (x) = lim fn (x) = lim (fn Kx ) = (f Kx ).
n

RKHS [24] [8].

F K(x, y).
RKHS 2.12.
K(
x, y) = exp(k
x yk2 / 2 ) RKHS
.
. (Representer theorem) , .
[25]. . [34].
2.3. X S =
((x1 , y1 ) , . . . , (xl , yj )) , (xi , yi ) X R.
K(x, x0 ) X X
F
RKHS k k.
c : (X 2 R)l R {}
.

110

 f F,
c((x1 , y1 , f (x1 )) , . . . , (xl , yj , f (xl ))) + (kf k)

(2.33)

f (x) =

l
X

i K(xi , x),

(2.34)

i=1

1 , . . . , l .
f F:
l

c((x1 , y1 , f (x1 )) , . . . , (xl , yj , f (xl ))) =

1X
(yi f (xi ))2 + kf k2 ,
l
i=1

> 0.
. , Kxi = K(xi , ) , . f F
f (x) = (f Kx ) x.
F
, Kxi , i = 1 , . . . , l, . f F
:
l
X
f=
i Kxi + f ,
i=1

(f Kxi ) = 0 i = 1 , . . . , l.
f (xj ) j = 1 , . . . , l:

l
X

f (xj ) = (f Kxj ) =
!
!
i Kxi + f

Kxj

i=1

l
X

i (Kxi Kxj ).

i=1

111

 , f (xj )
f . , c((x1 , y1 , f (x1 )) , . . . , (xl , yj , f (xl )))
(2.33) f .
l
P
f
i Kxi i=1

,
(kf k) = k

l
X

!
i Kxi + f k

i=1

u l
u X
= tk
i Kxi k2 + kf k2 >
i=1

> k

l
X

!
i (Kxi k ,

i=1

, f =
0. (2.33)
f = 0. (2.33)
(2.34):
f (x) =

l
X

i Kxi .

i=1

. 4
2.3 , (2.33) RKHS (
) Rn .
, SVM:
c((x1 , y1 , f (x1 )) , . . . , (xl , yj , f (xl ))) =
l

1X
=
max{0, 1 yi f (xi )) + kf k2 ,

i=1

112

 xi Rn yi {1, +1} i = 1 , . . . , l.
F K(x, x0 ).
f F, (2.33),
f=

l
X

i Kxi .

i=1

2.6.

, . .

2.6.1.
. .
F X R,
. X
F Rn . f F


1, f (
x) > 0,
h(
x) =
1 .
S = ((
x1 , y1 ), . . . , (
xl , yl )). i = yi f (
xi )
(
xi , yi ) X {1, 1}
f F. , i > 0 ,
f .
S = ((
x1 , y1 ), . . . , (
xl , yl ))
MS (f ) = (1 , . . . , l ).
mS (f ) = min i
i=1,...,l

113

 S f . mS (f ) > 0 , f
S .
> 0. (margin slack variable) (
xi , yi ) X {1, 1} f

i = max{0, yi f (
xi )}.
i f (
xi , yi ). .

1 1/kwk,
(w
x
) + b = 1.
, i > ,
(
xi , yi ) .
= (1 , . . . , l ) S = ((
x1 , y1 ), . . . , (
xl , yl )).
yi f (
xi ) + i > i.
.
i > , yi f (
xi ) < 0, .. (
xi , yi )
f . , i
(
xi , yi )
, .
i = 0 , yi f (
xi ) > ;
.
0 < i 6 ,
0 < yi f (
xi ) 6 , , ,
.

, .
114

 kk
.
,
f (
x) > 0 1.9
.
, . ,
- [32],
.
2.4. L
L(
x) = ( w
x
) + b kwk
= 1.
> 0. > 0 P X {1, 1} R
1
f L S l
1 :


2
c R2 + kk
el
128l
2
errP (f ) 6
log
log 2 + log
,
(2.35)
l
2
4R2

c , > 0.
. f (
x) =
(w
x
) + b, kwk
= 1. = (1 , . . . , l ),
S = (
x1 , . . . , x
l ),
yi f (
xi ) + i >

(2.36)

i = 1, . . . , l.
> 0 ,
. x
1 , . . . , x
l n x
01 , . . . , x
0l n+l,
:
x
0i = (xi,1 , . . . , xi,n , 0, . . . , , . . . , 0),
1
Rn , x
1 , . . . , x
l S.

115

 i = 1, . . . , l, (n + i)- x
0i ,
0.
S 0 = ((
x01 , y1 ) . . . , (
x0l , yl )).
f (
x) = ( w
x
) + b
f 0 (
x0 ) = ( w
0 x
0 ) + b,
1
1
w
0 = (w1 , . . . , wn , y1 1 , . . . , yl l ),

(2.37)

x
0 n + l.
(2.36) , S 0
(2.37) :
yi ((w
0 x
0i ) + b) = yi ((w
x
i ) + b) + (yi )2 i >

(2.38)

i = 1, . . . , l.
, 1.9 1.3,
(2.37).
1 2
1 2
kw
0 k2 = kwk
+ 2 kk
= 1 + 2 kk
.

, x
0i R0 ,
02
2
2
R =R + .
(2.38)
 0


w

b
0
yi

> 0 =
.
+
kw
0k i
kw
0k
kw
0k
i = 1, . . . , l. ,
1.4
(R2 + 2 )(1 +
R02
=
02
2

1
kk2 )
2

(R2 + 2 )(1 +

1
1
kk2 ) = R2 + kk2 + 2 + 2 R2 kk2 .
2

116

 2 = Rkk,

R2 + 2Rkk + kk2 = (R + kk)2 6 2(R2 + kk2 ).
1.9 1.3, (2.35). . 4

2.6.2.

. , S =
((
x1 , y1 ), . . . , (
xl , yl )) , i , i = 1, . . . , l.
w,
b,
l
X

i2 min,

(2.39)

yi ((w
x
i ) + b) > 1 i ,

(2.40)

i > 0

(2.41)

(w
w)
+C

i=1

i = 1, . . . , l. C
.
C ,
.
, i > 0 , w,
, b, i < 0, i = 0.
(2.39) (2.41)
l
1
CX 2

L(w,
b, ,
) = (w
w)
+
i
2
2
i=1

l
X

i (yi ((w
x
i ) + b) 1 + i ),

i=1

117

(2.42)

 i > 0 .


l
X

L(w,
b, ,
)
=w

yi i x
i = 0,
w

(2.43)

i=1

L(w,
b, ,
)
= C
= 0,

L(w,
b, ,
) X
=
yi i = 0,
b
i=1

(2.42) :

L(w,
b, ,
) =

l
X

l
1 X
yi yj i j (
xi x
j ) +
2
i,j=1

i=1

+
=

l
X

1
1
(

) (

) =
2C
C

l
1
1 X
(

).
yi yj i j (
xi x
j )
2
2C

(2.44)

i,j=1

i=1

,

W (
) =

l
X
i=1

l
1 X
1
yi yj i j ((
xi x
j ) + ij )
2
C

(2.45)

i,j=1

i > 0, i = 1, 2, . . . l, ij = 1 i = j ij = 0
i 6= j.

i (yi ((w
x
i ) + b) 1 + i ) = 0
i = 1, . . . , l.
(2.43) :
w
=

l
X

yi i x
i .

i=1

118

 , i = 0,
yi ((w
x
i ) + b) > 1, i = 0.
. x
i ,
i = Ci > 0.
yi ((w
x
i ) + b) < 1 i > 0.
, K(
xi , x
j ).
2.5. , K(
xi , x
j ), S = ((
x1 , y1 ), . . . , (
xl , yl )).

:
W (
) =

l
X

i=1
l
1 X
1

yi yj i j (K(
xi , x
j ) + ij ) max
2
C

(2.46)

i,j=1

l
X

yi i = 0,

i=1

i > 0 i = 1, . . . , l.

(2.47)

f (
x) =

l
X

yi i K(
xi , x
) + b,

i=1

b yi f (
xi ) = 1 i /C
i , i 6= 0.
sign(f (
x)) , , (2.39) (2.41)
, K(
x, z),

119

 :
1/2
1

)
.
j (
C

X
jsv

b i = Ci ,
:
i (yi ((w
x
i ) + b) 1 + i ) = 0
i = 1, . . . , l.
1
(w)
= |w|
,
(w
x
i ) + b) = 1 ( ),
:
l
X

(w
w)
=

yi yj i j K(
xi , x
j ) =

i,j=1

yj j

jsv

yi i K(
xi , x
j ) =

isv

j (1 j yj b) =

jsv

jsv

j j =

jsv

jsv

1
(

).
C

(2.46) K(
xi , x
j )
K 0 (
xi , x
j ) = K(
xi , x
j ) +

1
x (
y)
C

, .
(2.35)
, K(
x, z), . 120

 , .
2.4. (2.35)
w.
(2.39) (2.41),
(2.40) kwk,
w

. (2.35) i
i /kwk,
= 1/kwk.

(2.35) :
errP (f ) = P {yf (
x) 6 0} 6
c
2 ) log2 l + log 2 ).
6 ((kwk
2 R2 + kk
(2.48)
l

(2.48) ,
(2.39).
. ,
.
.
w,
b,
(w
w)
+C

l
X

i min,

(2.49)

i=1

yi ((w
x
i ) + b) > 1 i ,
i > 0

(2.50)

i = 1, . . . , l. C
.

l

X
1
L(w,
b, , , r) = (w
w)
+C
i
2

l
X

i (yi ((w
x
i ) + b) 1 + i )

i=1

i=1
l
X
i=1

121

ri i ,

. 1.2.

i > 0, ri > 0 i = 1, . . . , l.
:
l

X
L(w,
b, , , r)
=w

yi i x
i = 0,
w

i=1

L(w,
b, , , r)
= C i ri = 0,
i
l
L(w,
b, , , r) X
=
yi i = 0.
b
i=1

122

 :
L(w,
b, , , r) =

l
X
i=1

l
1 X
yi yj i j (
xi x
j )
2
i,j=1

i > 0, i = 1, . . . , l,
. , l
P
w
=
i yi x
i .
i=1

,
C i ri = 0 ri > 0
i 6 C. i > 0
ri = 0. , i = C i.
,
i (yi ((w
x
i ) + b) 1 + i ) = 0, i = 1, . . . , l,
i (i C) = 0, i = 1, . . . , l.
i
i = C.
,
yi ((w
x
i ) + b) > 1, i = 0 i = 0 (
);
yi ((w
x
i ) + b) < 1, i > 0 i = C ( , );
yi ((w
x
i ) + b) = 1, i = 0.
,
S = ((
x1 , y1 ), . . . , (
xl , yl )) (.., , yi = 1 yj = 1
- 1 6 i, j 6 l) b
x
i , :
yi ((w
x
i ) + b) = 1, i > 0 (. 2.12).
123

 x
i , i > 0 , . ,

i
kwk
(. . 1.2).
,
.
2.6. S = ((
x1 , y1 ), . . . , (
xl , yl ))
, K(
xi , x
j ).

:
W (
) =

l
X
i=1

l
1 X
yi yj i j K(
xi , x
j ) max
2

(2.51)

i,j=1

l
X

yi i = 0,

i=1

C > i > 0 i = 1, . . . , l.

(2.52)

f (
x) =

l
X

yi i K(
xi , x
) + b,

i=1

b yi f (
xi ) = 1 i , x
i
i > 0.
sign(f (
x))
, , (2.49) (2.50)
, K(
x, z),
:

1/2
X
=
yi yj i j K(
xi , x
j )
.
i,j

124

 , (2.49) (2.50) (2.39) (2.41)

, i 6 C. (box constraints), ,
i C, .
C i . ,
C, i , ..
, .
.
,
w

i , .

, 2.2 , .
:
l
X

i + C

i=1

yi

l
X

l
X

i min

i=1

i (
xi x
j ) + b > 1 i ,

j=1

i > 0, i > 0,
i = 1, . . . , l. C
.
,
.

125

2.7.

, RKHS. 2
F ,
X . , (RKHS), K(x, y) . f F f (x) = (f (x)), (x) = K(x, ).
RKHS : Rn RN .
F f (
x) = (w
(
x)),
x
Rn , w
RN (w
w
0 ) RN .
f kf k = kwk,
f g(
x) = (w
0 (
x)) (f g) = (w
w
0 ).
K(x, y) = ((x)(y)) .
f F

1, f (x) > 0,
h(x) =
1 .
F1 = {f F : kf k 6 1}.
F1 f (x) = (w
(
x))
kwk
6 1.
S = ((x1 , y1 ), . . . , (xl , yl )),
xi X yi {1, 1}.
K = (K(xi , xj ))li,j=1
P
S; tr(K) = li=1 K(xi , xi )
K.
F1 S.
2
- [33].

126

 2.7. F1 S = ((x1 , y1 ), . . . , (xl , yl ))

p
l (F1 ) 6 1 tr(K).
R
l

(2.53)

.
:
!
l
X
1
l (F1 ) = E sup
R
i f (xi ) =
f F l i=1
!!
l
1X
= E sup f
6
i (xi )
l
kf k61
i=1
l
!


1
X

6 E
i (xi ) =


l
i=1

!1/2
l
l
X
X
1
6
i (xi )
= E
i (xi )
l
i=1

i=1

1
E
l

l
X

1/2
i j K(xi , xj )

i,j=1

1
=
l

l
X

!1/2
K(xi , xi )

i=1

2- 3- , 3- 4-
. 4-
5- , 5- 6-
i , E(i ) = 1 E(i j ) = E(i )E(j ) = 0
i 6= j. . 4
, i = yi f (xi ) (xi , yi ) X {1, 1} f F.
127

, i > 0 ,
f .
S = ((x1 , y1 ), . . . , (xl , yl )).
> 0. (xi , yi )
f
i = max{0, yi f (xi )}.

(2.54)

= (1 , . . . , l ) S = ((x1 , y1 ), . . . , (xl , yl )).

f (x, y) = yf (x).
X {1, 1}:
F2 = {f (x, y) : f (x, y) = yf (x), f F1 }.


(x) =

1, x > 0,
0 .

, S
P .
,
P {(x, y) : y 6= sign(f (x))} = EP ((yf (x))) = EP ((f (x, y))).
K = (K(xi , xj ))ni,j=1
S.
, K.
2.8. > 0 1

s
l
X
p
ln 2
1
2
P {y 6= sign(f (x))} 6
i +
tr(K) + 3
. (2.55)
l
l
2l
i=1

128

, (2.55) . K
S. i
xi .
. , > 0 . g : R [0, 1]:

1, r > 0,
1 + r/, 6 r 6 0,
g(r) =

0 .
, g(r) > (r).
1.6 1
EP ((f (x, y))) 6 EP (g(f (x, y))) 6
r
S (g f )) + 2R
l (g F2 ) + 3 ln(2/) .
6E
2l

(2.56)

(2.54)
(
i
1 yi f(xi , yi f(xi < 1,
=

0 .

g(yi f (xi )) 6 i /
1 6 i 6 l.
F2 :
!
l
X
1
l (F2 ) = E sup
R
i f (xi , yi ) =
f F2 l i=1
!
l
1X
= E sup
i yi f (xi ) =
f F1 l i=1
!
l
1X
i f (xi ) =
= E sup
f F1 l i=1
p
l (F1 ) 6 1 tr(K).
=R
l
129

 g L =
1/, 1.12
l (g F2 ) 6 R
l (F2 )/ = R
l (F1 )/.
R
f F2
l
l
X
1 X
S (g f )) = 1
E
g(yi f (
xi )) 6
i .
l
l
i=1

i=1

(2.56) (2.53) 2.7

, 1 ,
r
l
1 X
2p
ln(2/)
EP ((f (x, y))) 6
i +
tr(K) + 3
.
(2.57)
l
l
2l
i=1

. 4
, f (x) S , :
2.1. , f (x) S , . > 0 1

s
p
ln 2
2
P {y 6= sign(f (x))} 6
tr(K) + 3
.
l
2l
f (
x) = ( w
(
x)), x
Rn , w
RN .
(1.27) (2.35), , (2.55) ,

.
,
, (2.55) , .
130

 k
xi k 6 R 1 6 i 6 l.
s
R2
2p
2 2
tr(K) 6
lR = 2
l
l
l 2
 2
R

O l
2
(1.27) 1.9 (2.35) 2.4.

2.8.
2.8.1.
S = ((
x1 , y1 ), . . . , (
xl , yl )),
n
x
i R , yi R i = 1, . . . , l.

f (
x) = (w
x
) + b,
S. ,
yi x
i i = 1, . . . , l.
XVIII
f (
xi ) yi i = 1, . . . , l.
.
x
(n 1), .. -

x1
x2

x
=
.

xn
131

 , (x1 , . . . , xn ), ..
(1n),
x
0 . A B AB
. (
x z) x
0 z :

0
x
z = (x1 , . . . , xn )

z1
z2

zn

= (x1 z1 + . . . xn zn )

(11) ,
.
, :
L(w,
b) =

l
X

(yi (w
x
i ) b)2 .

(2.58)

i=1

w
-

w1
w2

w
=
.

wn
b
, x
-

132

x
=

x1
x2

xn
1

:
f (
x) = (w
x
).

(2.59)

(l (n + 1)), - x
0i = (
x0i , 1)

X=

x
01
x
02

x
0l

x11 , . . . , x1n , 1
x21 , . . . , x2n , 1

xl1 , . . . , xln , 1

l- -

y1
y2

.
y =

yl
|f (
xi ) yi | ( yi f (
xi ) f (
xi ) yi ) w),
. - y (X

(2.58)
- :
w
w)
w).
L(w)
= kX
yk2 = (
yX
0 (
yX

133

 A A0 A.

:
w
L(w)
= kX
yk2 min .

(2.60)

,
y (
), X.

( w1 , . . . , wn , b) . n + 1
L(w)

0 y + 2X
0X
w
= 2X
= 0.
w

w
0 y.
0X
=X
X
, :
X 0 X
0 X)
1 X
0 y.
w
= (X

2.8.2.
, (ridge regression), .
, , ,
w
-

w1
w2

w
=

wn
b
134

 x
-

x1
x2

x
=
.

xn
1


f (
x) = (w
x
).
:
L(w)
= (w
w)
+

l
X

(yi (w
x
i ))2 =

i=1

w
= kwk
2 + kX
yk2 .

(2.61)

.
.
. .
.
L(w)
wi , i = 1, . . . , n + 1,
w

l
X
((yi (w
x
i ))
xi = 0.
i=1

0 y + X
0X
w
w
X
= 0.
:
0 X)
1 X
0 y,
w
= (I + X
135

 I .
0 X,
I I + X
0X
(n + 1) (n + 1).
X
0

X X , ..
z>0
0 X)
z0 (X
z.
0 X)
z = (X
z)0 (X
z) = kX
zk2 > 0.
z0 (X
0X
I, > 0,
X

z = k
zk2 > 0
0 X)
z k2 + kX
z0 (I + X
z 6=
0. , . > 0.
0X
.
= 0 X
. > 0 , . ,
w.

0X
, I + X
. .

.
, I 0X
X

.
0 X.

, X
2.9.2.

136

2.9.
2.9.1. SVM
. , , , ..
.
-

L (
x, y, f ) = |y f (
x)| = max{0, |y f (
x)| },

(2.62)

f Rn R.
= (1 , . . . , l ) ,
i = L (
xi , yi , f ), i = 1, . . . , l.
, -
L2 (
x, y, f ) = (|y f (
x)| )2 .

(2.63)

.

l
X
kwk
2+C
L2 (
xi , yi , f ),
i=1

C . f (
x) = (w
x
) + b.
 > 0. i i ,

.
C
.
137


(2.63) C 
:
kwk
2+C

l
X

(i2 + i2 ) min

i=1

((w
x
i ) + b) yi 6  + i , i = 1, . . . , l,
yi ((w
x
i ) + b) 6  + i , i = 1, . . . , l,
i , i > 0, i = 1, . . . , l.

(2.64)

C
.

l

1 X 2 2
1
,

2+ C
(i + i ) +
L(w,
b, ,
,
) = kwk
2
2
i=1

l
X

i ((w
x
i ) + b yi  i ) +

i=1

l
X

i (yi (w
x
i ) b  i ).

i=1

, , , i > 0,
i > 0 , , i < 0
i < 0, i = 0 i = 0.
.

w,
b, ,
w

w
=

l
X

(
i i )
xi .

i=1

b
l
X

(
i i ) = 0.

i=1

138

(2.65)

 i i i = C1 i
i = C1
i i.
, (2.64)
i i = 0 i.
i
i = 0.
:
l
X

yi (
i i ) 

i=1

l
X

(
i + i )

i=1

l
1
1 X
(
i i )(
j j )((
xi x
j ) + ij ) max

2
C
i,j=1

l
X

(
i i ) = 0,

i=1

i > 0, i > 0, i = 1, . . . , l,

(2.66)

ij = 1 , i = j.
:
i ((w
x
i ) + b yi  i ) = 0, i = 1, . . . , l,

i (yi (w
x
i ) b  i ) = 0, i = 1, . . . , l.

(2.67)

(2.67) ,
x
i ,  , i =
i = 0.
(2.65) ( ), .
, x
i (w
x
i ) + b <
yi  (w
x
i ) + b > yi + .
i ,
i ,
.  ,
.
 .
139

 SVM.

, .
.
2.9. S = ((
x1 , y1 ), . . . , (
xl , yl )), x
i
X yi R. ,
K(
x, z).

W (
) =

l
X

yi (
i i ) 

i=1

l
X

(
i + i )

i=1

l
1 X
1

(
i i )(
j j )(K(
xi , x
j ) + ij ) max
2
C
i,j=1

l
X

(
i i ) = 0,

i=1

i > 0,
i > 0 i = 1, . . . , l.
f (
x) =

l
P

(2.68)

(
i i )K(
xi , x
) + b, b

i=1

, f (
xi ) yi =  (
i i )/C i,
i > 0
i > 0.
f (
x) , K(
xi , x
),
(2.64).
.
(2.62).
2

kwk
+C

l
X

L (
xi , yi , f ),

i=1

140

 f (
x) = ( w
x
)+b, C ,
.
(2.62) C  :
2

kwk
+C

l
X

(i + i ) min

i=1

((w
x
i ) + b) yi 6  + i , i = 1, . . . , l,
yi ((w
x
i ) + b) 6  + i , i = 1, . . . , l,
i , i > 0, i = 1, . . . , l.

(2.69)

C
.

l

X
1
,

L(w,
b, ,
,
, r, r) = kwk
2+C
(i + i ) +
2
i=1

l
X
i=1
l
X

i ((w
x
i ) + b yi  i ) +

i (yi (w
x
i ) b  i )

i=1

l
X
i=1

ri i

l
X

ri i ,

(2.70)

i=1

i > 0,
i > 0 ri > 0, ri > 0.
(2.69)
2.6.2 (
).
w,
b, ,

141


w
=

l
X

(
i i )
xi .

(2.71)

i=1

b
l
X

(
i i ) = 0.

i=1

, C i ri = 0 C
i ri = 0,
i = 1, . . . , l.
(2.70) :
l
X

yi (
i i ) 

i=1

l
X

(
i + i )

i=1

l
1 X

(
i i )(
j j )(
xi x
j ) max
2
i,j=1

l
X

(
i i ) = 0,
i=1

i > 0, i > 0,

(2.72)

0 6 i ,
i 6 C, i = 1, . . . , l.

(2.73)

:
i ((w
x
i ) + b yi  i ) = 0, i = 1, . . . , l,

i (yi (w
x
i ) b  i ) = 0, i = 1, . . . , l,
(i C)i = 0, (
i C)i = 0,

(2.74)

i i = 0, i
i = 0, i = 1, . . . , l.

(2.75)

x
i , i > 0
i > 0. yi

(), i = C
i = C.
142

 0 < i < C 0 <

i < C x
i .
x
i , yi ,
. i = 0
i = 0,
(w
x
i ) + b yi < 
yi ((w
x
i ) + b) <  i = i = 0.

w
=

l
X

(
i i )
xi .

i=1

:
l
X
f (
x) =
(
i i )(
xi x
) + b.
i=1

:
l
X

yi (
i i ) 

i=1

l
X

(
i + i )

i=1

l
1 X

(
i i )(
j j )K(
xi , x
j ) max
2
i,j=1

l
X

(
i i ) = 0,

i=1

i > 0, i > 0,
0 6 i ,
i 6 C, i = 1, . . . , l.
:
f (
x) =

l
X

(
i i )K(
xi , x
) + b.

i=1

2.9.2.
-
(2.63),  = 0.
143


 = 0,
2.8.2.
,
2.8.1;
.
c x


f (
x) = ( w
x
).

kwk
2+

l
X

i2 min

i=1

yi (w
x
i ) = i , i = 1, . . . , l.

L(w,
,
) = kwk
2+

l
X

i2 +

i=1

l
X

i (yi (w
x
i ) i ).

(2.76)

i=1

(2.76)
wj j :
l
X

L(w,
,
)
= 2w

i x
i = 0,
w

i=1

2i i = 0
i = 1, . . . , l. :
l

w
=

1 X
i x
i ,
2
i=1
i
i = .
2
144


l
1 X
i j (
xi x
j ),
4

(w
w)
=

i,j=1

l
X

i (w
x
i ) =

l
1 X
i j (
xi x
j ).
2
i,j=1

i=1

(2.76),
W (
) =

l
X

yi i

i=1

l
1 X
i j (
xi x
j )
4
i,j=1

1X 2
i max .
4

(2.77)

i=1

W (
) = y0

1 0
1 0

max,
4
4

K ,
Ki,j = (
xi x
j ).
W (
)
(2.77) i , ,

1
1

K

+ y = 0.
2
2
:

= 2(K + I)1 y.

(2.78)

.


l

(w
x
) =

1 X
1 0
i (
xi x
) =
(
k),
2
2
i=1

145


= (1 , . . . , l ), k = (k1 , . . . , kl ) ki = (
xi x
).
K , K 0 = K.
(AB)0 = B 0 A0
(2.78)

0 = 2
y 0 (K + I)1 .

f (
x) = ( w
x
) = y0 (K + I)1 k.

(2.79)

. i l , K + I l l,
.
.
.
. .
K(
x, y).
.
x) RN .
x
(
RN K(
xi , x
j ) =
xi ) (
xj )).
((
K

K(
x1 , x
1 ), . . . , K(
x1 , x
l )
K(
x2 , x
1 ), . . . , K(
x2 , x
l )

.
K=

K(
xl , x
1 ), . . . , K(
xl , x
l )

146

z =

K(
x1 , x
)
K(
x2 , x
)

K(
xl , x
)

(2.79),
x1 ), . . . , (
xl ) RN , (

f (
x) = y0 (I + K)1 z,

(2.80)

z = (z1 , . . . , zl ), zi = K(
x, x
i ) i = 1, . . . , l.
(2.80) , K(
x, z).
, , (2.80) 5.9.3.
, . .
.

(Kriging).
x
1 , . . . , x
l , R(
xi , x
j ) = E(
xi x
j ). , , x
1 , . . . , x
l ,
.

2.10.
.
147

 , ()
. . , .
, .
. f (w),
gi (w),
hi (w),
i = 1, . . . , m, Rn , w
Rn .

inf f (w)

w

gi (w)
6 0, i = 1, . . . , m,

(2.81)

hi (w)
= 0, i = 1, . . . , m.

(2.82)

g(w)
6 0
w)
h(
=
0.
w)
= {w
Rn : g(w)
6 0, h(
= 0}
.
w
,
w
w
Rn , f (w)
< f (w
).

, w
f . w
,
. f .

f (w)
w,
g h
,
.
f , w,
u
Rn
0 6 6 1
f (w
+ (1 )
u) 6 f (w)
+ (1 )f (
u).
,
w)
h(
=
0.
= f (w)
w).
L(w,
)
+ h(

148

L(w,
)
= 0,
w

L(w,
)
= 0.

, L .
(2.82)
= f (w)
L(w,

, )
+

m
X

i gi (w)
+

i=1

m
X

i hi (w)
=

i=1

w).
= f (w)
+
g(w)
+ h(

.
, , .
= inf L(
w).
(
, )
, ,

,

, )
max (

(,
)

i > 0, i = 1, . . . , m.

(2.83)

.
2.10. w
(2.81)
(2.82) ( ,

), (
, )

(2.83). f (w)
> (
, ).
.
= inf L(
6 L(w,
=
(
, )
u,
, )

, )
u

w)
= f (w)
+
g(w)
+ h(
6 f (w).

w)

g(w)
6 0,
> 0 g(w)
6 0, h(
= 0. 4

149

(2.84)

 2.2. :
:
w)
sup{(
, )
>
0} 6 inf{f (w)
: g(w)
6 0, h(
= 0}.

,
.
2.3. f (w
) = (
, ),
> 0, g(w
) 6 0,
w
h(
) =
0, w
(
, )
.

g(w
) = 0.
. (2.84)
, .
, f (w
) = inf L(
u,
, )
g(w
) = 0. 4
u

. (w
,
, )
6 L(w
L(w
,
, )
,
, ) 6 L(w,

, )

w,

, .
.
2.11. g (.. hi (w),
Rn , h,

gi (w)
Ai w
+ bi , Ai ).
.
.
2.12.
Rn , f , h,
g (.. hi (w),
gi (w)
Ai w
+ bi , Ai
).
150

 w

inf f (w),
w
,

g(w)
6 0,
w)
h(
= 0
, (
, ) ,

L(w
,
, )
= 0,
w

L(w
,
, )
= 0,

i gi (w
) = 0, i = 1, . . . , m,

(2.85)
(2.86)

gi (w
) 6 0, i = 1, . . . , m,
i > 0, i = 1, . . . , m.


(2.85); L(w
,
, )
: hi (w
) = 0, i = 1, . . . , k.

L(w
,
, )
i
(2.86), i > 0, gi (w
) = 0 (

i ), gi (w
L(w
,
, )
) < 0 = 0.
L(w
,
, )
i
(2.86) .
,
i- , i > 0, i = 0.
.
1 0
w
Qw
kw
min
2
X w
6 c,

(2.87)

Q n n- , k n-,
c m-, w
n- , X (m, n)-.
151

, . :



1 0
max min
w
Qw
kw
+
0 (X w
c)
(2.88)
w

2

> 0.
w
(2.88)
w
= Q1 (k X 0
).
(2.87),
1 0
1

P

0 d k0 Qk max
2
2

> 0,

(2.89)

P = XQ1 X 0 , d = c XQ1 k.
, , .

2.11.
S = ((
x1 , y1 , . . . , (
xl , yl )), x
i Rn yi
{1, +1} 1 6 i 6 l.

, . (
xi , yi ) [42].
.
.
152

 SVM.
(SVM). SVM xi ) ,
x
i (
0
K(
x, x
). , , (2.43)
:
w
=

l
X

xi ),
yi i (

i=1

i .
2.5 :
f (
x) =

l
X

yi i K(
xi , x
) + b.

i=1

SVM S,
i .
(
xi , yi ) i .
. xi ))+
, i = 0, yi ((w
(
b) > 1. x
i . x
i , xi )) + b) 6 1, i > 0 i = i /C.
yi ((w
(
, ,
xi )) + b) < 1. yi ((w
(
i 6 C, C . :
i = 0 (
);
153

 i
,
, i .
(
xi , yi ).
p- (p-value):
pi =

|{j : j > i }|
.
l

0 6 pi 6 1. pi ,
(
xi , yi )
.
p- SVM.
S = ((
x1 , y1 , . . . , (
xl , yl )) x
l+1 ,
yl+1 {1, +1}.
 > 0.
-:
y {1, +1}
SVM S 0 = ((
x1 , y1 , . . . , (
xl , yl ), (
xl+1 , y)),
i , 1 6 i 6 l + 1
p-
p(y) =

|{j : j > l+1 }|

.
l+1

:
p(y) <  y,
;
p(y) >  y, y, p(y)
:
yl+1 = arg max p(y).
y

154


, ,
p- : P {pi 6 } 6 ,
P i
( . [42]).
. [42]
, SVM, ,
, , .
.
k- . x
k .
,
k , .
(
x, y), x
Rn , y D, D . , {
x1 , . . . , x
k } k
x
{y1 , . . . , yk } .
(
x, y) .
x
x
i yi = y x
x
i
yi 6= y:
(x,y) =

min16j6k,yj =y d(
x, x
j )
.
min16j6k,yj 6=y d(
x, x
j )

d(
x, x
0 ) .
(x,y) x

y, ..
(
x, y).

155

2.12.
1. 2.1.
2. Rn
(k = 3, 4, . . . ), (2.25).
3. K(x, y) :
p
K(x1 , x2 ) 6 K(x1 , x1 )K(x2 , x2 ) x1 , x2 X. (2.90)
(: (2 2)
K(xi , xj ) , .
).
4. , K(x, y) :
(i) K(x, x) > 0 x.
(ii) K(x, x) = 0 x, K(x, y) = 0 x y.
5. F
X, : f
f (x) . x X Kx F
, f (x) = (Kx f ).
K(x, y) = (Kx Ky ).
, K(x, y) = (Kx Ky ) .
6. K1 (x, y), K2 (x, y), . . .
X. , :
(i) 1 K1 (x, y) + 2 K2 (x, y), 1 , 2 > 0;
(ii) K(x, y) = lim Kn (x, y);
n

(iii) K1 (x, y)K2 (x, y) (:

() K = P P 0 );
P
(iv) K(A, B) =
K(x, y), A, B
xA,yB

X ( X).
156

 .
7. 2.5 , , F F1 : (f g) = lim (fn gn ), f1 , f2 , . . . g1 , g2 , . . . n
F1 ,
f, g F.
(i) ,
f1 , f2 , . . . g1 , g2 , . . . .
(ii) ,
F , .
(iii) . f (f Kx ) F (. [24]).
8. : RKHS F X, K(x, y) ,
K(x, ) F x X;
f (x) = (f, K(x, )) f F x X.
(i) , RKHS,
2.5: RKHS F
X, : f f (x) .
(ii) , K(x, y) RKHS F.
(iii) , K(x, y), , RKHS (. [24]).
9. X H X. H
. . K(x, y),
157

 Kx f f (x). , H RKHS.
10. ,
L2 ([0, 1], P )
R
f : [0, 1] R, R f 2 dP < ,
(f g) = f gdP RKHS.
11. , (2.68) b
i.
12. ,
S = ((
x1 , y1 ), . . . , (
xl , yl )) (.., , yi = 1 yj = 1
- 1 6 i, j 6 l) b
x
i , :
yi ((w
x
i ) + b) = 1, i > 0.
13. , (2.69) (2.73). (2.75) .
14. , Ki,j = (
xi x
j )ni,j=1
, x
1 , . . . , x
l .
15. (i)
.
n
P
(ii) H(p1 , . . . , pn ) =
pi ln pi
i=1
P
P

pi = 1,
ci pi = e.
16. (2.89) .
17. (2.68) .

2.13. SVM

SVM.
:
.
,
158

, .
.

, . .
, SVM.
() .

- . , .
, [1, 1] [0, 1].
, . ,
SVM :
1) K(
x, y) = (
x y),
2) K(
x, y) = ((
x y) + r)d , > 0,
3) K(
x, y) = e

k
x
y k2
2

4) K(
x, y) = tanh((
x y) + r).

k
x
y k2

K(
x, y) = e 2 . ,
. , ,
.
.
C . ,
,
159

 .
,
.

(cross-validation), N . ,
N 1
. ,
.
C
.
. C .
C ,
.
.

.
,
SVM . .
SVM :
http://www.csie.ntu.edu.tw, www.support-vector.net
SVM .
.
http://archive.ics.uci.edu .
1
160

 . MATLAB :
http://www.cs.toronto.edu
, USPS,

.
2

. , SVM
.
3

. .

161

 II

162

3.1.
: 1 , 2 , . . . , n1 , . . . , .
pn n
n 1 1 , 2 , . . . , n1 .
pn
n = 1 , i 0 1.
, pn n .
pn
.
. 1 1

,
, , , -

163

 , i ,

, p.
i {0, 1}.
, i = 1, 2, . . . i p = P {i = 1}
q = P {i = 0} = 1 p.
.
n = 1 , . . . n , n1 n2 , n1 +n2 = n.
pn1 (1 p)n2 ,
p. p ,

n p :
n

Z1

P ( ) =

pn1 (1 p)n2 dp.

.
3.1.

Z1

pn1 (1 p)n2 dp =

1
(n + 1)

n
n1

.

. R1
1
n1 . n1 = n pn dp = (n+1)
.
0

5000 (1826251 ).
,
. , . , ,
p
, p . , ,
1826252
.
1826253

164

,
Z1

pn1 +1 (1 p)n2 1 dp =

1
(n + 1)

.

n
n1 +1

,
Z1

n n1
p (1 p) dp =
n1 + 1
n1

n2

Z1

pn1 +1 (1 p)n2 1 dp =

n n1
1
=
n1 + 1 (n + 1)

=

n
n1 +1

1
(n + 1)

n
n1

.

. 4
n+1 = 1 n = 1 , . . . , n
P {n+1

P ( n 1)
= 1| n } =
=
P ( n )

1
(n+2)(nn+1
+1)
1

1
(n+1)(nn )

n1 + 1
.
n+2

, :
n1 + 1
,
n+2
n2 + 1
= 0| n } =
.
n+2

P {n+1 = 1| n } =
P {n+1

- .
:
Lp ( n ) = ln(pn1 (1 p)n2 ).
,
1 , n ,
n1 n2
1 p.
165

 ,
sup pn1 (1 p)n2 =

 n n1  n n2

06p61

Z1

L( ) = ln P ( ) = ln

pn1 (1 p)n2 dp.

, n
1 p0 . n
L( n ) Lp0 ( n ) 6
sup pn1 (1 p)n2
06p61
L( n ) inf Lp ( n ) = ln 1
=
06p61
R
pn1 (1 p)n2 dp
0




n1 n1 n2 n2
n

n
1
(n+1)(nn )

= ln

6 ln(n + 1).

, , , ln(n + 1) , .. ,
p0 , i .
, ,
.

n p
p
1/( p(1 p)) :
n

Z1

P ( ) =
0

pn1 (1 p)n2
p
dp.
p(1 p)

1 n n =
1 , . . . , n
n1 + 1/2
P (1| n ) =
.
n+1
166

 :
Z1
0

pn1 (1 p)n2
1  n1 n1  n2 n2
p
.
dp >
n
2 n n
p(1 p)

3.6
1 2.

:
sup pn1 (1 p)n2
L( n ) inf Lp ( n ) = ln
06p61

6 ln

n1 n1 n2 n2
n
n




n1 n1 n2 n2
1

n
2 n n

06p61
R 1 pn1 (1p)n2

0 p(1p) dp

1
6 ln(2 n) = ln n + ln 2.
2

,
.

3.2.
, i , ( ), ,
.
, , n = 1
n- , pn
,
1 , 2 , . . . , n1 n 1 .
,
, . , 80% ,
167

 pn = 0.8, .. n pn ,
pn p , p
.
pn [0, 1].
. ,
n = Sn
n = 1, 2, . . . .
. , . ,
. pn
n.
3.3
n {0, 1}, ,
: n [0, 1]; pn [0, 1].
pi ,
, P , ..
pn = P (n = 1|1 , 2 , . . . , n1 ) n = 1, 2, . . .
n = 1
1 , 2 , . . . , n1 .
P .
. , , .
1 , 2 , . . . , n1 , . . .
, . ,
.
168

FOR n = 1, 2, . . .
pn [0, 1].
n {0, 1}.
ENDFOR
. 3.1:
P
. ,
. ,
, .
, 1 , 2 , . . . , n1 ,
.

:
p [0, 1]
Pn
i
Pn i=1&pi p
p ,
(3.1)

p
p
i
i=1&pi p
(3.1)
n . , p
. pi p .
,
. 3.1.
, . , n pn ; n ,
pn
.
169

 ,
[16]. I =
[a, b], (a, b], [a, b), (a, b) [0, 1]


1, p I,
I(p) =
0 .
p1 , p2 , . . . 1 , 2 , . . . , I(p) [0, 1]
, ..
Pn
I(p )( pi )
i=1
Pn i i
0,
(3.2)
i=1 I(pi )
(3.2)
n . I(pi ) , i,
pi i .
, , ,
. , f = 1 , 2 , . . . ,

1, pi < 12 ,
i =
0 ,
pi = f (1 , . . . , i1 ), i = 1, 2, . . . . , I = [0, 12 ) I = [ 12 , 1) (3.2) .
= 1 , 2 , . . . .
i pi
.

. P[0, 1]
[0, 1].
170

FOR n = 1, 2, . . .
Pn
P([0, 1]).
n {0, 1}.
pn Pn .
ENDFOR
. 3.2:
, . 3.1, . 3.2. . ,
n , , pn
.
.2
n1 = 1 , . . . , n1 . n1 Pn
Q
Pr =
n=1 Pn p1 , p2 , . . . , pi [0, 1],
i = 1, 2, . . . . = 1 , 2 , . . . .
, Pr , , n p1 , . . . , pn1 n.3
, n Pn (d
pn ; p1 , . . . , pn1 ) ( ),
p1 , . . . , pn [0, 1]. ,
P1 (d
p1 ).
n
Qn [0, 1]n .
2

7.3.
. 3.2,
n p1 , . . . , pn1 , . pn .
3

171

 A [0, 1]n
Qn (A) =
Z
=

Z
P1 (d
p1 )

Z
P2 (d
p2 ; p1 ) . . .

Pn (d
pn ; p1 , . . . , pn1 )1A (
p1 , . . . , pn ),


1A (
p1 , . . . , pn ) =

1 (
p1 , . . . , pn ) A
0 .

[4] Pr [0, 1]
p1 , p2 , . . . ,
Qn (A) = Pr(A [0, 1] )
n A [0, 1]n .
, I [0, 1] Pr (3.2).
[17], [21]
i
.
i 0 1: > 0,


n
1 X



lim sup 
I(
pi )(i pi ) 6

n  n
i=1

Pr- 1, p1 , p2 , . . .
Pr, I(p) [0, 1].
i [0, 1]
3.3.

172

3.3.
. 1 , 2 , . . . {0, 1} [0, 1], . , pn [0, 1] n 1 , . . . , n1 . : 1 .

.
[0, 1]
= 1/K
vi = i, i = 0, 1, . . . , K. V
. p [0, 1]
,
p :
X
p=
wv (p)v = wvi1 (p)vi1 + wvi (p)vi ,
vV

p [vi1 , vi ], i = bp/ + 1c,

p vi1
vi p
wvi1 (p) = 1
, wvi (p) = 1
.

wv (p) = 0 v V .
p, , , vi1 wvi1 (p) vi wvi (p).
,
.
p1 , . . . , pn1 ( p1 = 0).
pn .

n1 (v) =

n1
X

wv (pi )(i pi ).

i=1

173


(n (v))2 = (n1 (v))2 + 2wv (pn )n1 (v)(n pn ) +
+(wv (pn ))2 (n pn )2 .

(3.3)

(3.3) v :
X
X
(n (v))2 =
(n1 (v))2 +
vV

vV

+2(n pn )

wv (pn )n1 (v) +

vV

(wv (pn ))2 (n pn )2 .

(3.4)

vV

X
wv (p)n1 (v) =

vV
n1
X

wv (p)
i=1
vV
n1
X X

wv (pi )(i pi ) =

wv (p)wv (pi ))(i pi ) =

i=1 vV
n1
X

(w(p)

w(p
i ))(i pi ) =

i=1

n1
X

K(p, pi )(i pi ),

i=1

w(p)

= (w1 , . . . , wvK ) = (0, . . . , wvi1 (p), wvi (p), . . . , 0)

, p [vi1 , vi ],
K(p, pi ) = (w(p)

w(p
i ))
174

(3.5)

 ().
K(p, pi ) .
(3.4)
pn .
, pn pn = p
X

wv (p)n1 (v) =

vV

n1
X

K(p, pi )(i pi ) = 0,

(3.6)

i=1

. (3.6) ( p )
pn , pn = 1,
, pn = 0.
pn .
(3.4) 1. ,
|i pi | 6 1 i, n
X
X
(wv (pn ))2 (n pn )2 6
wv (pn ) = 1.
vV

vV

(3.4), pi
,
X
vV

(n (v))2 6

n X
X

(wv (pi ))2 (i pi ))2 6 n.

(3.7)

i=1 vV

pi , v V wv (pi ) ( , p
wv (p)
, pi ). I(p)
[0, 1]. i
I(
pi )(i pi ) 4
X
E(I(
pi )(i pi )) =
wv (pi )I(v)(i v).
(3.8)
vV
4

i ,

p1 , . . . , pi1 .

175


(. 4.9 ) Pr- 1 :


n
n

1 X
1X


I(
pi )(i pi )
E(I(
pi )(i pi )) 0
(3.9)

n

n
i=1

i=1

n .
pi
wv (p)



X
X


wv (pi )I(v)(i v)
wv (pi )I(v)(i pi ) < (3.10)



vV

vV

i.
{n (v) :
v V } {I(v) : v V }, (3.10),


n X
X



wv (pi )I(v)(i pi ) =



i=1 vV


n
X

X


=
I(v)
wv (pi )(i pi ) 6


i=1
vV
sX
sX
(n (v))2
I(v) 6
6
vV

vV

Kn,

(3.11)

K = 1/ - .
(3.10) (3.11), (3.8) :
 n

X



E(I(
pi )(i pi )) =




 n i=1

X X


wv (pi )I(v)(i v) 6
(3.12)
=


i=1 vV
p
6 n + n/
(3.13)
176

 n.
(3.12) (3.9) , c Pr- 1 :


n

1 X


lim sup 
I(
pi )(i pi ) 6 .

n  n

(3.14)

i=1

.
3.1. > 0 ,
,
I [0, 1]


n
1 X



lim sup 
I(
pi )(i pi ) 6


n
n
i=1

, I(p)
.
ns , s = 1, 2, . . . , = s s 0 s ,
:
3.2. , ,
I [0, 1]
n
1X
lim
I(
pi )(i pi ) = 0
n n
i=1

.
3.5.1.

3.4.
:

177

. 2.1. 1 , 2 , . . . p1 , p2 , . . .

- , ( , , ); ;
- , , .
.
178

 .
, , , (. [21]).
.
, [17], [21], . [44] [47].
.
: , .
,
pi , (3.6) 3.3. ,
, .
X Rm m-
x
= (x1 , . . . , xm ),
m-
v
um
uX
k
xk = t
x2i .
i=1

, .
K0 = 1.
.
FOR n = 1, 2, . . .
x
n X.
p Sn : [0, 1] R.
pn [0, 1].
yn {0, 1}.
n
Kn = Kn1 + Sn (pn )(yn pn ).
ENDFOR

179

 , , .
3.3. ,
K0 > K1 > . . . Kn > . . .
. .
n
pn . Sn (p)
p [0, 1], pn = 1. Sn (p)
p [0, 1], pn = 0.
,
Sn (p) = 0,

(3.15)

p, .
pn .
, pn
, p
Sn (p), ..
K0 > K1 > . . . Kn > . . .
n. 4
K((p, x
), (p0 , x
0 )) -

([0, 1] X)2 . :
K((p, x
), (p0 , x
0 )) =


(p pi )2 k
xx
0 k2
= exp

,
12
22

(3.16)

1 , 2 .
,
n :
Sn (p) =

n1
X

K((p, x
n ), (pi , x
i ))(yi pi ).

i=1

180

 , 3.3. N

KN K0 =

N
X

Sn (pn )(yn pn ) =

n=1

1
2

N n1
X
X
n=1 i=1
N X
N
X

K((pn , x
n ), (pi , x
i ))(yi pi )(yn pn ) =
K((pn , x
n ), (pi , x
i ))(yi pi )(yn pn )

n=1 i=1
N

1X

K((pn , x
n ), (pn , x
n ))(yn pn )2 .
2

(3.17)

n=1

2.5 H : [0, 1] X H ,

K(a, b) = ((a)
(b))
a, b [0, 1] X, H ( k k ).
cH = sup k(a)k
a

(embedding constant). , : cH < .

(3.17)

2
N

1 X

KN K0 = 
(pn , x
n )(yn pn )



2

1
2

n=1
N
X

n, x
||((p
n )(yn pn )||2 .

n=1

x
cH = sup ||(p,
)|| < .
p,
x

181

(3.18)

 3.3 KN K0 6 0 N .
(3.18)

2
N

1 X
1

(pn , x
n )(yn pn ) 6 N C 2 .


2 
2

(3.19)

n=1

(3.19)


N
 X



n, x
(p
n )(yn pn ) 6 N C.




(3.20)

n=1

,
 N


C
1 X

(pn , x
n )(yn pn ) 6 .


N 
N
n=1

, , 3.3.
{(p , x
)}, (p , x
)
[0, 1] X, ..
K((p , x
), (p, x
)) = I(p ,x ) (p, x
).

(3.21)

Ip (p)
(3.16).
pi
3.1.


N
1 X

C2


I(p ,x ) (pn , x
n )(yn pn ) 6

N

N
n=1
(p , x
) [0, 1] X.

182

(3.22)

. (p, x
) H,
, x
x
K((p , x
), (p, x
)) = I(p ,x ) (p, x
) = ((p
) (p,
)).
(3.20)

N

X



I(p ,x ) (pn , x
n )(yn pn ) =



n=1

!
!
N


X



n, x
,x
=
(p
n )(yn pn ) (p
) 6


n=1
 N

X




n, x
6 
(p
n )(yn pn ) ||(p
,x
)|| 6 C 2 N .


n=1

(3.22). 4

N
X

I(p ,x ) (pn , x
n )

n=1

(pn , xn ),
(p , x
).
(3.22)



P
N



I
(p
,
x

)(y

p
)
n
n
n
n
(p
,
x
)


C2 N

 n=1
.
(3.23)
6 N

N


P
P


I(p ,x ) (pn , x
n )
n )
 n=1 I(p ,x ) (pn , x

n=1
(3.23)
N
X

I(p ,x ) (pn , x
n ) 

N,

n=1

..
.
183

3.5.

. 5 (Algorithmic trading) ( ).
- [14],

.
, .
,
. .

. .
5.8 , .
.

. 5

184


3.3 3.4. ,
3.2.
.
3.4 ,
,
.
[48].
, S1 , S2 , . . . ,
, . , , 0 6 Si 6 1 i.
. , C

. : C > 0, C
C , C C . ,
.
(
), i
Ci ,
(
) ( ).
, . ,
i- ()
Ci (Si Si1 ).
:

185

 i M
i ,
M
; D , D(xi ) ,
D ,
[0, 1], xi [0, 1],

. , Ci
i M D(xi ) D .
M
xi , 3.4, . xi
[0, 1] . ,
Si .
D i
xi ( ) D(xi ) .
.
M . M
Sj1 xj j 6 i.
, xi , D .
D .
. M . 3.3.
K . [0, 1]
K = 1/K vi = i, i = 0, 1, . . . , K. V
. p [0, 1]
,
p:
X
p=
wv (p)v = wvi1 (p)vi1 + wvi (p)vi ,
(3.24)
vV

p [vi1 , vi ], i = bp1 / + 1c, wvi1 (p) = 1 (p vi1 )/

186

wvi (p) = 1 (vi p)/. wv (p) = 0

v V.
p vi1 wvi1 (p) vi wvi (p).
w(p)

= (wv (p) : v V ) . , p vi1

wvi1 (p) vi wvi (p).
z, z 0 [0, 1] K(z, z 0 ) =
(w(z)

w(z
0 )), .

, , 1 > 2 > 0.

i . .
. i , i = 1, 2, . . . . ,
M
, i pi
,
. 3.4 3.5.1 . , .., pi .
, , . 3.3, i M
Si . i = 1 M
i = 1.
M
( ) Si1 , .., Si1 .
i , M
, . 3.3.
i > 0 M , M
; ,
. D. 6
6

187

 K0D = 0 K0M = 0.
FOR i = 1, 2 . . .
xi [0, 1].
i: = s ,
ns < i 6 ns+1 ( ns s , s = 1, 2, . . . ,
(3.47) 3.5 ).
pi ,
. 3.4, .
: pi .
: Si1 .
i Si1 ,
M M

1, pi > Si1 ,

Mi =
1, .
D D(xi ) Si1 , D
[0, 1].
Si .
i Si
M M
M +M
i (Si Si1 ).
: KiM = Ki1
D D(xi ) Si
D + D(x )(S S
: KiD = Ki1
i
i
i1 ).
ENDFOR
. 3.3:

188

 , .
( ) .
i M
pi . . 3.4 3.5.1.
M . D.
kDk = sup |D(x)|, D
06x61

[0, 1]. kDk+ = max{1, kDk } .

, S1 , S2 , [0, 1] x1 , x2 ,
[0, 1] . 3.3.
3.4, , , 1, , :
3.4. , D :
lim inf
n

1
D
KnM kDk1
+ Kn > 0
n

(3.25)

, .
, (3.25) D . (
)
. ,
.

189

 :
lim inf
n

1
KnM KnD > 0
n

D , kDk 6 1.
( 3.25)
M :
D(x) = 0 x, :
3.2. M
:
lim inf
n

KnM
>0
n

.
3.4 3.5.2, , , , , 3.5.1.

3.5.1.

, i.
M
, 3.2
3.3. S [0, 1]2 = [0, 1] [0, 1]


1, (p, x) S,
IS (p, x) =
0, ,
p, x [0, 1]. 3.5.2
S = {(p, x) : p > x} S = {(p, x) : p 6 x}.
190

 , . 3.4 3.5.2, i,
RKHS M

, .
, RKHS, , .
, F , X RKHS X, f f (x) x X.
k kF F.
cF (x) = sup |f (x)|. F kf kF 61

: cF = sup cF (x).
x

RKHS F X = [0, 1]
: cF < . RKHS F = H 1 ([0, 1]),

f : [0, 1] R
qR
R1
1
2
kf kF < , kf kF =
(f (t)) dt + 0 (f 0 (t))2 dt.
0q

2
cF = coth 1 = 1+e
(. [45]).
1e2
F RKHS X
(f g) f, g F.
x X kx F , f (x) = (kx f )
f F.
K(x, y) = (kx ky ).
, K(x, y)
RKHS F, :
X F, K(x, y) = ((x) (y)).
F RKHS [0, 1]
cF , k kF
R(x, x0 ) [0, 1].
, S1 , S2 , [0, 1] x1 , x2 , [0, 1]

191

 ,
. 3.3.
3.5.  > 0.
p1 , p2 , . . . , :
> 0, S [0.1]2
n, 1 ,
 n

X



IS (
pi , zi )(Si pi ) 6



i=1
r
n 2
ln
(3.26)
6 18n3/4+ (c2F + 1)1/4 +
2
p1 , p2 , . . . p1 , p2 , . . .
z1 , z2 , . . . z1 , z2 , . . . , zi = Si1 ,
i = 1, 2, . . . ;
D F


n
q
X



D(x
)(S

p
)
6
kDk
(c2F + 1)n

i
i
i 
F



(3.27)

i=1

n.
S [0, 1]2 , 1,
n

1X
lim
IS (
pi , zi )(Si pi ) = 0.
n n

(3.28)

i=1

. .
> 0 3.3 3.4. S1 , S2 , . . . . , :
0, .
192

 3.1. 3.5
, (3.27) D RKHS F n.
> 0, S [0, 1]2 n, 1 , :
 n

X



IS (
pi , zi )(Si pi ) 6



i=1
r
r
n(c2F + 1)
n 2
n + 2
+
ln .

2
. , p1 , . . . , pn1 ( p1 = 1/2). pn
pn .
V = {v0 , . . . , vK } (3.24). , pn vi1 wvi1 (pn ) vi
wvi (pn ).
zn = Sn1 vs1 wvs1 (zn ) vs
wvs (zn ), zn [vs1 , vs ].
Wv (pn , zn ) = wv1 (pn )wv2 (zn ), v = (v 1 , v 2 ) and v 1 , v 2
V , W (pn , zn ) = (Wv (pn , zn ) : v V 2 ) , V 2 = V V . : K(p, z, p0 , z 0 ) = (W (p, z) W (p0 , z 0 )).
R(x, x0 ) : R(x, x0 ) = ((x)
(x0 )).
Un (p) =

n1
X

(K(p, zn , pi , zi ) + R(xn , xi ))(Si pi ).

(3.29)

i=1

p1 , p2 , . . . . 3.4
3.1.
p1 , p2 , . . . (. . 3.4).
193

 p1 = 1/2. .
FOR n = 1, 2 . . .
Un (p) =

n1
P

(K1 (p, xn , pi , xi ) + K2 (zn , zi ))(Si pi ).

i=1

Un (p) > 0 p [0, 1], pn = 1;

Un (p) < 0 p [0, 1], pn = 0.
, pn -
Un (p) = 0 (
).
ENDFOR
. 3.4:
pn ,
. 3.4, n Sn U (pn )(Sn pn ) 6 0.

194

, , N :
0>

N
X

Un (pn )(Sn pn ) =

n=1

N n1
X
X
(K(pn , zn , pi , zi ) + R(xn , xi ))
n=1 i=1

(Si pi )(Sn pn ) =
=

1
2

N X
N
X

K(pn , zn , pi , zi )(Si pi )(Sn pn )

n=1 i=1
N

1
2

1X
(K(pn , zn , pn , zn )(Sn pn ))2 +
2

n=1
N
N
XX

R(xn , xi )(Si pi )(Sn pn )

n=1 i=1
N

1X
(R(xn , xn )(Sn pn ))2 =
2
n=1

2
N


1 X

=
W (pn , zn )(Sn pn )

2

(3.30)

n=1

N
1X

kW (pn , zn )k2 (Sn pn )2 +

(3.31)

n=1

N
2

1
X

+
(xn )(Sn pn )

2
n=1

N
1X

k(xn )k2F (Sn pn )2 .

(3.32)

n=1

(3.31), k k , (3.32),
k kF F.

195

 (Sn pn )2 6 1 n
X
kW (pn , zn )k2 =
(Wv (pn , zn ))2 6
vV 2

Wv (pn , zn ) = 1,

vV 2

(3.31) N .
k(xn )kF = cF (xn ) cF (x) 6 cF x, (3.32) c2F N .
:


N
X
q


(3.33)
W (pn , zn )(Sn pn ) 6 (c2F + 1)N



n=1


N
q
X



(3.34)
(xn )(Sn pn ) 6 (c2F + 1)N



n=1

N . :
n =
q
k
n k 6 (c2F + 1)n n.

n
P

W (pi , zi )(Si pi ). (3.33),

i=1

n = (n (v) : v V 2 }. v:
n (v) =

n
X

Wv (pi , zi )(Si pi ).

(3.35)

i=1

I(v) (3.35), I S [0, 1]2 , v V 2 . I = (I(v) : v V 2 ),

n = (n (v) : v V 2 ) :


 n

X X


=
W
(p
,
z
)I(v)(S

p
)
v
i
i
i
i


 i=1 vV 2





n
X

X

=
I(v)
Wv (pi , zi )(Si pi ) =
vV 2

i=1
q
k
= |(I
n )| 6 kIk
n k 6 |V 2 |(c2F + 1)n
(3.36)
196

 n, |V 2 | = (1/ + 1)2 6 4/2 .

pi , v V
wv (pi ). , zi ,
v V wv (zi ).
S [0, 1]2 I . i I(
pi , zi )(Si pi )
E(I(
pi , zi )(Si pi )) =
=

Wv (pi , zi )I(v)(Si v 1 ),

(3.37)

vV 2

v = (v 1 , v 2 ).
4.7
(. 4.6 4.7 ), ,
> 0, S n, 1 :


n
n

X
X


I(
pi , zi )(Si pi )
E(I(
pi , zi )(Si pi )) 6



i=1
i=1
r
n 2
6
ln .
(3.38)
2
, :
X
X
Wv (pi , zi )I(v)(Si pi )
Wv (pi , zi )I(v)(Si v 1 )
vV 2

vV 2

i, v = (v 1 , v 2 ). (3.37) i = 1, . . . , n (3.36), :


n
X



E(I(
p
,
z

)(S

))

i i
i
i =


i=1



 n
X X

1 

Wv (pi , zi )I(v)(Si v ) 6
=
 i=1 vV 2

q
6 n + 2 (c2F + 1)n/2
(3.39)
197

 n.
(3.38) (3.39) S n, 1:


n

X


I(
pi , zi )(Si pi ) 6



i=1
r
q
n 2
ln .
(3.40)
6 n + 2 (c2F + 1)n/2 +
2
:


N
X



D(xn )(Sn pn ) =



n=1


N
X



=  (Sn pn )(D (xn )) =


n=1
 N
!
 X



=
(Sn pn )(xn ) D  6


n=1
N

X



6 (Sn pn )(xn ) kDkF 6


n=1
qF
6 kDkF (c2F + 1)N .
. 4

3.5.
q

n+2 (c2F + 1)n/2 (3.39) (3.40)

1
1
= 2(c2F + 1) 4 n 4 .
, (3.39)
q

1
3
n + 2 n(c2F + 1)/2 = 2n = 2 2(c2F + 1) 4 n 4 .
(3.41)
, 2n (3.39).
(3.26) 1 > 2 > . . . , s 0 s .
198

 n1 < n2 < . . . s
ns 6 n < ns+1 [0, 1] s .
n1 = 1 1 = 1.
n2 , n3 , . . . ,


n
X



E(I(
p
,
z

)(S

))
(3.42)

i i
i
i  6 4(s + 1)s n


i=1

ns 6 n 6 ns+1 s > 1.
s. , ns (s > 1) ,


n
X



E(I(
pi , zi )(Si pi )) 6 4ss1 n
(3.43)



i=1

ns1 6 n 6 ns . ,
n

s
X



E(I(
pi , zi )(Si pi )) 6 4ss ns .
(3.44)



i=1

ns+1 . pi (
pi ),
= s+1 . ns
(, (3.45) (3.46); p1 , . . . , pns ).
pi i > ns , . ns+1 ,

199


n
 n

s+1
s
X
 X


 

E(I(
pi , zi )(Si pi )) 6 
E(I(
pi , zi )(Si pi )) +


 

i=1
 ni=1

ns
s+1
 X

X


+
E(I(
pi , zi )(Si pi )) +
E(I(
pi , zi )(Si p
i )) +


i=ns +1
i=1
n

s
X



+
E(I(
pi , zi )(Si p
i )) 6 4(s + 1)s+1 ns+1 .
(3.45)


i=1

(3.45) 4ss ns (3.44). 2s+1 ns+1 2s+1 ns , , = s+1 , (3.41).

(3.39) ns .
(3.44)
ns+1 >

2ss + s+1
ns .
s+1 (2s + 1)


 n

n
s
 X

X

 

E(I(
pi , zi )(Si pi )) 6 
E(I(
pi , zi )(Si pi )) +


 

i=1
 i=1

ns
n

 X
X


+
E(I(
pi , zi )(Si pi )) +
E(I(
pi , zi )(Si p
i )) +


i=ns +1
i=1
n

s
X



+
E(I(
pi , zi )(Si p
i )) 6 4(s + 1)s n
(3.46)


i=1

ns < n 6 ns+1 . : 4ss ns 6 4ss n

(3.44). 2s+1 n 6
2s n 2s+1 ns 6 2s n, . (3.39)
s . (3.43) .
(3.42) s


n

X


(3.47)
E(I(
pi , zi )(Si pi )) 6 4(s + 1)s n



i=1

200

 n > ns , s s+1 6 s (1
1
s+2 ) s.
, ns s
, .
 > 0 M = d2/e, dre, t r. ns = (s + M )M

1 1
s = 2(c2F + 1) 4 ns 4 . ,
, ns s ,
s; s > s0 . ni = ns0 1 6 i 6 s0 . ,
(3.43) (3.44) ni .
(3.43) ns 6 n < ns+1
4(s + 1)s n 6 4(s + M )s ns+1 =
2
1
M
= 4 2(cF + 1) 4 (s + M )(s + M + 1)M (s + M ) 4 6
1

6 18(c2F + 1) 4 ns4

+2/M
1
4

6 18(c2F + 1) n

3
+
4

,


n
X



2
1/4 3/4+
E(I(
p
,
z

)(S

))
n

i i
i
i  6 18(cF + 1)



(3.48)

i=1

n.
pi , zi )(Si pi ) E(I(
pi , zi )(Si pi )) =
q Vi = I(
1
2n

ln 2 , > 0. E p1 , . . . , pi1 . Vi
-
p1 , z1 , p2 , z2 . . .
> 0:

)
( n
1 X 
2


(3.49)
P 
Vi  > 6 2e2n

n
i=1

n (. 4.6 4.7).
201

 (3.48) (3.49), , > 0, S

n, 1 ,


r
n

X
n 2


2
1/4 3/4+
ln .
I(
pi , zi )(Si pi ) 6 18(cF + 1) n
+



2
i=1

(3.28) (3.26)
. 3.4 (. 3.5.2). 3.5 . 4

3.5.2. 3.4
i pi , 3.5.1, , ,
pi : = s =

1
1
2(cF + 1) 4 (s + M ) 4 ns = (s + M )M , ns 6 i < ns+1 .
Si1 Si1
. 3.5, zi = Si1 zi = Si1 .

;


n
s
 X
X


I(
pi > Si1 )(Si1 Si1 ) 6
(nt+1 nt )t 6



t=0

i=1

1
4

6 4(c2F + 1) ns4 6 4(c2F + 1) 4 n 4 ,

(3.50)

ns 6 n < ns+1 .
D(x) RKHS F.
, (3.50) , I(
pi > Si1 )
1
kDk+ D(xi ). , , D(x) > 0 x

1, pi > Si1 ,

Mi =
0, .

202

 :
1

1 (n) = 4(c2F + 1) 4 n 4 ,
r
3
1
n 2
2 (n) = 18n 4 + (c2F + 1) 4 +
ln .
2
q
3 (n) = (c2F + 1)n

(3.51)
(3.52)
(3.53)

i = 1, . . . n. .
> 0, 1 ,

203

 :
KnM =

n
X

i (Si Si1 ) =
M

i=1

X
pi >Si1

pi >Si1

(Si pi ) +

(Si Si1 ) =

(
pi Si1 ) +

pi >Si1

(Si1 Si1 ) >(3.54)

pi >Si1

>

(
pi Si1 ) 1 (n) 2 (n) >(3.55)

pi >Si1

> kDk1
+

n
X

D(xi )(
pi Si1 ) 1 (n) 2 (n) =

i=1

= kDk1
+

n
X

D(xi )(pi Si1 ) + kDk1

+

i=1

D(xi )(
pi pi )

i=1

kDk1
+
>

n
X

n
X

D(xi )(Si1 Si1 ) 1 (n) 2 (n) >(3.56)

i=1
n
X
kDk1
D(xi )(pi
+
i=1

Si1 ) 31 (n) 2 (n) =(3.57)

kDk1
+

n
X

D(xi )(Si Si1 )

i=1

kDk1
+

n
X

D(xi )(Si pi ) 31 (n) 2 (n) >(3.58)

i=1

>

kDk1
+

n
X

D(xi )(Si Si1 )

i=1

31 (n) 2 (n) kDk1

+ kDkF 3 (n) =
1
D
= kDk1
+ Kn 31 (n) 2 (n) kDk+ kDkF 3 (n).

(3.54) (3.55)
(3.26) 3.5 (3.50); , (3.51) (3.52). (3.56) (3.57) (3.50); (3.50)
. (3.57) (3.58) 204

 (3.27) 3.5; , (3.53).

, c > 0,
> 0 n, 1 ,
r
3
n 2
1 D
M
+
Kn > kDk+ Kn cn 4
ln .
(3.59)
2
(3.25) (3.59). -. , An

P
P (An ) , , An n=1

n 0.
, ,

q
1
2
. = 2n ln . =
2

2e2n . (3.59) :
1
1 M
1 1 D
Kn kDk+
Kn > cn 4 + .
n
n

(3.60)

(3.59), n > 0, (3.60)

P
2
2
e2n
2e2n .
n=1

> 0, -, > 0,
(3.60)
n. , :
lim inf
n

1
D
KnM kDk1
+ Kn > 0
n

.
3.4 D F. D F, ,
.

D(x)
RKHS.
205

 RKHS F X , X f X
k k F :
 > 0 D F ,
sup |f (x) D(x)| 6 
xX

(. [35], 4).
X = [0, 1].
F = H 1 ([0, 1]) RKHS (. : [35],
[45]).
RKHS [0, 1] 3.4:
pi , i
M
,
f .
lim inf
n


1 M
f
Kn kf k1
K
n >0
+
n

(3.61)

, .
(3.59)
f [0, 1] D () RKHS F.
i , (3.61),
M
.
, (3.61)
f .
(3.59)
RKHS.

206

3.6.
1. , 1 n
n = 1 , . . . , n
P (1| n ) =

n1 + 1/2
.
n+1

2. , :
Z1
0

pn1 (1 p)n2
1  n1 n1  n2 n2
p
dp >
.
n
2 n n
p(1 p)

3. . 1 , 2 , . . . ,
0 1, ,
t

1X
i
t t
lim

i=1

.
, p1 , p2 , . . . , p1 = 0
i1

1 X
pi =
j
i1
j=1

i > 1, 1 , 2 , . . . .
4. 3.4
D F, .
5. 3.4
D(x).

3.7.
, 3.3, . (3.6)
207

0 2

(3.5) K(p, p0 ) = e(pp ) ,

> 0.7
K(p, p0 ) = cos(2(p p0 )2 ) p, p0 [0, 1].
FINAM:
http://finam.ru
,
- : S0 , S1 , S2 , . . . , Sn
, Si [0, 1] i.
1
3.3.
p1 , p2 , . . . , pn
1 , 2 , . . . , n , i {0, 1}. .
S0 , S1 , . . . , Sn1 , Si =
Si Si1 .

1, Si > ,
i =
0 ,
.
, pi > .
.
2
- .
S0 , S1 , . . . , Sn1 , Si [0, 1].

p1 , p2 , . . . , pn S0 , S1 , . . . , Sn1 .
, pi > Si1 +   > 0.
.
-
, .
7
. (3.16).

208

.
. ,
.
.
.
, . , , .
,
.

, 209

 , , . . ,
, .
.
.
.
.
,
, . , , , ;
, . .
, ,
.
.
, . , . , .
. , .
.

4.1.
, . 210

 0 1. N (),
pit {0, 1}, i = 1, . . . , N .

1 , . . . , t1 pi1 , . . . , pit ,
i = 1, . . . , N , pt {0, 1}.
,
i : pit = t t.
. t = 1, 2, . . .
,
:
Bt = {i : pij = j 1 6 j 6 t 1}
pt = 1,
1
, pt = 0. ,

1, |{i : i Bt , pit = 1}| > |Bt |/2,
pt =
0, .
4.1. , i ,
pit = t t.
dlog2 N e , N . 1
. t,
: |Bt+1 | 6 d|Bt |/2e. |Bt | > 1 t.
|Bt | dlog2 N e. 4
, , , .
,
[27].
(.
. 4.1). : i, i = 1 , . . . , N , , .
1

dre r.

211

FOR t = 1, 2, . . . , T
i pit {0, 1}, i = 1, . . . , N

pt W M A():
P i
P
IF
wt >
wti
i:pit =0

i:pit =1

THEN pt = 0
ELSE pt = 1
ENDIF
t {0, 1}
:
Et = {i : pit 6= t } i,
t.
:

(1 )wti , i Et ,
i
wt+1 =
wti , .
ENDFOR
. 4.1: W M A()
, . ,
.
 , 0 <  < 1.
T
P
LiT =
|pit t | i,
t=1

LT =

T
P

|pt t | , .. -

t=1

W M A() T .
4.2. i


 
2
2
i
LT 6
LT +
ln N
1

t.

212

. Wt =

N
P
i=1

wti . m = min LiT

16i6N

T . i. i m .
Wt > wti > (1 )m

(4.1)

t , 1 6 t 6 T .
,
t,
X
wti > Wt /2.
iEt

,
Wt+1 =

(1 )wti +

wti =

i6Et

iEt

N
X

wti 

i=1

wti 6

iEt



6 Wt 1
.
2
Wt+1 6 Wt t.
T > 0
TY
1
WT
Wt+1 
 M
=
6 1
,
W0
Wt
2

(4.2)

t=0

M = LT W M A()
T .
N
P
, W0 =
w0i = N . (4.1) (4.2)
i=1


(1 )m
WT
 M
<
6 1
.
N
W0
2

213

 , :


m ln(1 ) ln N < M ln 1
2

m ln(1 ) ln N < M
2


1

m ln
+ ln N > M
1
2
  
  
1
2
2
ln
+
ln N > M
m

1



 
2
2
m+
ln N > M,
(4.3)
1

(4.3) ln(1 + x) 6 x, x > 1.
(4.3)



1
1
1
ln
6
.
y
1y
1y
ln(1 + x) 6 x
x = y/(1 y). 4
4.2 , WMA ,
.
4.1 4.2.
, -, .
1989 Weighted Majority Algorithm [27]. , 1990
, ..
(Aggregating Algorithm) ,
(. [38]).

214

4.2.

,
(- ). [18].
: ,
.
, ( ). ,
,
.
. :
, 1, 2, . . . , N , . ,
, .
t = 1, 2, . . . , T pt = (p1t , . . . , pN
t ),
1
N
i
pt + + pt = 1 pt > 0 i = 1, 2, . . . , N . t lti ,
i = 1, 2, . . . , N . t

(
pt lt ) =

N
X

pit lti ,

i=1

lt = (lt1 , . . . , ltN ) t.
,
, , lti [0, 1] i t.
, 215

 , , ; lt
1 lt .
i t = 1, 2, . . . , T

T
X
i
LT =
lti .
t=1

,
t = 1, 2, . . . , T
T
X
LT =
(
pt lt ).
t=1

pt , t = 1, 2, . . . , T ,
RT = LT min LiT .
i

Hedge() [18] (. . 4.2). (0, 1),

w
1 = (w11 , . . . , w1N ). ,
N
P

w1i = 1.
i=1

Hedge() .
4.1.
l1 , . . . , lT 1, . . . , N
!
N
X
ln
wTi +1 6 (1 )LT ,
(4.6)
i=1

LT Hedge() T .
. r 6 1 (1 )r r [0, 1] 0 < < 1.
216

FOR t = 1, 2, . . . , T
:
pt =

w
t
.
N
P
i
wt

(4.4)

i=1

i lti , i = 1, 2, . . . , N . lt =
(lt1 , . . . , ltN ) t.
: lt = (
pt lt ).
:
i

i
wt+1
= wti lt

(4.5)

i = 1, . . . , N .
ENDFOR
. 4.2: Hedge()
(4.4) (4.5),
N
X

i
wt+1
=

i=1

N
X

N
X

wti lt 6

i=1

wti (1 (1 )lti ) =

i=1

N
X

!
wti

(1 (1 )(
pt lt )).

i=1

217

(4.7)

 (4.7) t = 1, . . . , T ,
N
X

wTi +1 6

i=1

T
Y

(1 (1 )(
pt lt )) 6

t=1

6 exp (1 )

T
X

!
(
pt lt ) .

t=1

1 + x 6 exp(x) x.
N
P

w1i = 1 .
i=1

. 4
(4.6)

N
P i
ln
wT +1
i=1
.
LT 6
1

(4.8)

(4.5)
wTi +1 = w1i

T
Y

lt = w1i LT .

(4.9)

t=1

.
4.3.
l1 , . . . , lT i = 1, . . . , N
i T
LT 6

ln(w1i ) LiT ln
.
1

(4.10)

w1i = N1 i.
(4.10)
LT 6

ln(1/)
ln N
min LiT +
.
1 i
1
218

(4.11)

 (4.11) , Hedge() ,
ln(1/)
ln N
1 1 .
[39] , (4.11) . .
4.4. B
. , a c,
N l1 , . . . , lT t ) t = 1, . . . , T ,
, lt = (l1t , . . . , lN
LT (B) 6 c min LiT + a ln N.
i

(0, 1) :
c>

ln(1/)
1
a >
.
1
1

, (4.11)
.
0 6 R 6 R.

4.2. , 0 6 Lp6 L
R),
g(x) = (1 + 2/x)1 .
= g(L/

p
ln
1
R
+ R.
L+
R 6 L + 2L
1
1

(4.12)

. 2
: ln 6 1
2 (0, 1]. -

219

 :
L

ln
1
1+
1
+
R6L
+
R=
1
1
2
1


1
1
1
+
R=
= L 1+
2

1
s

1
2R
1
=L+ L
+
R6
1

2
1 q
L

2R
1+

r
6L+

sL

1
L
LR + R + R
6

2
2R
p
R
+ R.
6 L + 2L

, 0 6 lti 6 1 i t, : LiT 6 T
= T.
i T . (4.12) L

R = ln N . 4.2

LT 6 min LiT + 2T ln N + ln N,
i

LT Hedge() T .
,
T . . 4.4.
, ()
.

4.3.

Follow the Perturbed Leader FPL.

220

 . [19],
[23] - [28]. [20], .

. , ,
.
, . , .
,

.
.
t i = 1, . . . N sit . , t :
0 6 sit 6 1 i t.
t si1:t1 = si1 + + sit1
< t, i = 1, . . . N .
, i. ,
i: st = sit .
s1:t = s1:t1 + st = s1:t1 + sit .
, ,
, .
t = 0, 1, . . . , 6
1
s0,1,2,3,4,5,6 = ( 21 , 0, 1, 0, 1, 0, 1) and s20.1,2,3,4,5,6 = (0, 1, 0, 1, 0, 1, 0). .

221

 .
, ,
.
FPL i,
1
si1:t1 i ,

 , i , i = 1, . . . N , t = 1, 2, . . . ,
p(x) = exp{x}, x > 0.
,
.
: P { > a} = ea P { > a + b} = eb P { > a}
a b. 4.10.
t N sit [0, 1],
i = 1, . . . N ; i
si1:t = si1:t1 + sit .

t = a/ t t, a
. , si0 = v0 = 0 i 0 = .
FPL . 4.3.
T
P
s1:T =
sIt t FPL
t=1

6 T .

FPL.
4.5. FPL t =
222

FOR t = 1, . . . T
, < t:


1 i
i
It = argmini=1,2,...N s1:t1 .
t
i sit for i = 1, . . . , N .
st = sIt t .
ENDFOR
. 4.3: FPL
q

2 ln N
t

E(s1:T ) 6 min si1:T + 2 2T ln N

(4.13)
i

. FPL IFPL (Infeasible FPL) (see . 4.4).

IFPL si1:t , i = 1, . . . N , t.
FPL.
t FPL IFPL T
lt = E(sIt t ) rt = E(sJt t ),
l1:T =

T
X

lt r1:T =

t=1

T
X

rt ,

t=1

, sIt t FPL t sJt t

IFPL t, E .
, It = argmini {si1:t1 1t i } Jt = argmini {si1:t
1 i
}.
0
t

223

FOR t = 1, . . . T
, 6 t:


1 i
i
Jt = argmini=1,2,...N s1:t .
t
i sit for i = 1, . . . , N .
sJt t .
ENDFOR
. 4.4: IFPL
4.3. FPL
IFPL :
l1:T 6 r1:T +

T
X

t

(4.14)

t=1

T .
. c1 , . . . cN . 1 6 j 6 N
mj m0j :
1
ci } 6
i6=j
t
1
6 min{si1:t1 + sit ci } =
i6=j
t
1
= min{si1:t ci } = m0j .
i6=j
t
mj = min{si1:t1

:
P {It = j| i = ci , i 6= j} P {Jt = j| i = ci i 6= j}

224

 :
P {It = j| i = ci , i 6= j} =
1
= P {sj1:t1 j 6 mj | i = ci , i 6= j} =
t
= P { j > t (sj1:t1 mj )| i = ci , i 6= j} 6
6 et P { j > t (sj1:t1 mj + 1)| i = ci , i 6= j} 6
6 et P { j > t (sj1:t1 + sit mj )| i = ci , i 6= j} 6
6 et P { j > t (sj1:t m0j )| i = ci , i 6= j} =
1
= et P {sj1:t j 6 m0j | i = ci , i 6= j} =
t
= et P {Jt = j| i = ci , i 6= j}.

(4.15)

3- 4-
P { > a + b} 6 eb P { > a} , , a b .
ci ,
:
P {It = j} 6 et P {Jt = j}.

(4.16)

t = 1, 2, . . . j = 1, . . . N .
(4.16) t = 1 , . . . , T :
lt = E(sIt t ) =

T
X

sjt P {It = j} 6 et

j=1

T
X

sjt P {Jt = j} = et rt .

j=1

lt rt 6 t lt rt > er lt >
(1 r)lt r 6 1. t = 1 , . . . , T
0 6 lt 6 1 t.
l1:T 6 r1:T +

T
X

t 6 r1:T + 2a T .

t=1

. 4

IFPL.
225

 4.4.
IFPL
r1:T 6 min si1:T +
i

ln N
T

(4.17)

T .
. st = (s1t , . . . sN
t )
s1:t = (s11:t , . . . sN
)

1:t
. = ( 1 , . . . N )
.
:


1
1

st = st

(4.18)
t t1
1

s1:t = s1:t
(4.19)
t
t = 1, 2, . . . .
s = (s1 , . . . , sN )
d = (0, . . . , 1, . . . , 0)
M (s) = argmindD {d s},
D = {(0, . . . , 1), . . . , (1, . . . , 0)} ,
N N .
M (s) , i-
1, si = min sj .
16j6N

i, M (s) .
(M (s) s) = min sj .
16j6N

IFPL
r1:T = E

T
X

!
M (
s1:t )st

t=1

, .
226

 ,
T
X

M (
s1:t )
st 6 M (
s1:T )
s1:T .

(4.20)

t=1

T . T = 1 . ,
T 1 T .

s1:T =
s1:T 1 +
sT ,
M (
s1:T )
s1:T 1 > M (
s1:T 1 )
s1:T 1 ,

s1:T 1 , , .

(4.20) T
T 1:
M (
s1:T )
s1:T = M (
s1:T )
s1:T 1 + M (
s1:T )
sT >
> M (
s1:T 1 )
s1:T 1 + M (
s1:T 1 )
sT >
>

T
X

M (
s1:t )
st .

t=1

(4.18)
st , (4.20) :
T
X
t=1

M (
s1:t ) st 6 M (
s1:T )
s1:T +

T
X
t=1


M (
s1:t )

1
1

t t1


.(4.21)

, (4.19)
s1:t
,



M (
s1:T )
s1:T 6 M (s1:T ) s1:T
=
T
M (s1:T )
= min{d s1:T }
.
(4.22)
dD
T
227

 (M (s1:T ) ) = k k.
E() = 1 ,
(4.22)


M (s1:T )
1
1
E
=
E( k ) =
.
(4.23)
T
T
T
(4.21)


T
X
1
1
6
(M (
s1:t ) )

t t1
t=1


T
X
1
1
1
i
6
max
max i .

=
16i6N
t t1
T 16i6N

(4.24)

t=1

t < t1 t.

:
0 6 E( max i ) 6 1 + ln N.
16i6N

(4.25)

,
i , i = 1, . . . N ,
P {max i > a} = P {i( i > a)} 6
i

N
X

P { i > a} = N exp{a}.

(4.26)

i=1

Z
P { > y}dy.

E() =
0

228

(4.27)


4.10. (4.26)
E(max i ln N ) =
i

Z
=

P {max i ln N > y}dy 6

i

Z
N exp{y ln N }dy = 1.

6
0

, E(maxi i ) 6 1 + ln N . (4.25) (4.24) 1T (1 + ln N ).

(4.21)(4.24) (4.23),
!
T
X
r1:T = E
M (
s1:t ) st 6
t=1

6 min si1:T +
i

ln N
.
T

(4.28)

. 4.
.
(4.14) 4.3 (4.17) 4.4

T
X

1
1
+ ln N T 6
i
t a
t=1

1
6 min si1:T + 2a T + ln N T .
i
a

E(s1:T ) 6 min si1:T + a

(4.29)

T .
(4.29) a, a = 2 ln N . , (4.13)

E(s1:T ) 6 min si1:T + 2 2T ln N .

i

. 4
229

 . ,
,
.
, FPL.
1 , . . . , N . FPL IFPL .


t , t = 1, 2, . . . , , 1t , . . . , N
.
FPL . 4.3 t
t1 , . . . , tN . , < t:


1 i
i
It = argmini=1,2,...N s1:t1 t .
t
IFPL.
st , t = 1, 2, . . . ,
FPL .
4.3 ,
4.4 ,
(4.21), (4.22) (4.24)
, E(ti ) = 1 i
t.
4.1. > 0 1
:
r

T
1
i
s1:T 6 min s1:T + 2 2T ln N +
ln .
(4.30)
i
2

FPL (
):
lim sup
T

1
(s1:T min si1:T ) 6 0
i=1,...N
T
230

(4.31)

 1.
.
(4.60) 4.5:
X1 , X2 , . . . , i = 1, 2, . . . 0 6 Xi 6
1.  > 0:
( T
)


T
X
X
22
P
Xi E
.
(4.32)
Xi >  6 exp
T
i=1
i=1
q


2
T
2
= exp 2T .  =
2 ln . Xt = st
(4.32) , 1 :
r
T
X
T
1
ln
st 6 E(s1:T ) +
2

t=1

(4.13) 4.5 (4.30).

(4.31)
(4.61) :

(
)
T
1 X





P 
(Xi E(Xi )) >  6 2 exp 2T 2 .
(4.33)
T

i=1

Xt = st .
, :
1
(s1:T E(s1:T )) = 0
T T
lim

1. (4.13) 4.5 (4.31) .

4.5 4.1 , sit t
It0 , t0 < t.
2
2

,
si1 , si2 , . . . i = 1, . . . , N .

231

 , c 4.1 , t
It0 t0 < t.
Xt = st = sIt t , Xt E(Xt |X1 , . . . , Xt1 ) - X1 , X2 , . . . ,

(4.64) (4.65). ,
(4.31) .

4.4.
, R () . , (),
( , ). , , Rn .
.
( ) (, ), .
, [0, 1].
: , , . :
FOR t = 1, 2, . . .
: t .
: t .
: t .
t
:
Lt () = Lt1 () + (t , t ).

232

 t :
Lt = Lt1 + (t , t ).
L0 () = L0 = 0 .
ENDFOR
() .
, .
1 , 2 , . . . , t
Lt
, .. inf Lt ().

: t t ,
t , t
.
:
R,T =

T
X
((t , t ) (t , t )) = LT LT ().

(4.34)

t=1

,
lim sup
T

1
(LT inf LT ()) 6 0.

(4.35)

, , , ,
.
n- Rn . ,
.
Z Rn
, z, z 0 Z 0 6
p 6 1 pz + (1 p)z 0 Z.
233

 h(z), Z, , {(x, y) : y > h(x)}

. , z, z 0 Z
0 6 p 6 1
h(pz + (1 p)z 0 ) 6 ph(z) + (1 p)h(z 0 ).

(4.36)

.
(, ).
: = {1, . . . , N }.
,
Rn , (, )
.

N
P

t =

wi,t1 ti

i=1
N
P

N
X

wi,t1
ti ,

(4.37)

i=1

wj,t1

j=1

ti Rn i- t, wi,t1 , i = 1, . . . , N ,
, t,

wi,t1
=

wi,t1
N
P

(4.38)

wj,t1

j=1

. , t
t.


i

wi,t1 = eLt1 ,

(4.39)

i = 1, . . . , N , Lit1 i-
1 t 1, > 0 .

234


N
P

t =

ti eLt1

i=1
N
P

=
e

Ljt1

N
X

wi,t1
ti ,

(4.40)

i=1

j=1

wi,t1

eLt1
= N
P Lj
t1
e

(4.41)

j=1

i, i = 1, . . . , N .
.
4.6. , (, )
[0, 1].
> 0, T 1 , . . . , T

LT
=

min

i=1,..., N

LiT 6

ln N
T
+
.

(4.42)

q
p
8 ln N/T : 12 T ln N .

.
Wt =

N
X
i=1

wi,t =

N
X

eLt ,

(4.43)

i=1

W0 = N .
, 4.7 4.5. : X
a 6 X 6 b. s R
ln E(esX ) 6 sE(X) +
235

s2 (b a)2
,
8

 E .

T
ln W
W0 .
. ,
wi,0 = 1 i = 1, . . . , N ,
!
N
X
WT
LiT
ln
= ln
e
ln N >
W0
i=1


LiT
> ln
max e
ln N =
i=1, ..., N

min

i=1, ..., N

LiT ln N.

(4.44)

T
ln W
W0 . t

N
P

ln

e(t ,t ) eLt1

Wt
= ln i=1 N
Wt1
P

=
e

Lit1

i=1
N
P

= ln i=1

wi,t1 e(t ,t )
i

N
P

= E(e(t ,t ) ),

(4.45)

wj,t1

j=1

wi,t1
=

wi,t1
N
P

wj,t1

j=1

i = 1, . . . , N .
(4.58), 4.7. a = 0, b = 1, X

236


(t , ti ) wi,t1
. (, ) .
:
N
P

wi,t1 (t , ti )
2
Wt
i=1
6
+
6
ln
N
Wt1
8
P
wj,t1
j=1

N
P

wi,t1 ti

6 t , i=1
N
P

wj,t1

2
=
+
8

j=1

= (t , t ) +

2
,
8

(4.46)

t
(4.40).
, (4.46) t = 1, . . . , T ,
T

ln

X
WT
Wt
2
=
ln
6 LT + T.
W0
Wt1
8

(4.47)

i=1

(4.44) (4.47),
LT 6

min

i=1,..., N

LiT +

ln N

+ T.

(4.48)

. 4
p
,

=
8 ln N/T :
q
1
2T

ln N . , T
, .
, ,
.
237

4.5.

, [13]. , 4.4,
T . T .
, LiT i- T b T .
, L
: = {1, . . . , N },
Rn ,
(, ) .


i

wi,t1 = et Lt1 ,
i = 1, . . . , N , Lit1 i-
1 t 1, t > 0 .

.
4.7.
1 > 2 > . . ., n > 1
1 , . . . , n , ()
t
T

X
b T min Li 6 ln N + 1
L
t .
T
i=1,...,N
T
8
t=1

, t =

4 ln N
t ,

t = 1, . . . , n,

b T min Li 6
L
T
i=1,...,N

238

T ln N .

(4.49)

.
PNt
i
i
pbt = i=1 t wi,t1 /Wt1 , wi,t1 = et Lt1
PN
Wt1 = j=1 wj,t1 .
(t , pbt ) 6

N
X
wi,t1
i=1

Wt1

(t , ti ) .

wi,t1
i=1 Wt1

PN

(t ,ti )

>

N
X
wi,t1
i=1

Wt1

et (t ,t )t /8

et (t ,bpt ) >

N
X
wi,t1
i=1

Wt1

et (t ,t )t /8 .

(4.50)

si,t1 = et1 Lt1 +t1 (Lt1 8

b

Pt1

k=1

k )

,
t

1
(si,t1 ) t1
wi,t1
=PN
.
t
N
Wt1
1
(s
) t1
j=1 N

(4.51)

j,t1

P
1
, N
j=1 N sj,t 6 1 t.
t = 0 , si,0 = 1
P
1
i. , N
j=1 N sj,t1 6 1.

t
t1
N
N
t
X
X
1
1
t1

(sj,t1 )
6
sj,t1
6 1,
N
N
j=1

(4.52)

j=1

x 7 x x > 0
[0, 1] 0 6 t 6 t1 . (4.52) (4.51),
239

wi,t1 /Wt1 > (si,t1 ) t1 /N . (4.50),

e

t (b
pt ,yt )

N
t
X
1
i
2
>
(si,t1 ) t1 et (t ,t )t /8 .
N
i=1

,
t

si,t = (si,t1 ) t1 et (t ,t )+t (t ,bpt )t /8 .

P
1
N
j=1 N sj,t 6 1.
i
N

X 1
1
si,n 6
sj,n 6 1,
N
N
j=1

n Lin + n

X
bn 1
k
L
8

!
6 ln N,

k=1

(4.49).

4.6.
.
N . (, ).
,
.
.
L0 = 0, Li0 = 0, i = 1, . . . , N .
FOR t = 1, 2, . . .
i : ti , i = 1, . . . , N .
: t .
: t .
i t : Lt =
Lit1 + (t , ti ), i = 1, . . . , N .
240

 t :
Lt = Lt1 + (t , t ).
ENDFOR
, .
t = 1, . . . , T
LT =

T
X

(t , t ).

t=1

i t = 1, . . . , T
LiT

T
X

(t , ti ).

t=1

. , , (, )
, .
1 2. : = = [1, 2].
: (, ) = 1{6=} {(, ) : 6= }.
, .
, 1 , 2 , . . .
1 , 2 , . . . , , .. LT = T
T . ,
t = 1, 2, . . . :

2, t = 1,
t =
1 .
, 1,
t1 = 1, 2,
t2 = 2, t = 1, 2, . . . . Lit i- , i = 1, 2.
241

, t = 1
1, t = 2
2.
, ,
1 , 2 , . . . , T
T /2, T /2.
mini=1,2 Lit 6 T /2 T .
,
1 , 2 , . . . ,
LT min LiT > T /2
i=1,2

T .
, , > T /2 T .

.
. ,
4.2.
t
pt = {p1,t , . . . , pN,t } {1, . . . , N }.
. 4.5.
, {1, . . . , N } . , , .
.

242

 L0 = 0, Li0 = 0, i = 1, . . . , N .
FOR t = 1, 2, . . .
i : ti , i = 1, . . . , N .

:
p1,t , . . . , pN,t {1, . . . , N }.
: t .
: it {1, . . . , N }
pi,t .
i t :
Lit = Lit1 + (t , ti ),
i = 1, . . . , N .
t :
Lt = Lt1 + (t , tit ).
ENDFOR
. 4.5:

243

 .
It , It = i
,

i1
i
X
X
Ut
pj,t ,
pj,t ,
j=1

j=1

U1 , U2 , . . . .
, P {It = i} = pi,t t.
(t , tIt ) . :
LT min LiT =
i=1,...,N

T
X

T
X

(t , tIt ) min

i=1,...,N

t=1

(t , ti ).

(4.53)

t=1

, (4.53):
E(LT min LiT ) =
i=1,...,N

T
X

E((t , tIt ))

t=1
T X
N
X

min

T
X

i=1,...,N

t=1
T
X

(t , ti )pi,t min

i=1,...,N

t=1 i=1

(t , ti ) =
(t , ti ).

(4.54)

t=1

(4.4) 4.2. t

t1
P

pi,t =

s=1
N
P

lsi

t1
P

s=1

j=1

244

,
lsj

(4.55)

 lsi = (s , si ) i = 1, . . . , N , 0 < < 1.

(4.55)
.
4.2
4.8. LT
Hedge() T = g(T / ln N ), g(x) =
p
(1+ 2/x)1 .

E(LT ) 6 min LiT + 2T ln N + ln N.

(4.56)
i

, (4.31),
4.1 .
4.2. 0 < < 1.
1

T
X

(t , tIt )

t=1

min

i=1,..., N

T
X

(t , ti ) 6

t=1

r
2T ln N + ln N +

1
1
T ln .
2

.
(4.60) 4.5.
,

Xt = (t , tIt ) E((t , tIt ))
.

ST =

T
P

Xt -

t=1
3
4.5 , ti t
t It0 ,
t0 < t (
).

245


P {ST > c} 6 e

2c2
T

c . > 0
)
(
r
1
1
6 .
T ln
P ST >
2

(4.54) (4.56). 4

i = 1, . . . , N 1 , 2 , . . . , i- 1i , 2i , . . . ,
i = 1, . . . , N .
,
!
T
T
X
1 X
lim sup
(t , t ) min
(t , ti ) 6 0.
(4.57)
i=1,..., N
T T
t=1

t=1

1.
(4.31) 4.1.
4.3.
.
, , , 4.3. t = 1 t it = 1,
t = 2 - it = 2. : i1 , i2 , . . .

, ,
(4.57), 0.
4.2 ,
, , ( )
,
WMA.
246

4.7.
, .
.
4.5. X a 6 X 6 b.
s R
ln E(esX ) 6 sE(X) +

s2 (b a)2
.
8

(4.58)

.
ln E(esX ) = sE(X) + ln E(es(XE(X)) ),
, X
E(X) = 0, a 6 X 6 b,
E(esX ) 6 es

2 (ba)2 /8

esx 6

x a sb b x sa
e +
e
ba
ba

a 6 x 6 b.
a
. E(X) = 0, p = ba
,
x = X :
a sb
b sa
e +
e =
ba
ba
= (1 p + pes(ba) )eps(ba) = e(u) ,
E(esX ) 6

u = s(b a), (u) = pu + ln(1 p + peu ).

(u) u
0 (u) = p +

p
.
p + (1 p)eu

247

 (0) = 0 (0) = 0. ,
00 (u) =

p(1 p)eu
1
6 .
(p + (1 p)eu )2
4

, q = (1 p)eu .
pq
1
2
(p+q)
2 6 4 , (p q) > 0.
[0, u]
(u) = (0) + u0 (0) +

u2
s2 (b a)2
u2 00
() 6
=
,
2
8
8

u = s(b a). . 4
, .
4.4. X ,
P {a 6 X 6 b} = 1.

P {|X E(X)| > c} 6 2e

2c2
(ba)2

(4.59)

. . X , X > 0.
Z
Z
E(X) = XdP >
XdP > cP {X > c}
{X>c}

, P {X > c} 6 E(X)/c.
(4.58),
P {X E(X) > c} = P {es(XE(X)) > ecs } 6 ecs+

s2 (ba)2
8

s. s.
s = 4c/(b a)2 .
P {X E(X) > c} 6 e

2c2
(ba)2

P {X E(X) < c} 6 e
248

2c2
(ba)2

P {|X E(X)| > c} 6 2e

2c2
(ba)2

4
. 4
4.5. X1 , X2 , . . . , i = 1, 2, . . .
P {ai 6 X 6 bi } = 1.  > 0 :

( n
)
n
X
X

22
,
P

(4.60)
Xi E
Xi >  6 exp
n
P

i=1
i=1
(bi ai )2
i=1

( n
X
i=1

Xi E

n
X
i=1

)
Xi < 

6 exp

22
n
P

(bi ai )2

i=1

.
4.4. (4.58) 4

exp(x) = ex .

249

( n
X

)
(Xi E(Xi )) > 

i=1



n
P
E exp(s (Xi E(Xi )))
i=1

exp(s)
n
Q

E(exp(s(Xi E(Xi ))))

i=1

exp(s)

 2
n
2
Q
exp s (bi8ai )

i=1

6 exp
s +

exp(s)
n
P

s2

(bi ai )2

i=1

6 exp

22
n
P

(bi ai )2

i=1

X1 , X2 , . . . .

s.
. 4
.
4.6. X1 , X2 , . . . , i = 1, 2, . . .

250

 P {ai 6 X 6 bi } = 1.  > 0


( n
)
1 X


2n2 2


.
P 
(Xi E(Xi )) >  6 2 exp

P
n

2
i=1
(bi ai )
i=1

ai = 0, bi = 1 i,

( n
)

1 X




P 
(Xi E(Xi )) >  6 2 exp 2n2 .
n


(4.61)

i=1

V1 , V2 , . . .
- X1 , X2 , . . . , i > 1 Vi
X1 , . . . , Xi
E(Vi+1 |X1 , . . . , Xi ) = 0
1. .
4.6. V1 , V2 , . . . - X1 , X2 , . . . , , Vi [Ai , Ai + ci ]
Ai , X1 , . . . , Xi , k
P
ci . Sk =
Vi ,
i=1

s > 0
(s2 /8)

E(esSn ) 6 e

k
P
i=1

c2i

.
E(esSn ) = E(esSn1 E(esVn |X1 , . . . , Xn1 )) 6
6 E(esSn1 es
s2 c2n /8

=e

2 c2 /8
n

E(e

)=

sSn1

).

(4.62)

4.5.
251

 (4.62).
4
4.4.
4.7. V1 , V2 , . . . - X1 , X2 , . . . ,
, Vi [Ai , Ai + ci ]
Ai , X1 , . . . , Xi , n
P
ci . Sn =
Vi ,
i=1

n > 0

2c2
.
P {|Sn | > c} 6 2 exp
n
P

2
ci
i=1

.
P {X > c} 6 E(X)/c
(4.58). n :

n
P
s2
c2i

i=1

P {Sn > c} = P {esSn > ecs } 6 exp

cs +

8
s. s.
n
P
s = 4c/
c2i .
i=1

2c2
.
P {Sn > c} 6 exp
n
P

2
ci
i=1

2c2
.
P {Sn < c} 6 exp
n

P
2
ci
i=1

252

(4.63)

2c2
.
P {|Sn | > c} 6 2 exp
n

P
2
ci
i=1

4.8. 4.7, ci = 1
i,


1
2
P
|Sn | > c 6 2e2nc .
(4.64)
n
(4.64) ,
4.9. 4.7, B1 < ci < B2 i, B1 , B2
:


Sn
P lim
= 0 = 1.
(4.65)
n n

4.8.
, (Boosting).
.
() ,
.
AdaBoost ( ), [18].
,
, .
253


.
AdaBoost. , 4.2, .
.
, S = ((
x1 , y1 ), . . . , (
xl , yl )), x
i X yi Y .
n
, Y = {0, 1}, X R n . , i (
xi , yi )
P X Y .

, > 0 S 1 hS ,
. ,
1/, 1/ .
, ,
 6 21 , > 0
.
hS S. .
D(i) () . D(i) > 0 i

l
X
D(i) = 1.
i=1

: D(i) = 1/l i.
h S
D
X
 = D{i : h(
xi ) 6= yi } =
D(i).
i:h(
xi )6=yi

254

 , D(i) = 1/l :
 = |{i : h(
xi ) 6= yi }|/l.

D(i)
. ,
. ,
, (
xi , yi )
D(i). ,
, D(i).

. AdaBoost
( [18]). -,
,  6 12 ,
,
 ( D).
AdaBoost:
: S = ((
x1 , y1 ), . . . , (
xl , yl )), D {1, . . . , l}, WeakLearn.
: w1i = D(i) i = 1, . . . , l.
FOR t = 1, . . . , T
1) i = 1, . . . , l
pit =

wti
.
l
P
j
wt

j=1

2) WeakLearn, D(i) = pit

i ht .
3) ht :
t =

l
X

pit |ht (
xi ) yi |.

i=1

255

4) t = t /(1 t ).
5) i = 1, . . . , l :
1|ht (
xi )yi |

i
wt+1
= wti t

ENDFOR
: :

1, f (
x) > 12 ,
h(
x) =
0 ,
f WeakLearn
f (
x) =

T
X

qt ht (
x),

t=1

qt =

ln(1/t )
T
P

ln(1/t )

t=1

t = 1, . . . , T .


Hedge() (. 4.2), . , . , .
t lti = 1 |ht (
xi ) yi |,
, ht xi , (),
. ,
, . , AdaBoost , WeakLearn
.
256

 WeakLearn 1/2
.
AdaBoost .
4.9. , WeakLearn AdaBoost
t = 1, . . . , T 1 , . . . , T
( , p2 , . . . , pT ).
p1 = D,
X
 = D{h(
xi ) 6= yi } =
D(i)
h(
xi )6=yi

h, AdaBoost
T ,
 6 2T

T p
Y
t (1 t ).

(4.66)

t=1

. , 4.1 4.2
l
P
4.2,
wTi +1 . i=1

:
l
X

i
wt+1

i=1

l
X

l
X

1|ht (
xi )yi |

wti t

i=1

wti (1 (1 t )(1 |ht (

xi ) yi |)) =

i=1

l
X

!
wti

(1 (1 t )(1 t )).

(4.67)

i=1

(4.67) T ,
l
X
i=1

wTi +1 6

T
Y
(1 (1 t )(1 t )).
t=1

257

(4.68)

 t WeakLearn t :

t =

l
X

pit |ht (
x i ) yi | =

i=1

X
wti
xi ) yi |.
|ht (
l
P j
i=1
wt
j=1

4.7. h
x
i ,
!1/2
T
T
Y
Y
|h (
x )y |
t t i i >
t
.
(4.69)
t=1

t=1

. , h , yi = 0,
|h (
x )y |
h (
x)
t t i i = t t i t. h(
xi ) = 1 ,

T
X
t=1

1X
ln(1/t ).
ln(1/t )ht (
xi ) >
2

(4.70)

t=1

(4.70) (4.69).
yi = 1. ht (
xi ) 6 yi t.
|ht (
xi )yi |
(1ht (
xi ))
t
= t
t. 1 6
t6T
|ht (
xi )yi |

1+ht (
xi )

= t

(4.71)

h(xi ) = 0
!1/2
T
T
Y
Y
h (
x)
t t i <
t
.
(4.72)
t=1

t=1

(4.72)
!1/2
T
T
Y
Y
ht (
xi )
t
>
t
.
t=1

t=1

258

(4.73)

 (4.73) (4.71) (4.69) . . 4

, ,
wTi +1

= D(i)

T
Y

1|ht (
xi )yi |

(4.74)

t=1

4.7 (4.69) (4.74)

l
X
i=1

>

wTi +1 >

i:h(
xi )6=yi

wTi +1 >

D(i)

T
Y

!1/2
t

T
Y

!1/2

t=1

i:h(
xi )6=yi

=

(4.75)

t=1

 h D.
(4.68) (4.75),
T
Y
1 (1 t )(1 t )

.
6
t
t=1

(4.76)

(4.76) ,
t . t :


d
1 (1 t )(1 t )

=0
dt
t
: t = t /(1 t ). t
(4.76) (4.66). . 4

259

 4.10. h
!
T
X
 = D{i : h(
xi ) 6= yi } 6 exp 2
t2 ,
(4.77)
t=1

t = 12 t , t > 0 t = 1, . . . , T .
, t = t, (4.77)
 6 exp(2T 2 ).

(4.78)

. , (4.66) 4.9
t = 21 t
q
p
2 t (1 t ) = 1 4t2 .

T q
Y
6
1 4t2 =
t=1

= exp

T
X
1
t=1

!
ln(1 4t2 )

6 exp 2

T
X

6
!

t2

(4.79)

t=1

(4.77) .
(4.78)
.
(4.78) AdaBoost,
6  h :
T >

1
1
ln .
2
2

260

4.9.
1

www.finam.ru (
). Hedge()
, ,
.
, . Hedge() .
2
AdaBoost,
WeakLearn
SVM, 2.13.
: http : //www.cs.toronto.edu/roweis/data.html
USPS
MATLAB,
.

4.10.
1.
, , k
. .
2. ,
0T (01)T 1T . , : 1 t1 = 0
t = 1, . . . , 4T , 2 t1 = 1
t = 1, . . . , 4T , 3 t1 = 1/2
t = 1, . . . , 4T . (, ) = | |.
t = 1, . . . , 4T :
(i) ;
(ii) Hedge .
261

3. p(x) = ex : P { > a} = ea P { > a+b} =

eb P { > a} a b.
4. , p(t) :
Z
P { > y}dy.

E() =
0

Ry
(: p(y) = F 0 (y), F (y) = 0 p(t)dt =
1 P { > y}
R . , E() = 0 tp(t)dt).
5. 4.4 ,
t FPL IFPL t1 , . . . , tN , t = 1, 2, . . . .
6.
4.2. ,
. .
7. , 4.2 4.4.
(: =
[0, 1]N , l = (l1 , . . . , lN ),
N . p = (p1 , . . . , pN ),
(l, p) = (l p)
l [0, 1]N p .
i- i = (0, . . . , 1, . . . , 0)
N . t, 4.4,
:
N
X

t =
ti wi,t
= pt ,
i=1

262

 , . . . , w )
pt = (w1,t
N,t
t, (4.40) ).
8. , N , t = 1, 2, . . .
ti [0, 1], 1 6 i 6 N , : , ( ),

T

T O( T ln N ).
9. ,

.

263

4 ,
,

( ) O( T ln N ), T ,
N .
,
O(ln N ).
, O(ln N ).

4.

5.1.
, = {0, 1}
= [0, 1]. = {1, 1} = [1, 1]. , (, )
:
264

 (, ) ;
[0, 1] , (0, ) (1, )
;
[0, 1] , (0, )
(1, ) .
(, )

= {(x, y) : p ((0, p) = x, (1, p) = y)}

(5.1)

= {(x, y) : p ((0, p) 6 x, (1, p) 6 y)}.

(5.2)

[0, 1] , . (5.1)
(5.2).
[0, +)2 ,
,
.
> 0 E : [0, +)2 (0, 1]2

E (x, y) = (ex , ey )

(5.3)

x, y [0, +).
(5.1)
E ( ) = {(e(0,p) , e(1,p) ) : p },
(5.2)
E ( ) = {(x, y) : p (0 6 x 6 e(0,p) , 0 6 y 6 e(1,p) )}. (5.4)
265

 (, ) -, E ( ) .
, -
> 0.
: , , .
.
, , , : (, ) = ln {},
.
= {0, 1},
, 1 .
= [0, 1]

(, ) = ln( + (1 )(1 ))
, ,

(, ) =

ln , = 1,
ln(1 ), = 0.

1
(, ) = ln( + (1 )(1 )),

(5.5)

> 0 .

(, ) = c( )2 ,
c .
= {0, 1} = [0, 1].

= [1, 1] = [1, 1].
.
266

. 6.1.

(, ) = c| |,
c .
,
.
( ) = = {0, 1}.
( c = 1)

0, = ,
(, ) =
1 .
.
267

. 6.2.

.
, (5.1)
(5.5) :
{(x, y) : ex + ey = 1}.

(5.6)

(5.6)
, ..
{(x, y) : e(x) + e(y) = 1},

(5.7)

(, ).
, (x1 , y1 ) -,
(x2 , y2 ), x1 > x2 y1 > y2 .

268

. 6.3.
( )

A R2 - (5.6),
- , (5.7).
, ax + by = c,
a > 0 b > 0, , (5.3)
ex + ey = 1, . , , ax + by = c E
aex + bey = c,
.. ex + ey = 1


1 a 1 b
ln , ln
.
c c
269

. 6.4.

, - ax+by = c
ex +ey = 1
.
, E ( )
,
, ,

.

, .

270

 5.1. -
, (a, b),
, e(x) + e(y) = 1 ex + ey = 1,
(a, b), , - .

. ,
-,
.
,
1990 .. [38].
. , 1989 [27]. ,
.

5.2.

4.4 4.6 O( T ln N ),
T , N . ( 4.4 ).
,
(, ).

T . O(ln N ), N .
, -

271


LT 6 c() inf LT () + a() ln N

T , c() > 1
(0, ).
c() = 1
.
= {0, 1}
= {1, 2, . . . , N }. = R. (, ),
.

( ) . , .

.
L0 = 0, L0 (i) = 0, i = 1, . . . , N .
FOR t = 1, 2, . . .
i ti , i = 1, . . . , N .
t .
t .
i t :
Lt (i) = Lt1 (i) + (t , ti ).
t :
Lt = Lt1 + (t , t ).
ENDFOR
> 0 (learning rate),
= e .
P0 (i) . P0 (i) = 1/N i , N
.
272

 t = 1, 2, . . . i = 1, . . . , N
i

Pt (i) = (t ,t ) Pt1 (i).

(5.8)

, , , .
(5.8) :
Pt (i) =

Pt (i)
N
P

(5.9)

Pt (j)

j=1

1.
, :
gt () = log

N
X

(,t ) Pt1
(i).

(5.10)

j=1

, , (5.10), APA (Aggregating Pseudo Algorithm). APA T 1 , . . . , T :

LT (AP A) =

T
X

gt (t ).

(5.11)

t=1

APA .
5.1. T

LT (AP A) = log

N
X
i=1

273

LT (i) P0 (i).

. (5.8) ,
T
P

PT (i) = t=1

(t ,ti )

P0 (i) = LT (i) P0 (i).

log

N
X

LT (i) P0 (i) log

i=1

N
X

LT 1 (i) P0 (i) =

i=1
N
P

LT (i) P0 (i)

log Ni=1
P

LT 1 (i) P0 (i)

i=1
N
P

= log

LT 1 (i)+(T ,T ) P0 (i)

i=1
N
P

=
LT 1 (i) P

i=1
N
P

= log

0 (i)

(T ,T ) PT 1 (i)

i=1
N
P

=
PT 1 (i)

i=1

= log

N
X

(T ,T ) PT 1 (j) = gT (T ).

(5.12)

j=1

(5.10).
(5.12) T ,
T
N
P
P
LT (AP A) =
gt (t ) = log
LT (i) P0 (i). 4
t=1

i=1

gt () ,
.

. t = (gt ), , (, (gt )) 6 gt () .
274

. 6.5. . ,

M , N = ( (0, ) , (1, ) ) ,

, , (, t ) .
5.2. , .
. (, ) = e .
E ( ) = {(x, y) : p (0 6 x 6 (0,p) 0 6 y 6 (1, p))}

275

 (, ) , ,

(T , )

>

N
X

(T ,T ) PT 1 (j)

(5.13)

j=1

T {0, 1}. (5.13) ,




(0, ) , (1, )

N
N
X
X
j
j

(0,T ) PT 1 (j),
(1,T ) PT 1 (j) .
j=1

j=1

(gt ) = . (, (gt )) 6 gt () .
(, ) , , t
t = (gt ). (AA-, Aggregating Algorithm).

, (5.13).
(gt ) .
, (gt ) , 5.1 ,

LT (AA) =

T
X

(t , (gt )) 6

t=1

6 LT (AP A) = log

N
X

LT (i) P0 (i).

(5.14)

i=1

P0 (i) =
1/N . (5.14) , i ,
276

 T
!
N
1 X LT (i)
LT (AA) 6 log

6
N
i=1


ln N
1 LT (i)
= LT (i) +
6 log

.
N

(5.15)

(5.15) ,
AA , , ..,
, ( ), ,
, ,
.

5.3.
. , .
.
i = 1, . . . , N
.
- = {0, 1}, = [0, 1].
(, ), , > 0 , = e .
P0 (d) .
t = 1, 2, . . .

Pt (d) = (t ,t ) Pt1 (d).

(5.16)

, , , . (5.16) E
Z

Pt (E) =
(t ,t ) Pt1 (d).
E

277

 (5.16) :
Pt (d) =

Pt (d)
.
Pt ()

(5.17)

;
Pt () = 1.

Z

gt () = log
(,t ) Pt1
(d).
(5.18)

, ,
APA, APA T
LT (AP A) =

T
X

gt (t ).

(5.19)

t=1

(5.16) ,
T
P

PT (d) = t=1

(t ,t )

P0 (d) = LT () P0 (d),

PT (d) = R

LT ()
P (d).
LT () P (d) 0
0

(5.18)
Z
gT () = log

(,T )+LT 1 ()
R L ()
P0 (d).
T 1
P0 (d)

(5.20)

5.1.
5.2. T

Z
LT (AP A) = log
LT () P0 (d).
(5.21)

278

. 5.1. (5.8) ,
T
P

Pt (d) = t=1

(t ,t )

P0 (d) = LT () P0 (d).

(5.22)

:
Z
Z
LT ()
log

P0 (d) log
LT 1 () P0 (d) =

R L ()
T P0 (d)
= log R L ()
=
T 1
P0 (d)

R
= log

)
LT 1 ()+(T ,T
P0 (d)
R
L
()
T
1
P0 (d)

R
= log
Z

)
(T ,T
PT 1 (d)
R

PT 1 (d)

(T ,T ) PT 1 (d) = gT (T ).

= log

(5.23)

(5.18).
(5.23) T , . 4
,
(gt ) . ,
d , . , ,
, .
(, ) ,

(, ) 6 gt ()
, gt () (5.18). (gt ) = .
279

 5.2
LT (AA) =

T
X

Z
(t , (gt )) 6 log

LT () P0 (d).

(5.24)

t=1

5.4.
,
.
, , (mixability curve) c() :



Z
(,)
c() = inf c : P (, ) 6 c log

P (d)
.

c() .
,
: (, (g)) 6 c()g()

(5.25)

Z
g() = log
(,) P (d)

P .
.
(, )
.
g()

(g) arg min sup

(5.26)

(g),
(5.26), (5.25).
, ,
, (5.25).
.
280

 , , , (5.24)
Z
T
X
LT (AA) =
(t , (gt )) 6 c() log
LT () P0 (d). (5.27)

t=1

,
c().
(5.15)
!
N
1 X LT (i)
LT (AA) 6 c() log

6
N
i=1


ln N
1 LT (i)

= c()LT (k) + c()

6 c() log
N

T k = 1, . . . , N .

5.5.

, = P()
. ,
() = ({}) .
(, ) = ln ().
= 1, = e1 .
(,) = (),
.. , . .
i t
ti = ti () .
i t Qi , :
Qi (|1 , . . . , t1 ) = ti () .
281

(5.28)


i t- . (5.28)
, i- , 1 , . . . , t1 .
,
t i 1 , . . . , t

Qi (1 , . . . , t ) = 1i (1 )2i (2 ) . . . ti (t ).

(5.29)

(5.8). i- t :
i

Pt (i) = (t ,t ) Pt1 (i) =

= 1i (1 )2i (2 ) . . . ti (t )P0 (i) =
= Qi (1 , . . . , t )P0 (i).

(5.30)

(5.30)
Pt (i) =

Pt (i)
N
P

Pt (j)

j=1

Qi (1 , . . . , t )P0 (i)
N
P

(5.31)

Qj (1 , . . . , t )P0 (j)

j=1

Pt (i)
i 1 , . . . , t .
i
(,t ) = ti (), (5.10) ,
t ,
gt () = log

N
X

ti ()Pt1
(i).

(5.32)

i=1

(gt ) AA ,
, t .
AA,

t () = (gt ) =

N
X
i=1

282

ti ()Pt1
(i).

 t
, t ,
(t , t ) = ln t (t ) = log

N
X

ti (t )Pt1
(i) = gt (t ).

i=1

: t = 1, 2.
i t = 1
1i = 1i () .
AA
1 () =

N
X

1i ()P0 (i),

i=1

P0
.
1 , . :
i

P1 (i) = (1 ,1 ) P0 (i) = 1i (1 )P0 (i).

i 1 :
P1 (i) =

1i (1 )P0 (i)
.
N
P
j
1 (1 )P0 (j)

j=1

, P1 (i)
i 1 .
t = 2.

283

 i t = 2
2i () .
AA
N
X
2 () =
2i ()P1 (i)
i=1

P1
, , .
2 , .

i

P2 (i) = (2 ,2 ) P1 (i) = 2i (2 )P1 (i) = 1i (1 )2i (2 )P0 (i).

i 1 , 2 :
P2 (i) =

1i (2 )P1 (i)
.
N
P
j

2 (2 )P1 (j)

j=1

,
P2 (i) i 2 P1 (i),
.
,
AA
.
i- T
LT (i) =

T
X

(t , ti ) =

i=1

= ln(1i (1 ) . . . Ti (T )) =
= ln Qi (1 , . . . , T ).
284

(5.33)


(5.28), t.
, AA, T
LT (AA) =

T
X

= log

(t , (gt )) =

t=1
N
X

LT (i) P0 (i) =

i=1

= log

N
X

Qi (1 , . . . , T )P0 (i).

(5.34)

i=1

, T , AA, , 1 , . . . , T T .
(5.15)
LT (AA) = log

N
X

Qi (1 , . . . , T )P0 (i) 6

i=1

6 ln Qk (1 , . . . , T ) ln P0 (k)

(5.35)

T k = 1, . . . , N .

5.6.
,
,
= = {0, 1}. , .


0, = ,
(, ) =
1 .

285

 , .
N ; i t
ti {0, 1}.

g() = log

N
X

(,t ) Pt1
(i)

(5.36)

i=1

(g(0), g(1)) R2 .
(log (p + (1 p)), log (p + (1 p))),
(5.37)
P
0 < < 1 , p =
Pt1 (i) ti =1

P
, 1 t,
(i) ,
1p =
Pt1
ti =0

0 t.
(5.37) , (1, 0) (0, 1), p = 0 p = 1.
1/c() () y = x . p = 21 (5.37)


1
1+
= log
,
c()
2

c() =

ln 1
2
ln 1+

(5.38)

AA .
= (g) : (g) = 0, (g(0), g(1)), (5.37),
y = x, = (g) = 1, (g(0), g(1))
y = x.

286

 (5.25),
= 0 g(0) > (0, 0) = 0 g(1)
1
c()
1
1
(5.37). , g(1) > c()
= c()
(1, 0).
(, 0) 6 c()g() {0, 1}.
(, 1) 6 c()g()
{0, 1}.
, (g(0), g(1)) y = x,
, .. g(0) < g(1),
log (p + (1 p)) < log (p + (1 p)),
p < 12 .
= 0.
, .. (g(0), g(1))
( ) y = x,
log (p + (1 p)) > log (p + (1 p)),
p > 12 .
= 1.
, AA 1, , 1,
, 0; AA
0 . , AA .
(4.1).

!


ln 1
1+
LT (AA) 6
LT () ln
ln P0 ().
(5.39)
2
2
ln 1+

5.7.
, : = {1, 1},
287

 [1, 1].

(, ) = ( )2 .
= {1, 1}, .
= [1, 1].
5.3. -
, 6 12 .
. (g(1), g(1))
:
(eg(1) , eg(1) ).
[1, 1]
2

(x(), y()) = (e(1) , e(1) ).

-, , .. , (
). ,
2
: dd2 xy 6 0.

d2 y
d x0 ()y 00 () x00 ()y 0 ()
=
.
d2 x
dx
(x0 ())2

(5.40)

x() ,
,

d
dx < 0. -
d2 y
6 0,
d2 x

x0 ()y 00 () x00 ()y 0 () > 0.

288

 :
2

x0 () = 2(1 + )e(1+) ,
2

x00 () = 2(1 + 2(1 + )2 )e(1+) ,

2

y 0 () = 2(1 )e(1) ,
2

y 00 () = 2(1 + 2(1 )2 )e(1) .

(5.41)

- ,
[1, 1]
(1 + )(1 + 2(1 )2 )
(1 )(1 + 2(1 + )2 ) > 0,
1
(1 2 ) 6 ,
2
1
6 .
2

(5.42)

. 4
- (g)
= {1, 1} = {1, . . . , N }.
1
= 12 , = e 2 .

gt () = log

N
X

(,t ) Pt1
(i)

(5.43)

i=1

(e 2 gt (1) , e 2 gt (1) ) =
!
N
N
X
X
(1,ti )
(1,ti )

Pt1 (i),

Pt1 (i) ,
i=1

(5.44)

i=1



(1,) , (1,) ,
[1.1].
289

(5.45)

 , (5.44).
1
1
gt (1)
= e 2 gt (1) 2 gt (1) .
(5.46)
g
(1)
t

(1, ) , (1, ) (5.45)

,
(5.44) :

k=

(1,

(1,

> gt (1) ,
)

> gt (1) .

(5.47)

(5.47)
(1, ) 6 gt (1),
(1, ) 6 gt (1).

(5.48)

gt (1)
(1, )

gt (1)gt (1)
=

=
= (1, )(1, ) .
gt (1)
(1, )

(5.49)

(1, ) (1, ) = (1 )2 (1 )2 = gt (1) gt (1),

1
= (gt (1) gt (1)).
4

(5.50)

t
1
t =
4

log

N
X

(1,ti )

Pt1
(i) log

i=1

(1,ti )

i=1

N
X

N
P

12 (1ti )2

(i)
Pt1

e
1

i=1
= ln N
.
2 P 1 (1+i )2

t P
e 2
t1 (i)
i=1

290

Pt1
(i)

 = [1, 1] (. [41]).
.
() . -, (gt ) ,
(, (gt )) 6 gt ()
, gt (5.18).

5.8.
[14]. N , . t = 1, 2, . . . t
t 1 t:

t = (1,t , . . . , N,t ),

i,t =

Si,t
,
Si,t1

Si,t i t. i,t [0, ), , i,t .

N t [0, 1]N ,
t = (1,t , . . . , N,t )
1,t + + N,t = 1. i,t ,
t 1. ,
,
(

) =

N
X
i=1

291

i i

 t.
.
N - .

(
, ) = ln(

).

(5.51)

: . ,
.
AA
.

T (
)
T
Y
K(
) =
(

t ).
t=1

LT (
) =

T
X

ln(

t ).

t=1

, P0 (d
) .
(5.20) APA
Z
(, )+LT 1 ( )
R L ( )
gT () = log
P0 (d
),
(5.52)
P0 (d
)
T 1
= e , 0 < 6 1.
5.1. ( (5.51))
0 < 6 1.
Z

(gT ) = =
PT 1 (d
) =

Z
LT 1 ( )
P0 (d
).
(5.53)
=
R L ()
T 1
P0 (d
)

292

. ,


 Z
Z
) 6 log
(, ) P (d
).

, P (d

Z
 Z
f
P (d
) >
f (
)P (d
),

(5.54)

f (
) = (, ) = (

) . (5.54) f (
) 0 < 6 1. 4
(5.53) , (5.21) ( 5.2) T :
Z
LT ( ) P0 (d
).
LT (AP A) = log

= 1.
T
T
Y
LT (
) = ln (

t ),
t=1

:
R QT 1
(

t )P0 (d
)

T = R QTt=1
.
1

t )P0 (d
)
t=1 (

(1/2, . . . , 1/2):
P0 (d
) =

N
(N/2) Y 1/2
j
d
,
[(1/2)]N
j=1

293

Z
(a) =

xa1 ex dx.

, (N + 1) = N !.
[39], AA ( = 1).1
5.2.

LT (AA) 6 LT (
) +

N 1
ln T + c
2

(5.55)

T , c .
, AA,
KT (AA) = eLT (AA) ,
(5.55)
KT (AA) > T

N 1
2

KT (
),

KT (
) ,
.
, , ,
T .
, (5.56) T ,

T .
1
[14].

294

5.9.
5.9.1.

.
, , AA-
.
. (xt , yt ), xt Rn yt R t = 1, 2, . . .
t > 1 yt
(x1 , y1 ), . . . , (xt1 , yt1 ) xt . f (x) = ( x), , x Rn .
x = xt t.
.

t yt ,
.
Rn .
t :
t = ( xt ).
.
:
, .
FOR t = 1, 2, . . .
xt Rn .
t = ( xt ), Rn .
t R.
yt [Y, Y ].
ENDFOR

. t (yt ( xt ))2 .
295

 (yt t )2 .
AA
= 1/2Y . 2

2

P0 (d) = (a/)n/2 eakk d,

(5.56)

a ( , ), .
p
= (1 , . . . , n ), kk = 12 + + n2 . ,
( x) x0 , x0 -,
-.
Rn t :
(yt , x0t ) = (yt x0t )2 = 0 (xt x0t ) 2(yt x0t ) + yt2 .

(5.57)

, x0t = (x1,t , . . . , xn,t ), 0 = (1 , . . . , n ), xt ,

, . x0t x0t = 0 (xt x0t ),
- :

! n
n
X
X
x0t x0t =
xt,i i
xt,j j =
i=1
n
X

j=1

i xt,i xt,j j = 0 (xt x0t ).

i,j=1

Rn T
LT () =

T
X

(yt x0t )2 =

t=1

= 0

T
X
t=1

!
xt x0t

T
X
t=1

!
yt x0t

T
X

yt2 .

(5.58)

t=1

2
, (5.50).

296

 (5.8) (5.22)
Pt1 (d) = Lt1 () P0 (d).
t
Z
0

gt (y) = log (y,xt ) Pt1

(d) =
Z
Pt1 (d)
0
=
= log (y,xt )
Pt1 ()
Z
1
0
= log (y,xt )+Lt1 ()
P0 (d).
(5.59)
Pt1 ()
, (5.56) (5.57) ,
Z
1
0
P0 (d) = (5.60)
gT (Y ) = log (Y,xT )+LT 1 ()
PT 1 ()
0 (aI+

Z
=

T
P

t=1

TP
1

xt x0t )+2(

t=1

TP
1

yt x0t Y x0T )(

Rn

t=1

yt2 +Y 2 )

d
.
PT 1 ()

gT (Y ).
[Y, Y ] , (5.50) (. [41]).
, (5.60) gT (Y )

297

 gT (Y ),
1
1
(gT (Y ) gT (Y )) =

4Y
4Y

T =

R
log
R

0 (aI+
Rn e

t=1

0 (aI+
Rn

T
P

T
P
t=1

TP
1

xt x0t )+2(

t=1

TP
1

xt x0T )+2(

t=1
T
P

0 (aI+

TP
1

yt x0t Y x0T )(

t=1

TP
1

yt x0t +Y x0t )(
TP
1

xt x0t )+2(

t=1

yt2 +Y 2 )
yt2 +Y 2 )

yt x0t Y x0T )

t=1
t=1
d
1
n e
log R
=
=
TP
1
T
P
4Y
0 (aI+
xt x0t )+2(
yt x0t +Y x0T )
R
t=1
t=1
d
Rn e

F
1
log e
4Y

1
F
=
4Y
=

T
1
X

aI +

aI+

T
X

T
P

t=1

xt x0t ,2

xt x0t , 2

t=1

!
yt x0t

aI +

t=1

TP
1
t=1

T
1
X

!
yt x0t ,2Y

x0T

=
!

yt x0t , 2Y x0T

t=1
T
X

!1
xt x0t

xT .

(5.61)

t=1

PT 11 ()
2- . 2- 3- TP
1

yt2 +Y 2 )

e t=1

.
3- 4-
5.4, , 3-

inf n (0 A+c0 +x0 )
n/2 R

e
,
detA
3-
inf n (0 A+c0 x0 )
n/2 R

e
,
detA

298


A = aI +

c = 2

T
X
t=1
T
1
X

xt x0t ,
yt x0t ,

t=1

x = 2Y x0T .
4-
F (A, c, x) = infn (0 A + c0 + x0 )
R

infn (0 A + c0 x0 ),
R

(5.62)

5- 6- 5.5,
F (A, c, x) = c0 A1 x.
5.4 5.5.
5.4. Q() = 0 A + c0 + d, , c Rn , d R
A
(n n).
Z
n/2
eQ() d = eQ0
,
(5.63)
detA
n
R

Q0 = minRn Q().
.
Q() = 0 A + c0 + d

= 0 . = 0 Q()
= Q( + 0 ).
0 A.
, Q
= 0, 0 =
Q
(0, . . . , 0), . , 0
0 Q
.
299


Q()
Q0 , ,
Q0 . , Q()
= 0 A + Q0 .
,
Z

0
e A d = n/2 / detA.
Rn

3 ( 2.7) [1]. 4
5.5 , F (A, c, x) = c0 A1 x.
5.5. A (n n), b, x Rn .
F (A, c, x) = minn (0 A + c0 + x0 )
R

minn ( A + c0 x0 ) = c0 A1 x.
R

(5.64)

. 0 A+c0 +x0
i . = (1 , . . . , n ). 2A + c0 + x0 =
0. ,
1 = 12 A1 (c + x). , 1 = 21 A1 (c x).

. 4
, (5.61) T , T
A = aI +

b=

T
X
t=1
T
1
X

xt x0t ,
yt x0t ,

t=1

T = b0 A1 xT =
!
!1
T
1
T
X
X
0
0
yt x t
aI +
xt xt
xT .
t=1

t=1

, AAR :
300


A = aI; b0 = 0.
FOR t = 1, 2, . . .
xt Rn .
A = A + xt x0t .
t = b0 A1 xt .
yt [Y, Y ].
b0 = b0 + yt x0t .
ENDFOR
AAR .
5.3. T
T

1X
LT (AAR) 6 inf (LT () + akk ) + Y ln det I +
xt x0t

a
2

t=1

6 inf (LT () + akk2 ) + Y 2

n
X
t=1

1X 2
ln 1 +
xt,i
a

!
.

t=1

kxt k 6 X t,
LT (AAR) 6 inf (LT () + akk2 ) + nY 2 ln


T X2
+1 .
a

1
. = 2Y
. 5.2 APA (5.21).

Z
eLT () P0 (d) =
LT (AP A) = log
Rn

Z
= log

n/2

(a/)

0 (aI+

T
P
t=1

xt x0t )+2(

T
P
t=1

yt x0t )

T
P
t=1

yt2

d. (5.65)

Rn

(5.65) eF () ,
!
!
T
T
T
X
X
X
F () = 0 aI +
xt x0t 2
yt x0t +
yt2 .
t=1

t=1

301

t=1

 F () = 0 .
5.4 (5.65),
APA,
LT (AP A) =

n/2 eF (0 )
n/2

= log ((a/) ) s

 =
T

P
det aI +
xt x0t )
t=1

1X
1
xt x0t
= F (0 ) log det I +
2
a
1
1
ln det I +
= F (0 ) +
2
a

t=1
T
X

=
!

xt x0t

t=1
T
X

1
= F (0 ) + Y 2 ln det I +
2

=
!

xt x0t

t=1

(5.58)
!
!
T
T
T
X
X
X
F (0 ) = 00 aI +
xt x0t 2
yt x0t 0 +
yt2 =
t=1

t=1

t=1

= ak0 k2 +

T
X

(yt x0t 0 )2 =

t=1

= ak0 k2 + LT (0 ).
LT (AAR) 6 LT (AP A),
. 4

5.9.2.
,
302

 . , ,

.
:
A = aI +

b=
0

T
1
X

!
yt x0t

t=1

T
X
t=1
T
1
X

xt x0t ,
yt x0t ,

t=1
1

T = b A xT =
!1
T
X
aI +
xt x0t
xT .

(5.66)

t=1

K(x, x0 ) , x, x0 Rn ,
S = ((
x1 , y1 ), (
x2 , y2 ), . . . )
:
KT = (K(xi , xj ))Ti,j=1 ;
kT = (k(xi , xT ))Ti=1 KT ;
YT - ;
(YT 1 , 0) = (y1 , . . . , yT 1 , 0) - , .
(5.66)
.
T n
0

x1
x11 , x12 , . . . , x1T
x02 x21 , x22 , . . . , x2T

,
XT =
... =

...
0
xT
xn1 , xn2 , . . . , xnT
- x01 , . . . , x0T . ,
T
X

xt x0t = XT0 XT ,

t=1

303


T
1
X

yt xt = (YT 1 , 0)0 XT .

t=1

5.6. nm B mn
C , aIn + CB aIm + BC ,

B(aIn + CB)1 = (aIm + BC)1 B,

(5.67)

a In n. 3
. (5.67)
(aIn + BC)B = B(aIm + CB),
. 4

T = b0 A1 xT =
!
!1
T
1
T
X
X
0
0
yt xt
aI +
xt xt
xT =
t=1

t=1

1
= (YT 1 , 0) XT aI + XT0 XT
xT =


0
0 1
(YT 1 , 0) aI + XT XT
XT xT =

1
T
= (YT 1 , 0)0 aI + K
kT ,
0

T = XT X 0 kT = XT xT . ,
K
T
XT XT0 = (xt xt0 )Tt,t0 =1 ,
kT = (xt xT )T ,
t=1

n .

304

(5.68)

..
x1 , . . . , xT .

T = XT X 0 kT = XT xT
K
T
KT = (K(xi , xj ))Ti,j=1 kT = (k(xi , xT ))Ti=1 .

T = (YT 1 , 0)0 (aI + KT )1 kT .

5.4. |yt | 6 X kxt k 6 X t,


T X2
2
2
LT (AAR) 6 inf (LT () + akk ) + nY ln
+1

a
T .

5.9.3.
A
x1 , x2 , . . . xT . K(x, y), x, y Rn .
(xi xj ) K(xi , xj ).
,

T +1 = w0 x = ((aI + XT0 XT )1 XT0 YT )0 xT +1

5.6
T +1 = w0 x = ((aI + XT0 XT )1 XT0 YT )0 xT +1 =
= YT0 XT (aI + XT0 XT )1 xT +1 =
= YT0 (aI + XT XT0 )1 XT xT +1 =
= YT0 (aI + XT XT0 )1 kT ,
kT = (xT +1 xt )Tt=1 .
305


T +1 = YT0 (aI + KT )1 kT ,
KT = (K(xi , xj ))Ti=1 kT = K(xT +1 , xt )Tt=1 .
, (5.68) :
T +1 = (YT , 0)0 (aI + KT +1 )1 kT +1 ,
+1
+1
KT +1 = (K(xi , xj ))Ti=1
kT +1 = K(xT +1 , xt )Tt=1
.

5.10.
1. ,
, ,
> 0. ,
( ).
2. , > 0
f () = e(,) , (, ) .
3. ,
(:
(, ) = ( )2 ).
4. , (, ) = | |
. c() ( a())
(5.27) {0, 1} [0, 1].
(:
c() =

1
2

ln

ln 1

 = c() =
2
1+

= e .

306

ln

2

2
1+e

,


c() :
e(,) >

N
X

!c()
(,ti )

Pt1
(i)

i=1
.
, , ti Pt1
:


e , e(1)

0 6 6 1. c() ,
,
i ).
5. (5.37).
6. c(), (5.38).
7. (5.39). (5.39) .

O( T ln N ) ,

4.2.

5.11.

: http : //www.csie.ntu.edu.tw; UCI Machine Learning Repository http : //archive.ics.uci.edu.

.
1
, ,

SVM, 2.8 2.8.2, 2.9.1.
.
307

 2
.
.
3

5.9. .

308

 III

309

.
. ,
.
. .
7 8.

6.1.
X Y .
. x X; y Y .
.
f (x, y) ,
. f (x, y) X Y .
f (x, y) < 0 ,
.. .
,
.
310

 x, inf yY f (x, y)
.
.
:
v = sup inf f (x, y)
xX yY

.
x0 ,
inf f (x0 , y) = v.

yY

, y
: supxX f (x, y) .

v = inf sup f (x, y)
yY xX

.
y 0 ,
sup f (x, y 0 ) = v.
xX

6.1. v 6 v, ..
sup inf f (x, y) 6 inf sup f (x, y).

xX yY

yY xX

. x X y Y
inf f (x, y) 6 f (x, y) 6 sup f (x, y).

yY

xX

inf f (x, y) 6 sup f (x, y).

yY

xX

x, .

sup inf f (x, y) 6 sup f (x, y)

xX yY

xX

311

 y, ,
v = sup inf f (x, y) 6 inf sup f (x, y) = v.
xX yY

yY xX

. 4
(x0 , y 0 ) X Y
f ,
f (x, y 0 ) 6 f (x0 , y 0 ) 6 f (x0 , y)

(6.1)

x X y Y .
(6.1)
max f (x, y 0 ) = f (x0 , y 0 ) = min f (x0 , y).
xX

yY

(6.2)

, min inf max sup,

,
.
, ,
f (x, y) (x0 , y 0 ). v = f (x0 , y 0 )
, , x0 , y 0
, (x0 , y 0 , v) . .
6.1. 1) f (x, y)
, ,
max inf f (x, y) = min sup f (x, y).
xX yY

yY xX

(6.3)

2) (6.3). (x0 , y 0 ) , x0 , y 0
.
. 1) 2).
(x0 , y 0 ) f (x, y).
v 6 sup f (x, y 0 ) = f (x0 , y 0 ) = v = inf f (x0 , y) 6 v.
yY

xX

312

(6.4)

 v 6 v. 6.1 v = v.
(6.4) , x0 , y 0
.
. , (6.3) . x0 , y 0 .
, (x0 , y0 ) . ,
f (x0 , y 0 ) > inf f (x0 , y) = v = v = sup f (x, y 0 ) > f (x0 , y 0 ).
yY

xX

,
. , (x0 , y 0 ) . 4
,
0 1, ,


1 1
1 1
.
: v = maxi minj ai,j = 1,
(..
) : v = minj maxi ai,j = 1.
.

6.2.

,
.
, Z Rn Rn ,
z, z 0 Z 0 6 p 6 1 pz + (1 p)z 0 Z.
h(z), Z, , z, z 0 Z
0 6 p 6 1
h(pz + (1 p)z 0 ) 6 ph(z) + (1 p)h(z 0 ).
313

(6.5)

 h(z) ,
(6.5), 6 >.
6.2. X, Y Rn Rm
( n m ),
Y , f (x, y) X Y ,
,
f (x, ) ( y)
x X, f (, y) y Y .

sup inf f (x, y) = inf sup f (x, y).

xX yY

yY xX

. 6.1 ,
inf sup f (x, y) 6 sup inf f (x, y).

yY xX

xX yY

, f (x, y) [0, 1].

 > 0
n. Y ,
- Y , .. {y 1 , . . . , y N } ,
y Y - y i .
y1 , y2 , . . . , yn Y x1 , x2 , . . . , xn X . x0
X. t = 1, . . . , n :
N
P

yt =

y i e

i=1
N
P

Pt1

s=0

f (xs ,y i )

,
e

(6.6)

Pt1

j
s=0 f (xs ,y )

j=1

p
(8 ln N )/n xt ,
f (xt , yt ) > sup f (x, yt )
xX

1
.
n

f ,
4.6 (x, y) = f (x, y).
314

 (6.6)
, i = 1, . . . , N , xt , t = 1, . . . , n, yt
. (4.42)
r
n
n
X
X
1
i
f (xt , yt ) 6 min
f (xt , y ) +
n ln N .
i=1,...,N
2

yi

t=1

t=1

n :
n

t=1

t=1

1X
1X
f (xt , yt ) 6 min
f (xt , y i ) +
i=1,...,N n
n

ln N
.
2n

(6.7)

f , (6.7),
inf sup f (x, y) 6
!
n
1X
6 sup f x,
yt 6
n
xX
yY xX

t=1

n
1X
f (x, yt ) 6
6 sup
xX n
t=1

n
1X
6
sup f (x, yt ) 6
n
xX
t=1

n
1X
1
6
f (xt , yt ) + 6
n
n
t=1
r
n
X
1
1
ln N
i
6 min
f (xt , y ) +
+ 6
i=1, ..., N n
2n
n
t=1
! r
n
ln N
1X
1
6 min f
xt , y i +
+ 6
i=1, ..., N
n
2n
n
t=1
r
ln N
1
6 sup min f (x, y i ) +
+ .
2n
n
xX i=1, ..., N

(6.8)

1- 2- ;
2- 3- f (x, ); 3- 4- , ;
315

 4- 5- xt ;
5- 6- (6.7); 6- 7-
f (, y); 7- 8-
.
, , n
r
ln N
1
inf sup f (x, y) 6 sup min f (x, y i ) +
+ .
yY xX
2n
n
xX i=1, ..., N
n
inf sup f (x, y) 6 sup min f (x, y i ).

yY xX

xX i=1,...,N

 0
inf sup f (x, y) 6 sup inf f (x, y).

yY xX

xX yY

. 4
6.2
, 1-, 5- 8- (6.8) ,
n
P
f (xt , yt ) n1
t=1

 n.

6.3.
6.3.1.
X = {1, . . . , N } Y = {1, . . . , M }. . f (i, j) = ai,j
. , , ai,j , ,
.
. (X, Y, f (x, y)) (X , Y, f (
p, q)),
316

 X , Y
, f (
p, q) p q :
X = {
p = (p1 , . . . , pN ) :
Y = {
q = (q1 , . . . , qM ) :

N
X

pi = 1, pi > 0};

i=1
M
X

qi = 1,
i=1
N X
M
X

f (
p, q) =

qi > 0};

f (i, j)pi qj .

i=1 j=1

. .
6.3. :
max min f (
p, q) = min max f (
p, q).
pX qY

qY pX

. , f (
p, q)
. 6.2. X Y
, . f (
p, q)
, . 4
. : p X , q Y; . - ,
. , ,
maxpX minqY f (
p, q).
q Y, , , p X . ,
317

 ,
q Y , minqY maxpX f (
p, q).
D - 6.3:
max min f (
p, q) = min max f (
p, q).
pX qY

qY pX

6.3.2.
X = {1, . . . , N }, Y = {1, . . . , M }
f (i, j) = ai,j . , .
1i = (0, . . . , 1, . . . , 0) ,
X, i X ( N , i- 1, 0).
Y . ,
f (1i , 1j ) = f (i, j) = ai,j .
6.4. (
p , q )
( ) (X , Y, f ), ,
f (1i , q ) 6 f (
p , q ) 6 f (
p , 1j )

(6.9)

i X j Y .
. 6.1.
,
p = (p1 , . . . , pN )
N
P
p =
pi 1i ,
i=1

q =

M
P

qj 1j . -

j=1

(6.9).

318


f (
p, q ) =

N
X

pi f (1i , q ) 6

i=1
M
X

f (
p , q ) =

N
X

pi f (
p , q ) = f (
p , q ),

i=1
M
X

qj f (
p , q ) 6

j=1

qj f (
p , 1j ) = f (
p , q)

j=1

p q. :
f (
p, q ) 6 f (
p , q ) 6 f (
p , q)
p q. 4
6.5.
min f (
p, q) = min f (
p, 1j ),
q

max f (
p, q) = max f (1i , q).
p

. ,
min f (
p, q) 6 min f (
p, 1j ),
q

max f (
p, q) > max f (1i , q).
p

f (
p, q) =

N X
M
X

ai,j pi qj =

i=1 j=1

>

min
j

= min
j

M
X

N
X

j=1

i=1

N
X

!
ai,j pi

!
pi ai,j

i=1
N
X

M
X

qj >

qj =

j=1

pi ai,j = min f (
p, 1j ),
j

i=1

319

 q. ,
,
. ,
min f (
p, q) > min f (
p, 1j ).
q

. 4

6.1.
v = max min f (
p, 1j ) = min max f (1i , q),
p

v .
.
(2 2)


1 1
.
1 1
: p = (p, 1 p) q = (q, 1 q),

f (
p, q) =

2
X

ai,j pi qj =

i,j=1

= q(p + 1 p) + (1 q)(p (1 p)) =




1
1
= 4 p
q
.
2
2
:



1
1
f (
p, q) = 4 p
q
2
2

(6.10)

v(p) = min f (
p, 1j ) = min{1 2p, 2p 1}.
j

320

 6.1 p ,
v(p) p = 12 .
, q = 12 .
: v = f (
p , q ) = 0. (p , q )
(6.10).

6.3.3. (2 M )

(2 M ) ( (N 2))
. 6.1

v = max min (a1,j p + a2,j (1 p)).
p

16j6M

p = (p, 1 p) ,
j .
, p = p ,

v(p) = min (a1,j p + a2,j (1 p))

16j6M

p [0, 1].
v(p ) .
M
Lj (p) = a1,j p + a2,j (1 p),
j = 1, . . . , M .
p [0, 1] .
y = v(p) .
(
p ) ( v(p )).
.
:


7
3 3 1 1 0
.
1 1 1 0 5 3
321

 Lj (p) = a1,j p + a2,j (1 p) j =

1, . . . , 6 :
L1 (p) = 7p (1 p),
L2 (p) = 3p (1 p),
L3 (p) = 3p + (1 p),
L4 (p) = p,
L5 (p) = p + 5(1 p),
L6 (p) = 3(1 p).
. p 4 5, .. p = p + 5(1 p).
: p = 5/7, v(p ) = 5/7.
.
6.6. (
p , q ) , v .
pi > 0 f (1i , q ) = v ,
qj > 0 f (
p , 1j ) = v .
. . f (1i , q ) 6 v , i = 1, . . . , N .
, i0 , pi0 > 0 f (1i0 , q ) < v .
f (1i , q ) 6 v pi , i = 1, . . . , N , ,
,

v = f (
p , q ) =

N
X

f (1i , q )pi < v = f (

p , q ).

i=1

.
. 4
6.2. (
p , q )
, v .
322

 f (1i , q ) < v pi = 0,
f (
p , 1j ) > v qj = 0.
f (
p , 1j ) = pa1,j +(1p)a2,j > v , p
() , .

.
1-, 2-, 3-, 6- (..
)
f (
p , 1j ) = Lj (p ) > v
j = 1, 2, 3, 6.
6.2
q = (q1 , q2 , q3 , q4 , q5 , q6 )
q1 = 0, q2 = 0, q3 = 0, q6 = 0, q4 = q, q5 = 1 q.

i = 1, 2. q = (0, 0, 0, q4 , 1 q4 , 0) .
v = min max (ai,4 q + ai,5 (1 q)) = max (ai,4 q4 + ai,5 (1 q4 )).
q

16i62

16i62

j = 4, 5 : q4 (1q4 ) = 5/7 5(1q4 ) = 5/7.

q4 = 6/7, q5 = 1/7.



5 2

p =
,
,
7 7
6 1
q = (0, 0, 0, , , 0),
7 7
5
v = .
7

323

6.3.4. (N M )

A = (ai,j ), i = 1, . . . , N , j = 1, . . . , M .
, A
; v . 1
6.1
v = max min f (
p, 1j ) = min max f (1i , q),
p

(6.11)

v . p = (p1 , . . . , pN ) , , f (
p, 1j ) > v
1j . ,
N
X

ai,j pi > v j = 1, . . . , M,

i=1
N
X

pi = 1,

i=1

pi > 0 i = 1, . . . , N.
xi = pi /v, i = 1, . . . , M .

N
X

ai,j xi > 1 j = 1, . . . , M,

i=1
N
X

xi = 1/v,

i=1

xi > 0 i = 1, . . . , N.
1

,
.

324

 : x1 , . . . , xN ,
N
X

xi min

i=1

N
X

ai,j xi > 1 j = 1, . . . , M,

i=1

xi > 0 i = 1, . . . , N.
(6.11) q = (q1 , . . . , qN )
, , f (1i , q) 6 v 1i . ,
M
X

ai,j qj 6 v i = 1, . . . , N,

j=1
M
X

qj = 1,

j=1

qj > 0 j = 1, . . . , M.
: x0j = qj /v, j = 1, . . . , M .

M
X

ai,j x0j 6 1 i = 1, . . . , N,

j=1
M
X

x0j = 1/v,

j=1

x0j

> 0 j = 1, . . . , M.

: x01 , . . . , x0M ,
M
X

x0j max

j=1

325


M
X

ai,j x0j 6 1 i = 1, . . . , N,

j=1

x0j > 0 j = 1, . . . , M.
,
xi , i = 1, . . . , N .

6.3.5. K
K
. k {1, . . . , K}
Nk ( ).
i = (i1 , . . . , iK ) K , ij {1, . . . , Nj }, j = 1, . . . , K.
k- f k (i) = f k (i1 , . . . , iK )
( f k (i)).
k- pk = (pk1 , . . . , pkNk )
{1, . . . , Nk }. pkj
j {1, . . . , Nk }.
I k , i
{1, . . . , Nk } pki .
I = (I 1 , . . . , I K ) ,
. i = (i1 , . . . , iK ),
ij {1, . . . , Nj }, j = 1, . . . , K.
, I 1 , . . . , I K
. I = p1 pK ,
i = (i1 , . . . , iK ), :
(i) = (I = i) = p1i1 . . . pK
iK .

326

 k-
X
=
E (f k (I))
(i)f k (i) =
i

N1
X
i1 =1

NK
X

k
p1i1 . . . pK
iK f (i1 , . . . , iK ).

iK =1

. K
= (
p1 , . . . , pk , . . . , pK )
, k = 1, . . . , K
p0k
E (f k ) > E0 (f k ),

0 = (
p1 , . . . , p0k , . . . , pK )

k- pk p0k .
, ,
,
.
.
.
f 1 (i, j) = f (i, j) f 2 (i, j) =
f (i, j), f (i, j)
.
, (
p0 , q0 ) ,
p q
f (
p, q0 ) 6 f (
p0 , q0 ) 6 f (
p0 , q),
f (
p, q) ,
f (
p, q) .
.
327

 6.1. (
p , q )
,

q {
q : min f (
p, q) max},
p

p {
p : max f (
p, q) min}.
q

(
p , q ) f (
p , q ) = v, v
.
6.1
.
K .
6.7. .
.
.
, .
1. , 0 1,
,


1 1
1 1
,
:
p = ( 21 , 12 )
q = ( 21 , 12 ).
.

:

0
1

0
(-1,1)
(1,-1)
328

1
(1,-1)
(-1,1)

 2. . , . ,
,
. :

(2,1)
(0,0)

(0,0)
(1,2)

(,) (,).
3. . .
, ,
.
:

(3,3)
(4,1)

(1,4)
(2,2)

.
(,) ( ).2
. .
P
K
Y

{1, . . . , Nk }

k=1
2

( 3), (,).
, ( 4), (,), (,).
, ,
( 1). ,
. (,)
2 .

329

 i = (i1 , . . . , iK ), K ,
, k = 1, . . . , K
h : {1, . . . , Nk } {1, . . . , Nk }
EP (f k (i)) > EP (f k (h(ik ), ik )),

(6.12)

i = (i1 , . . . , iK ) P ,
ik = (i1 , . . . , ik1 , ik+1 , . . . , iK )
(h(ik ), ik ) = (i1 , . . . , ik1 , h(ik ), ik+1 , . . . , iK ).
ik
, P .
.
6.2. P
K
Y

{1, . . . , Nk }

k=1

i = (i1 , . . . , iK ) , k {1, . . . , K}
j, j 0 {1, . . . , Nk }
X
P (i)(f k (i) f k (j 0 , ik ) > 0,
(6.13)
i:ik =j

(j 0 , ik ) = (i1 , . . . , ik1 , j 0 , ik , . . . , iK ).
(6.13) :
E(f k (i)|ik = j) > E(f k (j 0 , ik )|ik = j),
E P .
330

. (6.12)
:
X
P (i)(f k (i) f k (h(ik ), ik )) > 0,
(6.14)
i

k {1, . . . , K} h
h : {1, . . . , Nk } {1, . . . , Nk }.
j, j 0 {1, . . . , Nk } h,
, h(j) = j 0 h(ik ) = ik ik 6= j. (6.14) ,
i, ik = j,
. , (6.14)
(6.13).
, . 4
(6.13) ,
Q

K
k=1 {1, . . . , Nk }.

QK
k=1 {1, . . . , Nk } j Ak 1 6 k 6 K.
Pi (|i
Qk = j)
K
s=1,s6=k {1, . . . , Ns }

ik ik = j.

f k (j 0 , Pk (|ik = j))) = EPk (|ik =j)) (f k (j 0 , ik )).
:
f k (j 0 , Pk ) = EPk (f k (j 0 , ik )),
, Pk ik , P ik = j.
(6.13) :
331


6.3. P
QK

k=1 {1, . . . , Nk } i =
(i1 , . . . , iK )
, k {1, . . . , K}
j, j 0 {1, . . . , Nk } 3
f k (j, Pk (|ik = j)) = max
f k (j 0 , Pk (|ik = j)).
0
j Ai

(6.15)

, .
, .

6.4.
1. , ()
:
min f (
p , q) = min f (
p , 1j ),
q

max f (
p, q ) = max f (1i , q ),
p

(6.16)

(
p , q ) ( ).
2. 6.1.
3. , 2 ( (6.3.5))
:
32 13 ;
13 32 .
?
4. , 3 ( (6.3.5))
. (,).
5. , .
3

.

332

-
, [31].
, ,
, . ,
, ..
[31] 7.1
.

,
3.
7.3.

7.1. -

. -

333

 [31] (, )
, ()
, ,
, , . 1
, . ,

.
. ,
,
.
, ,
, .

: ,
.
:
K0 = 1.
FOR n = 1, 2, . . .
pn [0, 1].
Mn R.
n [0, 1].
: Kn = Kn1 + Mn (n pn ).
ENDFOR
.
n Mn pn .
n ,
1

0 1 1.

334

 . , Mn < 0.
|Mn |
.
, Kn > 0 n
sup Kn = (
); .
: p1 , 1 , p2 , 2 , . . . .
. ,
. , : Mn

:
Mn = Mn (p1 , 1 , p2 , 2 , . . . , pn1 , n1 , pn ).
-
.
7.1. ,

. ,
, Kn > 0 n
:
lim sup Kn =
n

1X
(i pi ) = 0.
n n
lim

(7.1)

i=1

. , (7.1)
. ,  > 0
n
1X
(i pi ) > 2
(7.2)
n
i=1

335

 n  > 0
n

1X
(i pi ) < 2
n

(7.3)

i=1

n. ,  < 12 .
. |i pi | 6 1,


n
X

(i pi ) 

i=1

n
X

(i pi )2 > 2 n

i=1

n. tt2 6 ln(1+t),
t > 12 ,
n
X

ln(1 + (i pi )) > 2 n

i=1

n.
n
Mn = Kn1 ,
Kn1 . ,
n
n
Y
Kn =
(1 + (i pi )),

(7.4)

i=1

ln Kn =

n
X

ln(1 + (i pi )) > 2 n,

i=1

,
lim sup
n

ln Kn
> 2 ,
n
336

(7.5)

.. supn Kn = .
, (7.4), Kn > 0 n
i pi .
, (7.3) n,
Mn = Kn1 ,
Kn1 .
,
, (7.2)
(7.3) n,
 > 0 .
, ,
,
 > 0. k = 2k k = 1, 2, . . . .
K01,k = 1 K02,k = 1 k.


1,k
,
Mn1,k = k Kn1
2,k
Mn2,k = k Kn1
,

X
2k Mn1,k ,
Mn+ =
k=1

Mn =

2k Mn2,k ,

k=1

1
Mn = (Mn+ + Mn ).
2

337

 Mn Kn :
Kn+ =

2k Kn1,k ,

k=1

Kn =

2k Kn2,k ,

k=1

1
Kn = (Kn+ + Kn ).
2
, n Kn1,k 6 2n k. |Mn2,k | 6 2n1
n, n = 0 n = 1
.
,
1,k
Kn > 0 Kn2,k > 0 n k.
(7.1) , (7.2)
(7.3)  = k n. (7.5), Kn = Kns,k , ,

lim sup
n

ln Kns,k
>0
n

s = 0 1. ,
lim sup
n

ln Kn
> 0.
n

, , , , Kn > ecn n, c > 0.

. 4
- 7.1.
7.1. ,

338

 :
n

1X
(i pi ) = 0,
n n

sup Kn < lim

n

i=1

Kn n.
,
,
, .

7.2. -
- [31] .
.
. - . , (
).
, K.
( )
.
.
K().
. () :
w , m.
w1 , w2 , . . . . .
w W , , w, m
339

 S
.
M .
m = M(), , .
KM (), .
KM
W S. m
w
:
KM (w) = KM () + (w, m),
KM () = K0M .
, : 1 m + 2 m0 S m, m0 S
1 , 2 ,
:
(w, 1 m + 2 m0 ) = 1 (w, m) + 2 (w, m0 )
m, m0 1 , 2 . ,
M1 M2 1 2 M = 1 M1 +2 M2
:
KM () = 1 KM1 () + 2 KM2 ()
.

x = x() .
,
.
, x () x() () .
340

 x ,
,
x() ,
. , ( )
x().

x() . , , ..,
M ( ) x() :
KM () + > x() .
, y, KM > y
, KM () > y() .
, x
.
x ,
x ,
M , x()
:
Ex = inf{ : M(KM > x )}
,
x() .
, ( ) x().
x() .
,
x().
x ,
N , -

341

 KN > x:
Ex = sup{ : N (KN > x)}
,
x() .
x ,
x .
E(x) = sup{ : N (KN > + x)} =
= inf{ : M(KM > x )} = Ex
, M , KM () 6 K0M ,
KM . , ,

.
7.1. , Ex 6 Ex
Ea = Ea = a, a , a() = a
.
. Ex > Ex,
1 < 2 ,
Ex < 1 < 2 < Ex,
M1 M2 , KM1 > x 1 KM2 > 2 x.
M = M1 + M2
KM = KM1 + KM2 = 2 1 > 0,
.
. 4
Ex = Ex, Ex.
342

, , .
, : Ex = Ex
x:
V x = E(x Ex)2 V x = E(x Ex)2 , Ex
x.
S
- . E:

1, S,
1S () =
0 .
, . .
S
P (S) = E(1S ).
P (S) 6 1 S
P (S) = inf{ : M(KM () + > 1 S,
KM () + > 0 6 S)}.
S
P (S) = E(1S ).
,
0 6 P (S) 6 P (S) 6 1

P (S) = 1 P ( \ S).
-
.
343

 .
: .2
0 <  6 1 > 0 : K0 = .
FOR n = 1, 2, . . .
Mn R.
xn [1, 1].
: Kn = Kn1 + Mn xn .
ENDFOR
N
P
SN =
xn . ,
n=1
 
 
Kn > 0 n = 1, . . . , N Kn > 1  SNN  < .
, ,
Mn


1, Mn < 0,
xn =
1 .
Kn 6 K0 n.
7.2. N >
1
. ,
2






 SN 
 SN 
1




P 
.
> 6P 
> 6
N 
N 
N 2
, .
. ,
2
Sn2 = Sn1
+ 2xn Sn1 + x2n =
2
= Sn1
+ 2xn Sn1 + 1,

S0 = 0.
2

344

 : Mn =
n .

2Sn1
N

N
X
2Sn1 xn =
N
n=1
 2

SN
2
= (SN N ) =
1 .
N
N

KN K 0 =

S2

, KN = NN .

 r
 SN 
KN


 N  6 N .

(7.6)

, , KN > 1. KN < 1
N > 12 , (7.6)


 SN 


 N  < .
, .
SN : SN = SN ().
, , :

 

 SN () 


E= :
> ,
N 
.


 SN () 
M
 > ,
P (E) = inf{ : K0 = M(KN () > 1, 
N 


 SN () 
M

 6 )}.
KN () > 0, 
N 
, M
= N12 . ,






 SN 
 SN 
1




P 
> 6P 
> 6
.


N
N
N 2
. 4
345

7.3.
, 7.1 , , .
: , .
:
K0 = 1.
FOR n = 1, 2, . . .
Sn : [0, 1] R.
pn [0, 1].
n {0, 1}.
: Kn = Kn1 + Sn (pn )(n pn ).
ENDFOR
:
, Kn ; .
7.3. .
. ,

1, p < 0.5,
Sn (p) =
1 .
n

1, pn < 0.5,
n =
0 .
n > 0, n = 0, pn >
n pn 6 12 Sn (pn ) = 1; n = 1, pn <
n pn > 12 Sn (pn ) = 1. ,
Kn > Kn1 +
346

1
2

1
2
1
2

 n, . 4
.
,
. ,
,
.
: , , , , {0, 1}, P{0, 1}
{0, 1}. 3
.
K0 = 1 F0 = 1.
FOR n = 1, 2, . . .
Sn : [0, 1] R.
: Pn P[0, 1].
n {0, 1}.
fn : [0, 1] R,
Pn ,
R
, fn (p)Pn (dp) 6 0.
pn [0, 1].
: Kn = Kn1 + Sn (pn )(n pn ).
: Fn = Fn1 + fn (pn ).
ENDFOR

. , , , .
: Sn
, Kn > 0 n
.
3

Q P{0, 1} (q, 1 q), q = Q{1}

1.

347

 : Pn fn , Fn > 0 n . 4
:
, , . , .
, (i)
Fn (ii) Kn
; .
,
.
7.4.
.
. n
, .
pn [0, 1], n {0, 1}. ( )
:
F (n , pn ) = S(pn )(n pn ).
Qn P{0, 1}, pn = Q{1}. pn Pn (pn ) = 1, Pn (r) = 0
r [0, 1] \ {pn }.
4

Fn , fn (p) -
Fn Fn1 . R , F0 = 1
Fn > 0
R
n. fn (p)Pn (dp) 6 0 n Fn Pn (dp) 6 Fn1
n.
. ,
p1 , p2 , . . . , .

348

 Q pn
F (Qn , Pn ) = Q{0}F (0, pn ) + Q{1}F (1, pn ) =
= Q{0}S(pn )(pn ) + Q{1}S(pn )(1 pn ) =
= (1 Q{1})S(pn )(Q{1}) + Q{1}S(pn )(1 Q{1}) = 0.
, Q P F (Q, P ) 6 0
sup inf F (Q, P ) 6 0.
P

(7.7)

, .
, n {0, 1}.
,
, , . > 0
[0, 1] - N , , ,
 , ,
/2,
pn N .
- , |Sn (p)| 6 Kn1 6 2n1
p ( n). 5 (7.7)

sup inf F (Q, P ) 6 /2.
Q

,
inf sup F (Q, P ) = sup inf F (Q, P ) 6 /2.
P

P P[0, 1],
N , ,
sup F (Q, P ) 6 ,
Q
5
Sn (p) , Kn > 0 n
.

349

 ,
Z
Sn (p)(n p)P (dp) 6

(7.8)

n = 0 n = 1.
E P[0, 1] , P , (7.8) n = 0 n = 1 .
P[0, 1] . , P[0, 1]
. , E .
0 i , i = 1, 2, . . .
, E i 6= .
Pn E i P[0, 1],
,
Z
Sn (p)(n p)Pn (dp) 6 0
(7.9)
n = 0 n = 1 .
. n Pn , .
fn

fn (p) = Sn (p)(n p).

Fn = Kn n.
fn Pn 0 (7.9), ..
fn Pn .
Fn = Kn ,
: , . . 4
, , supn Fn < .
350

7.4.
, ,
Sn (p), ,
,
, .

, - . , 7.1.
 , 0 <  < 21 . K01 = 1.
1
Sn1 (p) = Kn1
,

..
n, ,
< n.
n
Kn1 =

n
Y
(1 + (i pi )),

(7.10)

i=1

1 , . . . , n , , p1 , . . . , pn , 1, . . . , n.
|i pi | 6 1 i, Kn1 > 0 n
, ..
.
7.4
. ,
,
.. supn Fn < ,
1 , . . . , n , ,
351

 p1 , . . . , pn
Kn1 , C > 0 :
n
Y

(1 + (i pi )) 6 C

i=1

n.
n
X

ln(1 + (i pi )) 6 ln C,

i=1

n
X

(i pi ) 

n
X

i=1

(i pi )2 6 ln C,

i=1

n
X

(i pi ) 6 ln C + 2 n,

i=1
n

ln C
1X
+
(i pi ) 6
n
n

(7.11)

i=1

n. ln(1 + t) > t t2
|t| 6 0.5.
,
n

lim sup
n

1X
(i pi ) 6 .
n

(7.12)

i=1

, K02 = 1
2
Sn2 (p) = Kn1
,

,
,
n

1X
lim inf
(i pi ) > .
n n

(7.13)

i=1

,
(7.12)
352

 (7.13). Sn1 (p) Sn2 (p)

Kn1 (p), Kn2 (p) .

1
Sn (p) = (Sn1 (p) + Sn2 (p)).
2
n
1
Kn = (Kn1 + Kn2 ).
2
, Kn1 > 0 Kn2 > 0 n. S1 (p) = 0,
S11 (p) = S12 (p), Sn1 (p) Sn2 (p) ,
Kn1 (p)
Kn2 (p).
, .. supn Fn < .
Kn
Kn1 Kn2 . , (7.12) (7.13).
, ,
(7.12) (7.13)  > 0.
k = 2k k.
K01,k = 1 K02,k = 1 k.
1,k
Sn1,k (p) = k Kn1
,
2,k
Sn2,k (p) = k Kn1
,

X
Sn+ (p) =
2k Sn1,k (p),
k=1

Sn (p) =

2k Sn2,k (p),

k=1

1
Sn (p) = (Sn+ (p) + Sn (p)).
2
353


Kn+ =

2k Kn1,k ,

k=1

Kn =

2k Kn2,k ,

k=1

1
Kn = (Kn+ + Kn ).
2
, n
Kn1,k 6 2n k (7.10). |Sn2,k (p)| 6 2n1 n.
, Kn1,k > 0 Kn2,k > 0 n k. Kn Kn1,k Kn2,k .
, (7.12) (7.13)
k , k = 1, 2, . . .
, , 7.4, , :
n

1X
lim
(i pi ) = 0.
n n

(7.14)

i=1

(7.14) . , 1 , 2 , . . . = 0, 1, 0, 1, 0, 1, 0, 1, . . . p1 , p2 , . . . = 12 , 21 , 12 , 12 , 12 , 21 , 12 , 12 , . . . , ,
( ) ,
.
.
354

 1 , 2 . . . , p1 , p2 , . . .
F (p1 , 1 , p2 , 2 , . . . , pn1 , n1 , pn ),

p1 , 1 , p2 , 2 , . . . , pn1 , n1 , pn ,
pn n, n = 1, 2, . . . , : 0 1.
p1 , p2 , . . .
1 , 2 , . . . F (p1 , 1 , p2 , 2 , . . . , pn1 , n1 , pn ),

sup
n

n
X

F (p1 , 1 , p2 , 2 , . . . , pi1 , i1 , pi ) <

i=1

n
P

lim

F (p1 , 1 , p2 , 2 , . . . , pi1 , i1 , pi )(i pi )

i=1
n
P

= 0. (7.15)
F (p1 , 1 , p2 , 2 , . . . , pi1 , i1 , pi )

i=1

7.5.
Fk , k = 1, 2, . . . , ( Pn
p1 , 1 , p2 , 2 , . . . , pn1 , n1 ), , n
p1 , 1 , p2 , 2 , . . . , pn1 , n1 , pn ,
p1 , p2 , . . . , ,
Fk , supn Fn < (..
).
355

.
.
Sn1,k (p) Sn1,k (p) k
k Fs , k, s = 1, 2, . . .
:
1,k,s
Sn1,k,s (p) = k Fs (p1 , 1 , p2 , 2 , . . . , pi1 , i1 , p)Kn1
,
2,k,s
Sn2,k,s (p) = k Fs (p1 , 1 , p2 , 2 , . . . , pi1 , i1 , p)Kn1
.

- (k, s). i
p(i) = q(i) = s. p(i) q(i)
.
Sn+ (p) =

2j Sn1,p(j),q(j) (p),

j=1

Sn (p) =

2j Sn2,p(j),q(j) (p),

j=1

1
Sn (p) = (Sn+ (p) + Sn (p)).
2

k .
,
(7.11) i,
F (p1 , 1 , p2 , 2 , . . . , pi1 , i1 , pi ) = 1.
(7.11) (7.14) ,
(7.15), n
n
X

F (p1 , 1 , p2 , 2 , . . . , pi1 , i1 , pi ).

i=1

356

7.5.
1. , 7.1 lim sup Kn =
n

lim Kn = (: n
Mn = Kn1

Mn , Kn1 6 2C ,
C
Mn =
0, ,
C . ,

n = P 2C M C . M
n
C=1
C
Kn .

, n ,

n , K
n = P 2C KC .
M
n
C=1

, lim sup Kn = ,
n

n = ).
lim K
n
2. ,
x,x1 , x2 :
a) Ex 6 sup x().

b) Ea = Ea = a, a .
c) E(x1 + x2 ) 6 Ex1 + Ex2 .
d) E(x + ) = Ex + , .
e) E(x) = Ex > 0.
f) x1 6 x2 , Ex1 6 Ex2 .
3.
Ex x.
4. , 0 6 P (S) 6 1 6 P (S) 6 1.
5. ,
E, E1 E2
:
a) 0 6 P (E) 6 P (E) 6 1.
b) P () = P () = 1.
c) P (E) = 1 P ( \ E).
d) P (E1 E2 ) 6 P (E1 ) + P (E2 ).
357

e) P (E1 E2 ) > P (E1 ) + P (E2 ) 1.

f) E1 E2 , P (E1 ) 6 P (E2 ).
6. 7.2 ( ) {1, 1}
N . ,
0 N 2N , ,
S 2N |S|
(: 0 . 0 K0 = 2N
: M1 = 2N Mt = Kt1 t > 2
. Mt = 0
t 1 0 . ,
0 . ,
0 . KN (0 ) = 1 KN () = 0 6= 0 .
= 2N
M1 = Mt = Kt1 t > 2
. Mt = 0 . K0 = 0.
P

i
s
s s1
i=0 2 = (2 1).
s,
2s
. , (2N 1).
).
, .
7. ,
, .
N = 1 , . . . , N N .
, M n = 1 , . . . , n
N
n < N n+1
= n+1 , . . . , N ,
358

N . , KM ( N ) 6 KM ( n ), N = n n+1
,
M
.
,
S

P (S) = inf{ : M(n(KM ( n ) + > 0)

KM () + > 1 S)}.
, , ,
.
8. 7.1 7.5.

359

.

. , , .
,
.
8.1 ,
, .
8.2 ,
. 8.3
.
, 8.4 ,
, , , ,
360

8.1.
, .
, t = 1, 2, . . . It {1, . . . , N } pt = (p1,t , . . . , pN,t ) ( ),
Jt {1, . . . , M } qt = (q1,t , . . . , qN,t ).
pt qt . t
f (
pt , qt ), f (
pt , qt ).

n
:
n
n
X
X
max
f (i, Jt )
f (It , Jt )
i=1,..., N

t=1

t=1

M
X
i=1

f (It , Jt )

min

j=1,..., N

n
X

f (It , j)

t=1

.

.
{1, . . . , N } . i
i {1, . . . , N }.
,
t It . Jt {1, . . . , M }
.

361

,
{1, . . . , M } . j
j {1, . . . , M }.
,
t Jt . It {1, . . . , N }
.
,
. 1 (Jt , It ) = f (It , Jt ),
Jt , It t.
2 (It , Jt ) = f (It , Jt ), It
, Jt t.
( ) ( ) ,
t
pt ( qt ).
( ) .
,
, (. (4.57)). , 4.2 4.6.

pt = (p1,t , . . . , pN,t ) :

 P
1 (J , i)

exp t1
s
s=1
,
 P
pi,t = P
t1 1
N
s=1 (Js , k)
k=1 exp
i = 1, . . . , N , . Js
.
4.3
, ..,

!
n
n
1X 1
1X 1
(Jt , It ) min
(Jt , i) 6 0
(8.1)
lim sup
i=1,..., N n
n
n
t=1

t=1

362

 1,
pt , t = 1, 2, . . . .

. , .. 1
!
n
n
1X 2
1X 2
lim sup
(It , Jt ) min
(It , j) 6 0.
(8.2)
j=1,..., M n
n
n
t=1

t=1

(8.1) : 1
!
n
n
1X
1X
lim inf
f (It , Jt ) max
f (i, Jt ) > 0,
(8.3)
n
i=1,..., N n
n
t=1

t=1

I1 , I2 , . . . p1 p2 . . . .
(8.2) : 1
!
n
n
1X
1X
lim sup
f (It , Jt ) min
f (It , j) 6 0,
(8.4)
j=1,..., M n
n
n
t=1

t=1

J1 , J2 , . . . q1 q2 . . .
.
, ,
,
.

,
,
.

363

 8.1. ,
, .

n

lim inf
n

1X
f (Ii , Jt ) > v,
n

(8.5)

t=1

, v .
, , , 1,

1X
lim
f (Ii , Jt ) = v.
n n

(8.6)

t=1

.
v = max min f (
p, q) = min max f (
p, q).
p

,
f (
p, q) =

N X
M
X

pi qj f (i, j),

i=1 j=1

f (
p, j) =

f (i, q) =

N
X
i=1
M
X

pi f (i, j),

qj f (i, j).

j=1

(8.3), (8.5) ,
J1 , J2 , . . .
n

max

i=1,..., N

1X
f (i, Jt ) > v
n
t=1

364

(8.7)

 n. ,

i=1,..., N

n
P

t=1

t=1

1X
1X
f (i, Jt ) = max
f (
p, Jt ),
p n
n

max

f (
p, Jt ) p, -

t=1

,
{1, . . . , N }, .

n
1X
qj,n =
1{Jt =j}
n
t=1

, j.
qn = (
q1,n , . . . , qM,n ).
max
p

t=1

j=1

X
1X
f (
p, Jt ) = max
qj,n f (
p, j) =
p
n
= max f (
p, qn ) > min max f (
p, q) = v
p

J1 , J2 , . . .
(8.6) (8.4) ,
n

min

j=1,..., M

1X
f (It , j) 6 v = max min f (
p, q)
p
q
n
t=1

I1 , I2 , . . . . (8.7).

n

lim sup
n

1X
f (Ii , Jt ) 6 v,
n

(8.8)

t=1

, v .
(8.8) (8.5) (8.6). . 4
365

8.2.
8.1 , ,
, ,
.


S . (Blackwell approachability theorem). ,
,
1
S, ,
.
1956 . [11]
.
,
.
- .
f(i, j) d- Rd . , I = {1, . . . , N },
J = {1, . . . , M }.
I J . P(I) P(J ) . p = (p1 , . . . , pN ) P(I) q = (q1 , . . . , qM )

366

P(J )
f(
p, j) =
f(i, q) =

N
X
i=1
M
X

pi f(i, j),
qj f(i, j),

j=1

f(
p, q) =

N X
M
X

pi qj f(i, j).

i=1 j=1

,
v
u d
uX
k
x yk = t (xi yi )2
i=1

x
, y Rd . S Rd x
Rd ,
x
S
dist(
x, S) = inf k
x yk.
yS

S dS (
x) -
y S, dist(
x, y) .
S , .
S Rd (approachable), p1 , p2 , . . . ,
J1 , J2 , . . . P - I1 , I2 , . . .
!
T
1X
lim dist
f (It , Jt ), S = 0,
T
T
t=1

P I1 , I2 , . . .
, p1 , p2 , . . . .
Rd .
, S f(i, j) Rd .
367

 8.2. S
Rd . x
6 S x ,
dS (
x) , x
dS (
x).
, x
6 S p P(I) , f(
p, 1), . . . , f(
p, M )
x
x .
S .
. I1 , I2 , . . . J1 , J2 , . . . - .
t
t

1X
m
t =
f (Ii , Ji ).
t
i=1

< t I1 , . . . , It1
J1 , . . . , Jt1 . , m
t1 6 S.
m
t1 )
t1 , dS (m
, m
t1 dS (m
t1 ),

(w
t1 x
) bt1 = 0,

w
t1 =

m
t1 dS (m
t1 )
km
t1 dS (m
t1 )k

bt1 = (w
t1 dS (m
t1 )).
, m
0 = 0.
, m
t1 (
).
x
= m
t1
pt ,
f(
pt , 1), . . . , f(
pt , M )

368

 :
(w
t1 f(
pt , j)) bt1 6 0
j = 1, . . . , M .
max (w
t1 (f(
pt , j) dS (m
t1 ))) 6 0.

16j6M

(8.9)

pt (8.9).
, m
t S.

d(m
t , S) = km
t dS (m
t )k 6 km
t dS (m
t1 )k.

(8.10)

,
m
t =

1
t1
m
t1 + f(It , Jt ).
t
t

(8.11)

(8.10)
(8.11) :

2
t 1

1

d(m
t , S) 6
m
t1 + f (It , Jt ) dS (m
t1 )
=
t
t
2


t 1
1

(m
t1 dS (m
t1 )) + (f (It , Jt ) dS (m
t1 ))
=
=
t
t


t1 2
km
t1 dS (m
t1 )k2 +
=
t
t1
+2 2 ((m
t1 dS (m
t1 )) (f(It , Jt ) dS (m
t1 ))) +
t
1
+ 2 kf(It , Jt ) dS (m
t1 )k2 .
(8.12)
t
2

S f(i, j)
Rd ,
kf(It , Jt ) dS (m
t1 )k 6 2.
369

 (8.11) (8.12)

t2 km
t dS (m
t )k2 (t 1)2 km
t1 dS (m
t1 )k2 6
6 4 + 2(t 1)((m
t1 dS (m
t1 )) (f(It , Jt ) dS (m
t1 ))). (8.13)

t1
km
t1 dS (m
t1 )k.
T
0 6 Kt1 6 2 t 6 T . t = 1, . . . , T
(8.13) T 2 :
Kt1 =

km
T dS (m
T )k2 6
T
4
2X
6 +
Kt1 (w
t1 (f(It , Jt ) dS (m
t1 ))) 6
T
T
t=1

T
4
2X
+
Kt1 (w
t1 (f(It , Jt ) f (
pt , Jt )).
T
T

(8.14)

t=1

(8.9).
(8.14)
-. 1 4.9 0 T .

d(m
T , S) = km
T dS (m
T )k 0 T
1. . 4
,
.
8.3. S Rd , q P(J ) p P(I) , f(
p, q) S.
1
Ept (f(It , Jt )) = f(
pt , Jt ), E . Kt1 .

370

. , q P(J ) p P(I) , f(
p, q) S. x
0 6 S dS (
x0 )
x
0 S. a(i, j) = ((dS (
x0 )
x0 ) f(i, j)).

max min a(
p, j) = min max a(i, q).
p

(8.15)

q P(J ) p ,
f(
p, q) S. (8.15)
max min((dS (
x0 ) x
0 ) f(
p, j)) =
p

= min max((dS (x0 ) x

0 ) f(
p, q)) >
q

> min((dS (
x0 ) x
0 ) s) =
sS

= ((dS (
x0 ) x
0 ) dS (
x0 ))

(8.16)

j. (8.16)
dS (
x0 ) S.

L(
x) = ((dS (
x0 ) x
0 ) x
) ((dS (
x0 ) x
0 ) dS (
x0 )) = 0,
dS (x0 ) dS (
x0 )
x
0 . ,
((dS (x0 ) x
0 ) x
0 ) < ((dS (
x0 ) x
0 ) dS (
x0 )).
L(
x0 ) < 0, .., x
0 L(
x) = 0.

(8.16) , p P(I)
, j = 1, . . . , M
((dS (
x0 ) x
0 ) f(
p, j)) > ((dS (
x0 ) x
0 ) dS (
x0 )).
, L(f(
p, j)) > 0 j = 1, . . . , M .
, L(
x) = 0
x
0 . , S 8.2.
371

 , , q0 P(J ) , f(
p, q0 ) 6 S p P(I).
8.2 f0 (i, j) = f(j, i)
T (
q0 ) = {f(
p, q0 ) : p P(I)}.
f0 (
q0 , 0), . . . , f0 (
q0 , M ) T (
q0 ).
T (
q0 ) , x
6 T (
q0 )
0
0

x
f (
q0 , 0), . . . , f (
q0 , M )
x , dT (q0 ) (
x) , x
dT (q0 ) (
x).
8.2 T (
q0 )
q0 .
, T (
q0 ) S = .
S T (q0 ) . ,
S
(. 8.5). . 4
8.2
.
I = {1, . . . , N } J = {1, . . . , M } . P(I) P(J )
. l(i, j), 0 6 l(i, j) 6 1 i, j.
t pt ,
J1 , J2 , . . . 1
!
T
T
1X
1X
lim sup
l(It , Jt ) min
l(i, Jt ) 6 0,
(8.17)
16i6N T
T
T
t=1

t=1

I1 , I2 , . . . P
p1 , p2 , . . . .
, 8.2

S = {(u1 , . . . , uN ) : ui 6 0, i = 1, . . . , N },

372

l(i, j) l(1, j)

...

.
l(i,
j)

l(k,
j)
f(i, j) =

...
l(i, j) l(N, j)
, f(i, j) N -

N
.
1/ N , f(i, j)
.
x
0 6 S. dS (
x0 ) = 0
x0 (w
x
) = 0, wi
w
.
, (8.17), ,
p P(I) , f(
p, 1), . . . , f(
p, M )
(w
x
) = 0, ..
N
X

wk (l(
p, j) l(k, j)) 6 0

k=1

j = 1, . . . M . ,

p = N
.
P
wi
i=1

8.2 p1 , . . . , pt , . . . , (8.17) 1.

8.3.

373

 8.3.
[29].
3.2 pi i . . {0, 1} ,
pi i {0, 1}.
, A = {a1 , . . . , am }. P(A)
A. ( ) p = (p1 , . . . , pm )
1.
k
pk1 =
max16i6m |pi |. k
pk2 Rm . ,
Rm . k
pk .
i ] = (0, . . . , 1, . . . , 0) [a
, ai A.
i- 1, .
. t pt P(A),
at A.
, pt t ] .
, [a
p1 , p2 , . . . , , t pt P(A),
Pt P(P(A)).
, .
.

374

 - [4] Pt P p1 , p2 , . . . .
 > 0. pt P , p P(A) a1 , a2 , . . .
P - -:


T
1 X


t ] pt )
lim sup
Ikpt pk6 ([a
(8.18)
6 ,

T T
t=1

p1 , p2 , . . . P

1, k
pt pk 6 ,
Ikpt pk6 =
0, .
pt
P .
- - P = {
s1 , . . . , sN }
P(A). ,
p P(A) - si P ,
k
p si k < .
Pt P(P(A)),
P .
P = {
s1 , . . . , sN } I = {1, 2, . . . , N },
t Pt
I.
P i1 , i2 , . . . P Pt .
(8.18)
: P -


N
T

X
1 X

lim sup
I{it =k} ([at ] sk ) 6 .
(8.19)



T
T
k=1

t=1

- .
375

 8.4.  > 0 Pt , t = 1, 2, . . .,
, P - -
(8.18), P
p1 , p2 , . . . Pt .2
. 8.3, 3 I =
{1, 2, . . . , N },
J = A. N |A| :

f (k, a) =

0
...
0

[a] sk
0
...
0

m- , m = |A|,
k I a J , 0

[a] pk
m, k- .
RmN .
RmN = (
N - Rm : X
x1 , . . . , x
N ),
m
x
i R .
(
)
N
X
:
k
xk k 6  .
C= X
k=1

8.3 C , q P(J )
p P(I) , f(
p, q) C.
2

(8.18) (8.19).
P = {
s1 , . . . , sN } I =
{1, 2, . . . , N }.
3

376

 8.3 C, q P(J ) = P(A)

, sk P , k
q sk k 6 ,
.. f(k, q) C. 8.3
I = {1, . . . N },
p [k]
k, 1 6 k 6 N . f(
p, q) C.
8.2 Pt P(I) , a1 , a2 , . . .

T
1 P

I
(
[a
]

)
t
1
T t=1 {it =1}

1X
.
.
.
.
f (it , at ) =

T
T

P
t=1
1
t ] sN )
I
(
[a
{i
=N
}
t
T
t=1

C, i1 , i2 , . . .
P Pt .
, (8.19) . 4
() , -
 > 0.
,
P = {
s1 , . . . , sN } (8.19),
- .
8.4 .
8.5. Pt , p P(A)
 > 0


T

1 X

t ] pt )
lim
(8.20)
Ikpt pk6 ([a
=0

T T
t=1

.
377

 i , i 0 i .
,
i - P(P(A)).
i - i- i1 -
i- .
3.5 3.5.1.
(8.19)
:
8.6. Pt ,


T

X
1 X

lim
I{pt =p} ([at ] p) = 0
(8.21)

T

T
pP(A)

t=1

.
, (8.21) :
p {
pt : 1 6 t 6 T }.

.

8.4.
,
, , , ,

.
378

 A N N - :
p. a A
p(a) a- p. , p(a) a A.
p k
pk RN (
) dist(
p, q) = k
p qk. p RN S RN :
dist(
p, S) = inf dist(
p, q).
qS

p1 , p2 , . . .
S,
lim dist(
pt , S) = 0.
t

k , . i (): Ai = {1, . . . , Ni }, i = 1, . . . , k. , i
f i (i1 , . . . , ik ), is As , s = 1, . . . k, .
s As .
s1 , . . . , sl As1 . . . Asl .
Q
Q
A = kj=1 Aj Ai = j6=i Aj . pti
, i, .
, pi P(Ai ).
:
X
f i (a, pi ) = Epi (f i (a, )) =
f i (a, a
i )
pi (
ai ),
a
i Ai

a
i = (a1 , . . . , ai1 , ai+1 , . . . , ak ),
(a, a
i ) = (a1 , . . . , ai1 , a, ai+1 , . . . , ak ),
a Ai , a
i Ai , E
pi .
379

 t =
1, 2, . . . .
FOR t = 1, 2, . . .
i = 1, . . . , k, i j 6= i
pti ( )
ati Ai , ,
, pti :
ati argmaxaAi f i (a, pti ).

(8.22)

ENDFOR
a i, f i (a, pti ),
pti . , ati , - .
.
a
t = (at1 , . . . , atk ) t.

T
1 X t
pT =
[
a]
(8.23)
T
t=1

- , a]

Qk T . [
i=1 |Ai |, , a
, 1,
0.
pT
a
= (a1 , . . . , ak )
a
t = (at1 , . . . , atk ), t =
at ],
1, . . . , T . Q
pT , [
k
t
t
(a1 , . . . , ak ): i=1 |Ai |.
a
= (a1 , . . . , ak )
1
pT (
a) = |{t : 1 6 t 6 T, a
t = a
}|
(8.24)
T
380

 a
( pT ).
, , (8.21), (8.22) ,

C .
8.7. i p1i , p2i , . . . a
1i , a
2i , . . . i, pT , (8.23), C
.
. , ,

dist(
pT , C) 0
T , C
. , C 6= .

Q
A = ki=1 Ai ( |A|)
. {
pT :
T = 1, 2, . . . }, (8.23), pTj . p
. , p .
i a Ai i ,
X
p (a) =
p (
a) > 0,
a
:ai =a

a
= (a1 , . . . , ak ), as As , s = 1, . . . , k.

p (a) = 0, a
; , i- a.
a .
4

381

 f = f i (
P(Ai ) :
i a) B, B
B = {
qi : f (a, qi ) = max
f (a0 , qi )}
0
a Ai

i,
a .
, B .




t
0

B = qi : t qi = qi )&f (a, qi ) = max

f (a , qi )
0
a Ai

,
i t = 1, 2, . . . ,
B.
a . B

, B
,
.
a
i , i, ai = a ( a
) p :
p (
ai |ai = a) = p ((a, a
i )|ai = a) =

p (a, a
i )
.
p (ai = a)

(8.25)

QK 6.3 p
= (a1 , . . . , aK )
k=1 {1, . . . , Nk } a
,
i {1, . . . , K}
a, a0 {1, . . . , Ni }
f i (a, p(|ai = a)) = max
f i (a0 , p(|ai = a)).
0
a Ai

, p
, p (|ai = a) B i a Ai .
, p (|ai = a) B
.
382


NT (a) = |{t : 1 6 t 6 T, ati = a}|
6 T , i a,
NT (
pi ) = |{t : 1 6 t 6 T, pti = pi }|
6 T , i
pi P(Ai ).
pT pT (|ai = a) ,
i, ,
a
i ai = a pT :
pT (
ai |ai = a) =

pT (a, a
i )
.
pT (ai = a)

(8.26)

(8.25) pTj (ai |ai = a) p (ai |ai = a) j .

a Ai
B
t

a
i- pti B.
, (a, a
i )

{
at : 1 6 t 6 T }
a
i
1 6 t 6 T }.
{
ati : pti B,
(8.24) :
pT (a, a
i ) = pT (
a) =

1
a
|{t : 1 6 t 6 T, pti B,
ti = a
i }|.
T

pT (ai = a) = NTT(a) . ,

383

 a
t , ati = a, t = 1, . . . , T :
pT (|ai = a) =

=

T
NT (a)

1 X
T

1
NT (a)

at ] =
[
i

16t6T, pti B

at ] pi ) +
I{pti =pi } ([
i

(8.27)


X  NT (
pi )
pi .
NT (a)

(8.28)

16t6T
pi B

pi B

p (ai = a) > 0 p pTj



Tj
j , NT (a) .
j

NT (a) =

NT (
pi ).

pi B

, (8.21) (8.27) pTj (|ai = a) Tj

B,
B.
dist(
pTj (|ai = a), B) 0
Tj .
pTj p j , a
i pTj (
ai |ai = a) p (
ai |ai = a)
Tj . B ,
p (|ai = a) B i a Ai . , ,
p
. . 4
8.4 8.7, :
8.1. , 1 6 i 6 N p1i , p2i , . . . i,
:
384


.
pT C
T 1.

8.5.
1. (8.8).
2. ,
.
3. , S f (i, j),

f 0 (i, j) = f (j, i).

385


[1] ., . . .: , 1965.
276 c.
[2] .., .. ( ). .: , 1974
416 c.
[3] ..
: . . .: : , 2010.
231.
[4] .. . .: , 2007. 968 .
[5] .., .. :
. .: , . , 2006. 280 .
[6] Alon N., Ben-David S., Cesa-Bianchi N., Haussler D. // Scalesensitive dimensions, uniform convergence, and learnability. J.
ACM V. 1997. 44(4). P. 615-631.
[7] Anthony M Bartlett P.L. Neural network learning: Theoretical
foundations, Cambridge: Cambridge University Press, 1999.
[8] Aronszajn N. Theory of reproducing kernels // Transactions of
the American Mathematical Society. 1950. V. 68. P. 337404.
[9] Bartlett P., Mendelson S. Rademacher and Gaussian Complexities: Risk Bounds and Structural Results // Journal of Machine
Learning Research. 2002. V.3. P. 463-482.
386

[10] Bartlett P., Bousquet O., Mendelson S. // Local Rademacher

Complexities. The Annals of Statistics. 2005, V. 33, No. 4, 1497
1537.
[11] Blackwell D. An analog of the minimax theorem for vector payoffs
// Pacific Journal of Mathematics. 1956. V. 6. P. 18.
[12] Bousquet, Olivier, Stephane Boucheron, and Gabor Lugosi. Introduction to statistical learning theory. Advanced Lectures on
Machine Learning. 2004. P. 169207.
[13] A. Chernov, F. Zhdanov. Prediction with expert advice under discounted loss. Technical report, arXiv:1005.1918v1 [cs.LG],
2010.
[14] Cover T., Ordentlich E. Universal portfolio with side information
// IEEE Transaction on Information Theory 1996. V. 42.
P. 348363.
[15] Cristianini N., Shawe-Taylor J. An Introduction to Support Vector Machines. Cambridge UK: Cambridge University Press,
2000.
[16] Dawid A.P. Calibration-based empirical probability [with discussion] // Ann. Statist. 1985. V. 13. P. 12511285.
[17] Foster D.P., Vohra R. Asymptotic calibration // Biometrika.
1998. V. 85. P. 379390.
[18] Freund Y., Schapire R.E. A Decision-Theoretic Generalization of
On-Line Learning and an Application to Boosting // Journal of
Computer and System Sciences 1997. V. 55. P. 119139.
[19] J. Hannan. Approximation to Bayes risk in repeated plays. In
M. Dresher, A.W. Tucker, and P. Wolfe, editors, Contributions
to the Theory of Games 3, pages 97-139, Princeton University
Press, 1957.
[20] M. Hutter and J. Poland. Adaptive online prediction by following
the perturbed leader // Journal of Machine Learning Research,
6:639660, 2005.
387

[21] Kakade, S.M., Foster, D.P. Deterministic calibration and Nash

equilibrium // Lecture Notes in Computer Science Berlin:
Springer, 2004. V. 3120. P. 3348.
[22] Sham Kakade and Ambuj Tewari. Topics in Artificial Intelligence (Learning Theory) - Spring 2008. Lecture Notes. http :
//ttic.uchicago.edu/ tewari/LTS P 2008.html
[23] A. Kalai and S. Vempala. Efficient algorithms for online decisions. In Bernhard Scholkopf, Manfred K. Warmuth, editors,
Proceedings of the 16th Annual Conference on Learning Theory COLT 2003, Lecture Notes in Computer Science 2777, pages
506521, Springer-Verlag, Berlin, 2003. Extended version in Journal of Computer and System Sciences, 71:291307, 2005.
[24] Yuri Kalnishkan. Kernel Methods (Introduction), 2008 (Unpublished manuscript).
http
:
//onlineprediction.net/?n
=
M ain.KernelM ethods#toc8
[25] Kimeldorf G. S. and Wahba G. Some results on Tchebycheffian
spline functions // J. Math. Anal. Appl. 1971 V. 33 82-95.
[26] Ledoux, M. and Talagrand, M. Probability in Banach Spaces:
Isoperimetry and Processes. Springer, New York. 1991.
[27] Littlestone N., Warmuth M. The weighted majority algorithm //
Information and Computation 1994 V. 108 P. 212261.
[28] Lugosi G., Cesa-Bianchi N. Prediction, Learning and Games.
New York: Cambridge University Press, 2006.
[29] Mannor S., Stoltz G. A Geometric Proof of Calibration //
arXiv:0912.3604v2. 2009.
[30] McDiarmid C. On the method of bounded differences. London
Mathematical Society Lecture Notes Series. Surveys in Combinatorics. Cambridge University Press. V. 141. pp. 148188. 1989.
[31] Shafer G., Vovk V. Probability and Finance. Its Only a Game!
New York: Wiley. 2001.
388

[32] Shawe-Taylor J., Cristianini N. Margin distribution bounds on

generalization // In Proceedings of the European Conference
on Computational Learning Theory, EuroCOLT99. P.263273.
1999.
[33] Shawe-Taylor J., Cristianini N. Kernel Methods for Pattern Analysis. Cambridge UK: Cambridge University Press, 2004.
[34] Scholkopf B. and Smola A. Learning with Kernels. MIT Press,
Cambridge, MA, 2002.
[35] Steinwart I. On the influence of the kernel on the consistency of
support vector machines. Journal of Machine Learning Research,
2, 67-93, 2001
[36] Valiant L.G. A theory of the learnable, Communications of the
ACM V. 27(11). P. 1134-1142. 1984.
[37] Vapnik V.N. Statistical Learning Theory. New York: Wiley,
1998.
[38] Vovk V. Aggregating strategies // Proceedings of the 3rd Annual
Workshop on Computational Learning Theory (M. Fulk and J.
Case, editors,) San Mateo, CA: Morgan Kaufmann, 1990. P.
371383.
[39] Vovk V. A game of prediction with expert advice // Journal of
Computer and System Sciences 1998 V. 56. No. 2.
P. 153173.
[40] Vovk V., Watkins C. Universal portfolio selection // Proceedings of the 11th Annual Conference on Computational Learning
Theory New York: ACM Press, 1998. P. 1223.
[41] Vovk V. Competitive on-line statistics // International Statistical Review 2001 V. 69. P. 213248.
[42] Vovk V, Gammerman A., Shafer G. Algorithmic Learning in a
Random World. Springer, New York, 2005.

389

[43] Vovk V., Shafer G. Good randomized sequential probability forecasting is always possible // J. Royal Stat. Soc. B. 2005 V.
67 P. 747763.
[44] Vovk V., Takemura A., Shafer G. Defensive forecasting // Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics (ed. by R. G. Cowell and Z. Ghahramani)
Cambridge UK: Society for Artificial Intelligence and Statistics,
2005. P. 365372.
[45] V. Vovk. On-line regression competitive with reproducing kernel Hilbert spaces (extended abstract). TAMS Lecture Notes in
Computer Science Berlin: Springer, 3959, 2006, 452463
[46] Vovk V. On-line regression competitive with reproducing kernel
Hilbert spaces (extended abstract) // Lecture Notes in Computer
Science Berlin: Springer, 2006. V. 3959. P. 452463.
[47] Vovk V. Predictions as statements and decisions // Theoretical
Computer Science 2008. V. 405. No. 3. P. 285296.
[48] Vyugin V., Trunov V. Universal algorithmic trading. Journal of
Investment Strategies. V.2 (1), Winter 2012/13 P. 6388.

390