Вы находитесь на странице: 1из 29

1.

, ,
:
1. a = b,
, a b. a2 = b2 ,
, , .. ,
. a3 = b3 ,
;
2. .
,
;
3. () ().
. .,
, ..
;
4. .
. X, -
Y = X +Z,
Z , .
,
. Y , X. ( Z ),
Y . Y
X.
1865 .
.
1921 . ,
,
.
1948 .
.

, ,

, ,
.
4
x = 5
x > 3?

1
2.


,
.
.
X Y ,
P(X = Xi) = pi, P(Y = Yj) = qj

P(X = Xi, Y = Yj) =pij ,
, X Y ,

X Y ,
pX(t1), pY (t2) pXY (t1, t2),

, ,

X
H(X) = HX = I(X,X).
:
1) I(X, Y ) 0, I(X, Y ) = 0 , X Y ;
2) I(X, Y ) = I(Y,X);
3) H(X) = 0 , X ;
4) I(X, Y ) = H(X) + H(Y) H(X, Y ), H(X, Y ) = i,j pij log2 pij ;
5) I(X, Y ) I(X,X). I(X, Y ) = I(X,X), X Y .


.
2
. . X1, X2 Y . X1 X2
, 1- 2- , Y = X1+X2.
. I(Y,X1), I(X1,X1), I(Y, Y ).
... X1 X2 , ..
.

.. j = 1...6 qj = P(X1 = j) = 1/6.

... Y ,

P(Y = i) = P(X1 + X2 = i), i = 2...12,

, X1, X2

P(X1 = n,X2 = m) = P(X1 = n)P(X2 = m),


, Y :14

.. i = 2...12, pi = P(Y = i) = (6 |7 i|)/36.

... X1 Y

3
pij = P(Y = i,X1 = j) = P(Y = i/X1 = j)P(X1 = j),

P(Y = 2,X1 = 1) = P(Y = 2/X1 = 1)P(X1 = 1) == P(X2 = 1)P(X1 = 1) = 1/36.


4
0 < I(Y,X1) = I(Y,X2) < I(X1,X1) = I(X2,X2) < I(Y, Y ),
.

1/36 2 log2 6 = I(X1,X1)/18
I(X1, Y ) 36, Y = 2
Y = 12, X1. , Y = 7,
X1,
6 log2 1 = 0.
, 4- ,

I(Y,X1) = H(X1)+H(Y) H(X1, Y ) = H(Y) H(X1) 3.272.58 = 0.69


/.
4- ,
, .

5
. . .. X ,
, . . . Y 0,
, 1, .
I(X, Y ) I(Y, Y ).
. ... X Y .

, i = 1...6 pi = P(X = i) = 1/6 , , j =


0...1 qj = P(Y = j) = 1/2.
...

,
.. 1 . I(X, Y ) = I(Y, Y ) = 1 / 3- ,
X Y , , ..
I(X, Y ) 6= I(X,X) = 1+log2 3 2.58 /.
, Y X, X Y
.

H(X, Y ) = i,j pij log2 pij = log2 6 = 1 + log2 3 = H(X),

I(X, Y ) = H(X) + H(Y) H(X) = H(Y) = 1 /.


... X,

6

. . . X1 X2
, ... Y ,
. X1 Y ?


X1 ... Z = (X1+1)2X2,
. . . X1 X2
0, 1?
H(X1) H(Z). X1 Z?


. . . X1, X2 .
.. .
I(X1,X2), X1 X2


. . . X1 X2 ,
1 4. . . . Y ,
, . . Y = X1 + X2.
I(X1, Y ), H(X1) H(Y)


X1 ... Z = X1_X2,
H(Z). ... X1 X2 .


... X1 1, 0 1
. ... X2
0, 1 2. X1 X2 .
Y = X1 2 +X2.
I(X1, Y ), I(X2, Y ), H(X1), H(X2), H(Y) .

7

. . . X, Y , Z ,
Z = X + Y Y . X Y

3.
... ,
...
(). 4
, .. 1/4.
. . . X, . H(X) = 2.

. : 1
00, 201, 310, 411.
L(X), ,
X, . . ML(X)
, X. L
L(X) = len(code(X)), code(X) X
, , , len
.
ML(X) = H(X).
... X
P(X = 1) = 3/4, P(X = 2) = 1/8, P(X = 3) = P(X = 4) =1/16,
.. 1 .

HX = 3/4log24/3+1/8log2 8 +1/8log2 16 =19/8 3/4log2 3 1.186 /.
: 10, 210, 3110, 4111, . . ,
(
). 16 1-
12 , 2- 2-, 3- 1- 4- 1-. ,
(1 _ 12 + 2 _ 2 + 3 _ 1 + 3 _ 1)/16 = 1.375 /
. . L(X). , L(X)
: P(L(X) = 1) = 3/4, P(L(X) = 2) = 1/8, P(L(X) = 3) =
1/8.
,
ML(X) = + 2/8 + 3/8 = 11/8 = 1.375 /.
, ML(X) > H(X).

8
,
.
,
,
. ,
N ,
. ,
pi ( iN pi = 1), i
. , ,
, iN pi log 2 pi ,
N, . ,
,
, , .


. . . X
...


... X ,
.
X. X,
.


... X P(X = 2n) = 1/2n, n = 1, 2, . . .
... X,
.

9
... X , .
X,
.

. . .
X.

4.

50- XX
.
, ,
,
.

, ,
.
inf(s) = log2 p(s), s ,
, p(s) s.
:
1) s1 s2 ( s1 s2) , inf(s1) inf(s2);
2) inf(s) 0;
3) s , inf(s) = 0;
4) inf(s1s2) = inf(s1) + inf(s2) , p(s1 s2) = p(s1)p(s2), . .
s1 s2.
- ,
. : s1 a > 3 s2 a = 7
, s2 s1 inf(s2) inf(s1); , s2
, s1.
-
cont(s) = 1 p(s). , cont(s) = 1 2inf(s) inf(s) = log2(1 cont(s)).


inf(s) cont(s) s1, ,
50%, s2, 25%.

5.
,
,
( ,

10
, ,
). ,
, ,
, ..
- ZIP, GZ, ARJ
.
,
( 80- ),
.
.
n
, ... Xi
n ... X.
[20], ,
. . ., , ...,
.. ML(X) > HX ... X .
, [20] ,
(-, Fano), HX > ML(X) 1.
... X1 X2, . HX1
= HX2 I(X1,X2) = 0, ,
H(X1,X2) = HX1 + HX2 I(X1,X2) = 2HX1.
X1 X2 . . . ~X = (X1,X2).
n- ... ~X = (X1,X2, . . . ,Xn)
, H ~X = nHX1.
L1( ~X ) = L( ~X )/n, ~X = (X1,X2, . . . ,Xn), .. L1( ~X )
~X . ML1( ~X )

~X . ML( ~X ) 1 6 H ~X 6 ML( ~X )
- ~X ML1( ~X ) 1/n 6 HX1 6 ML1( ~X )
.
,
, , n ,
-

.
- ,
. ,
,
.


11
, .
.
> 0 s,
s ( n/s ),
- , ,

, ". ,
Y = (Y1, Y2, . . . , Yn/s), Y1 = (X1,X2, . . . ,Xs), Y2 = (Xs+1,Xs+2, . . . ,X2s) . .,
. . Yi = (Xs(i1)+1,Xs(i1)+2, . . . ,Xsi).

H(Y1 )= sH(X1) sML1(Y1) = ML(Y1) H(Y1 ) + 1 = sH(X1) + 1,
,
ML1(Y1) H(X1) + 1/s,
.. s = 1/. s "
1/".
. ... X1,X2, . . .Xn ,
P(Xi = 0) = p = 3/4
P(Xi = 1) = q = 1/4 i 1 n.

HXi = 3/4log24/3+1/4log2 4 = 2 3/4 log2 3 0.811 /.
0 1 1 .

1. 2.
2- ... X = (X1,X2)



ML1( X ) = (1 9/16 + 2 3/16 + 3 3/16 + 3 1/16)/2 = 27 /32 = 0.84375,
.. , . 3
_ 0.823,
4 _ 0.818 ..
, ...
( 0 1). ...
m . ... X ,
12
ML( X ) H (X) / log2m ML1( X ) H(X1)/ log2m.
, ,
ML( X ) 1 H (X) / log2m

ML1( X ) 1/n H(X1)/ log2m, n = dim(X ).
, ,
,
, .. ,
...,
.

6.
- , ...
,
, 0,
1.

. , . .
B C.

(Huffman) 1952 .
-, ,
. ()
. ... ;
, . , ,
.
, ;
,
. ,
, 0 1. . .

13
. , ,
.
-
. ,
2 , 10 C, 0 A
..
...

ML1( X ) = ML( X )/2 = 27/32 = 0.84375 /.

ML1(X) = ML(X) = 1.6 /.


ML1( X ) X. 2
. ... X .

14
H(X) ML(X) - X.
... X :

7.
, ,
.
, , , ,
10 .
0 1 . .
.., 0.469 /.

1 . ,
.
,
70- XX .

. . . ,
...;
[0, 1],
. . .
,
...
, ,
.
.

,
, -,
. .
n1, n,
, ...
- (
).
,
n.

, .. .
. ..
, .

( -
15
). ,
,
, ..
.
, ,
. ,
,
.
. ... X
0 1 2/3 1/3 .
0 [0, 2/3], 1 [2/3, 1]. ... X ,
dim(X ) = 3, H(X) = H (X) /3 = log2 3 2/3 0.9183 /.,

ML1( X ) = 65/81 0.8025 /. (),


ML1( X ) = 76/81 0.9383 /. ( ),
ML1(X) = ML(X) = 1 /. ().


, . ,
, -
,
.

.
1. . . . ,
,
.
, .

16
2.
, .
. 1.
, , 111.
7/8 2 [2/3, 1], ,
1. 7/8 2/3 1/3,
5/8 2 [0, 2/3], , 0. , (5/8
0) _ 3/2 = 15/16 2 [2/3, 1], 1, . .
101 . , - ,
,
1 ..



. . .,
, -,
.
, ...
0 1.



AAB, . . . X
P(X = A) = 1/3, P(X = B) = 2/3.


011
... . ,
.

23
BAABC,
... X P(X = A) = 1/4, P(X = B) =

17
1/2, P(X = C) = 1/4. ,
X P(X = A) = 1/3, P(X = B) = 7/15, P(X = C) = 1/5?

24
. .. X .
4 X . . . ~X
X. ~X
?
. . .,
X ?

25
, ( 2 3)
ABAAAB, .
...,
, .

8. .

, ,
. , .. ,
, ,
,
, ..
.
.

, 0.
: .
escape, hESCi. ASCII
8- , .. 0 255.

- .

. .
0, 1.

ACCBCAAABC, 10-
... X 2- :

18
L1(ACCBCAAABC) = 4.1 /. ,
L1(ACCBCAAABC) = 8 /. . ..
ML1(X) = 1.6 / HX _ 1.523 /.

,
-, ..

.
A0C100
B1001010100101. ,

19
, ,
ASCII+. escape-
0.
.

, ..
.
,
, .
,
,
, , .
-
.
. 4 .

20
.4
,

: ,
, .
-
.
-
.
hESCi , , .

:

21
L1(ACCBCAAABC) = 4.1 /.
26
BBCBBC,
.

27
AABCDAACCCCDBB,
,
.
ASCII+ .

28
A0F00X0111110101011011110100101,
,
.

9. -
. -

-,
.
.
,
.
,
.
LZ77 1977 .
(Ziv) (Lempel).

LZ77. LZ
.

22
LZ77 ,

. LZ77 .
,
. LZ77
, . ,
, . ,
, ,
.
,
. ( )
, .
LZ77 , :
- ,
;
- ;
- , .
. 20 , 12 , 8.

MICROSOFT. .
, . ,
, ,
0 3 ,
. , h0,3,i.

+1
. ,
.
h0,0,i,
. ,
. ..
LZ77 , ..
.
LZ77:
1) -
;
2) .
,
. ,
LZ77
.
. LZ77 .
23
, , . . .
:
,
1. ,
log2( ),
log2( +1).
8 (, ASCII+).
9_(3+3+8) =126 ,
14 _ 8 = 112 .
1982 . (Storer) (Szimanski) LZ77
LZSS, LZ77
.
, LZSS, ,
.
: , LZ77. LZSS
1,
. LZSS ,

.
. LZSS .

24
7 _ 9 + 4 _ 7 = 91 .
LZ77 LZSS :
1) ,
, ;
2) , ,
.
,
, ..
,
. ,
-.
1978 . LZ77 LZ78,
.
LZ78 ,
.
( ).
,
.
, , ,
,
, .
. ,
.

, LZ78
25
. ,
,
+8 ( - ASCII).
. LZ78 ,
16

0 15,
.
10 _ (4 + 8) = 120
.
LZ77, LZ78 LZSS
.
1984 . (Welch) LZ78
LZW.
-.
1.
( 256 ASCII).
w .
2. K .
3.
w

wK
wK

26
2

w
wK
K
2.

LZ78 LZW
: LZW- ,

.
. LZW .
500 .

12 _ 9 = 108 .
, . .
,
, , .

27
LZW , ,
.
.
LZW. LZW
IBM Unisys,
Unisys, . ,
LZW Unix
compress.

30
AABCDAACCCCDBB,
, ,
, LZ77 ( 12 , 4 ), LZ78
( 16 ), LZSS ( 12 , 4 ),
LZW ( ASCII+ 16 ).
31
LZ78
LZW ? . LZW
ASCII+.

10. -
LZ ()
,
(, )
: GZIP, ARJ, PKZIP, ...
,
,
.
-
, ,
,
, ,
.
,
, RAR. Unix (gzip, bzip2, ...)
.
1992 WEB Technologies
DataFiles/16,
1024 .
, Byte.

.
28
. ,
100 (
2800). ,
.
, 100
1 + 28 + 216 + + 2792 = (2800 1)/255 < 2800,
, .
, ,
.
, ,
-, ,
.

29