Академический Документы
Профессиональный Документы
Культура Документы
Basic Information
Theory
Thinh Nguyen
Oregon State University
What is information?
1.
2.
What is information?
1.
2.
What is information?
1.
2.
What is information?
f ( s l ) lf ( s)
A message of length
possibilities.
l,
s possibilities.
distinguishable
sl
I log( s ) l log( s )
l
10 11 12
4
8
13 14 15 16
Shannons Information
Theory
The
Claude Shannon:
Shannon A Mathematical Theory of Communication
Bell System Technical Journal, 1948
H pi log( pi )
i 1
symbols 1, 2,
probability of occurrence of
pi
n,
each with
Shannons Entropy
a
We want to characterize the average
information generated by the source!
0.5
source
Intuition on Shannons
Entropy
Why
H pi log( pi )
i 1
Suppose you have a long random string of two binary symbols 0 and 1, and the
probability of symbols 1 and 0 are
1
0 and
Ex: 00100100101101001100001000100110001 .
If any string is long enough say N, it is likely to contain
The probability of this string pattern occurs is equal to
p p0Np0 p1Np1
Np0 Np1
Hence, # of possible patterns is 1 / p p
p1
0
# bits to represent all possible patterns is
Np0 0s
and
pi log pi
i 0
Np1 1s.
Self Information
Why?
Assume two independent events A and B, with
probabilities P(A) = pA and P(B) = pB.
For both the events to happen, the probability is
pA pB. However, the amount of information
should be added, not multiplied.
Self Information
Example 1:
Example 2:
Which logarithm? Pick the one you like! If you pick the natural log,
youll measure in nats, if you pick the 10-log, youll get Hartleys,
if you pick the 2-log (like everyone else), youll get bits.
Self Information
On average over all the symbols, we get:
Entropy
Example: Binary Memoryless Source
BMS
01101000
Let
Ofte
n
Then
den
oted
1
The uncertainty (information) is greatest when
0
0.5
Example
Three symbols a, b, c with corresponding probabilities:
P = {0.5, 0.25, 0.25}
What is H(P)?
2.
3.