EECS 554

Homework 3
Due: 10/7

1 Compression
1.1 In the lecture we stated two equivalent conditions for a prefix free (PF) code to be “full”.
Condition 1: No additional codeword can be added without destroying the prefix-free prop-
erty
Condition 2: No codeword can be shortened without destroying the prefix-free property
Show that these two conditions are not equivalent for non-binary codes. (You can con-
sider ternary codes and come up with a counterexample)
1.2 In the proof of the converse of the Kraft inequality we constructed a prefix free (PF)
code by first sorting the given codeword lengths in increasing order l1 ≤ l2 ≤ · · · ≤ lN .
Is this necessary for the construction? In other words, do you see any problem if the
construction is done without the lengths being sorted?
1.3 Show that the entropy of a DMS is a non-negative number and is zero if and only if the
source is deterministic.
∑ ∑M
1.4 Let p1 , . . . , pM and q1 , . . . , qM be arbitrary positive numbers with Mi=1 pi = i=1 qi = 1.
Show that
∑ M
1 ∑ M
1
pi log2 ≤ pi log2
i=1
pi i=1
qi
with equality if and only if pi = qi for all i.
Hint: Use the inequality ln x ≤ x − 1 for x = qi /pi .
1.5 Using the above result, show that the entropy of a DMS with an M-ary alphabet is at
most log2 M .
1.6 In the derivation in class of the bounds for the optimal average length of prefix free
codes we used a necessary condition for optimality, without showing that it is also sufficient.
Using the inequality ln x ≤ x − 1, show that H ≤ L̄ for every prefix free code. Did you use
the fact that the code is full?
1.7 A source has an alphabet {a1 , a2 , a3 , a4 } with corresponding probabilities {0.1, 0.2, 0.3, 0.4}.
1. Design a Huffman code for the source and evaluate the average length of the code.
2. One can group two consecutive symbols into symbol pairs (or super-symbols) and design
a Huffman code for the source emitting such super-symbols. There are 16 such super-
symbols. Find the probability of each such super-symbols and design a Huffman code.
What is the average codeword length? What is the average required bits per each source
letter ? Compare with the previous answer (for a fair comparison, both lengths are
expressed in bits per single letter).

1
 1.8 A source emits one of four messages randomly every 1 microsec with probabilities 0.5,
0.3, 0.1, and 0.1. Design a Huffman code for this source.

1. What is its average length? What is the average transmission rate?

2. Now suppose that we want to encode this source using not a binary (i.e., {0, 1}) but a
ternary (i.e., {0, 1, 2}) alphabet. Design a ternary Huffman code using the same ideas
as in binary codes, but with ternary trees instead of binary trees. What is the average
code length (in “trits/symbol”) and data rate (in “trits”/sec)?

1.9 Consider a DMS with finite alphabet A and probabilities p(a) for a ∈ A. For any
sequence x ∈ An and for any a ∈ A, let Nx (a) denote the average number of occurences of
the symbol a in x. Given ϵ > 0 and n, define the set of strongly typical sequences of length
n as
Tϵn = {x ∈ An : |Nx (a) − p(a)| < ϵ, ∀a ∈ A}.
def

Find a lower bound on the size |Tϵn | of this set that is asymptotically tight with the upper
bound derived in class.