Introduction to Algorithms

6.046J/18.401J/SMA5503

Lecture 9
Prof. Charles E. Leiserson
 Binary-search-tree sort
T←∅ ⊳ Create an empty BST
for i = 1 to n
do TREE-INSERT(T, A[i])
Perform an inorder tree walk of T.

Example: 33
A = [3 1 8 2 6 7 5] 11 88
Tree-walk time = O(n), 22 66
but how long does it 55 77
take to build the BST?
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.2
 Analysis of BST sort
BST sort performs the same comparisons as
quicksort, but in a different order!
3 1 8 2 6 7 5

1 2 8 6 7 5
2 675
5 7
The expected time to build the tree is asymptot-
ically the same as the running time of quicksort.
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.3
 Node depth
The depth of a node = the number of comparisons
made during TREE-INSERT. Assuming all input
permutations are equally likely, we have
Average node depth
 n 
= E ∑ (# comparisons to insert node i )
1
n  i =1 
= 1 O(n lg n) (quicksort analysis)
n
= O(lg n) .
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.4
 Expected tree height
But, average node depth of a randomly built
BST = O(lg n) does not necessarily mean that its
expected height is also O(lg n) (although it is).
Example.

≤ lg n
h= n
Ave. depth ≤ 1  n ⋅ lg n + n ⋅ n 
n 2 
= O(lg n)
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.5
 Height of a randomly built
binary search tree
Outline of the analysis:
• Prove Jensen’s inequality, which says that
f(E[X]) ≤ E[f(X)] for any convex function f and
random variable X.
• Analyze the exponential height of a randomly
built BST on n nodes, which is the random
variable Yn = 2Xn, where Xn is the random
variable denoting the height of the BST.
• Prove that 2E[Xn] ≤ E[2Xn ] = E[Yn] = O(n3),
and hence that E[Xn] = O(lg n).
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.6
 Convex functions
A function f : R → R is convex if for all
α,β ≥ 0 such that α + β = 1, we have
f(αx + βy) ≤ αf(x) + βf(y)
for all x,y ∈ R.
f
f(y)
αf(x) + βf(y)

f(x)
f(αx + βy)

x αx + βy y
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.7
 Convexity lemma
Lemma. Let f : R → R be a convex function,
and let {α1, α2 , …, αn} be a set of nonnegative
constants such that ∑k αk = 1. Then, for any set
{x1, x2, …, xn} of real numbers, we have
 n  n
f  ∑ α k xk  ≤ ∑ α k f ( xk ) .
 k =1  k =1
Proof. By induction on n. For n = 1, we have
α1 = 1, and hence f(α1x1) ≤ α1f(x1) trivially.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.8

 Proof (continued)
Inductive step:
 n   n −1 
αk
f  ∑ α k xk  = f α n xn + (1 − α n ) ∑ xk 
 k =1   k =11 − α n 
Algebra.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.9

 Proof (continued)
Inductive step:
 n   n −1 
αk
f  ∑ α k xk  = f α n xn + (1 − α n ) ∑ xk 
 k =1   k =11 − α n 
 n−1 α k 
≤ α n f ( xn ) + (1 − α n ) f  ∑ xk 
 k =11 − α n 

Convexity.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.10

 Proof (continued)
Inductive step:
 n   n −1 
αk
f  ∑ α k xk  = f α n xn + (1 − α n ) ∑ xk 
 k =1   k =11 − α n 
 n−1 α k 
≤ α n f ( xn ) + (1 − α n ) f  ∑ xk 
 k =11 − α n 
n −1
αk
≤ α n f ( xn ) + (1 − α n ) ∑ f ( xk )
k =11 − α n

Induction.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.11

 Proof (continued)
Inductive step:
 n   n −1 
αk
f  ∑ α k xk  = f α n xn + (1 − α n ) ∑ xk 
 k =1   k =11 − α n 
 n−1 α k 
≤ α n f ( xn ) + (1 − α n ) f  ∑ xk 
 k =11 − α n 
n −1
αk
≤ α n f ( xn ) + (1 − α n ) ∑ f ( xk )
k =11 − α n
n
= ∑ α k f ( xk ) . Algebra.
k =1
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.12
 Jensen’s inequality
Lemma. Let f be a convex function, and let X
be a random variable. Then, f (E[X]) ≤ E[ f (X)].
Proof.
 ∞ 
f ( E[ X ]) = f  ∑ k ⋅ Pr{ X = k}
 k =−∞ 
Definition of expectation.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.13

 Jensen’s inequality
Lemma. Let f be a convex function, and let X
be a random variable. Then, f (E[X]) ≤ E[ f (X)].
Proof.
 ∞ 
f ( E[ X ]) = f  ∑ k ⋅ Pr{ X = k}
 k =−∞ 
∞
≤ ∑ f (k ) ⋅ Pr{X = k}
k =−∞

Convexity lemma (generalized).

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.14

 Jensen’s inequality
Lemma. Let f be a convex function, and let X
be a random variable. Then, f (E[X]) ≤ E[ f (X)].
Proof.
 ∞ 
f ( E[ X ]) = f  ∑ k ⋅ Pr{ X = k}
 k =−∞ 
∞
≤ ∑ f (k ) ⋅ Pr{X = k}
k =−∞
= E[ f ( X )] .
Tricky step, but true—think about it.
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.15
 Analysis of BST height
Let Xn be the random variable denoting
the height of a randomly built binary
search tree on n nodes, and let Yn = 2Xn
be its exponential height.
If the root of the tree has rank k, then
Xn = 1 + max{Xk–1, Xn–k} ,
since each of the left and right subtrees
of the root are randomly built. Hence,
we have
Yn = 2· max{Yk–1, Yn–k} .
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.16
 Analysis (continued)
Define the indicator random variable Znk as
1 if the root has rank k,
Znk =
0 otherwise.

Thus, Pr{Znk = 1} = E[Znk] = 1/n, and

n
Yn = ∑ Z nk (2 ⋅ max{Yk −1 , Yn−k }) .
k =1

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.17

 Exponential height recurrence
n 
E [Yn ] = E ∑ Z nk (2 ⋅ max{Yk −1 , Yn−k })
k =1 
Take expectation of both sides.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.18

 Exponential height recurrence
n 
E [Yn ] = E ∑ Z nk (2 ⋅ max{Yk −1 , Yn−k })
k =1 
n
= ∑ E [Z nk (2 ⋅ max{Yk −1 , Yn−k })]
k =1

Linearity of expectation.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.19

 Exponential height recurrence
n 
E [Yn ] = E ∑ Z nk (2 ⋅ max{Yk −1 , Yn−k })
k =1 
n
= ∑ E [Z nk (2 ⋅ max{Yk −1 , Yn−k })]
k =1
n
= 2∑ E[ Z nk ] ⋅ E[max{Yk −1 , Yn−k }]
k =1
Independence of the rank of the root
from the ranks of subtree roots.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.20

 Exponential height recurrence
n 
E [Yn ] = E ∑ Z nk (2 ⋅ max{Yk −1 , Yn−k })
k =1 
n
= ∑ E [Z nk (2 ⋅ max{Yk −1 , Yn−k })]
k =1
n
= 2∑ E[ Z nk ] ⋅ E[max{Yk −1 , Yn−k }]
k =1
n
≤ 2 ∑ E[Yk −1 + Yn−k ]
n k =1
The max of two nonnegative numbers
is at most their sum, and E[Znk] = 1/n.
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.21
 Exponential height recurrence
n 
E[Yn ] = E ∑ Z nk (2 ⋅ max{Yk −1 , Yn−k })
k =1 
n
= ∑ E[Z nk (2 ⋅ max{Yk −1 , Yn−k })]
k =1
n
= 2∑ E[ Z nk ] ⋅ E[max{Yk −1 , Yn−k }]
k =1
n
≤ 2 ∑ E[Yk −1 + Yn−k ]
n k =1
n −1
= 4 ∑ E[Yk ]
Each term appears
n k =0 twice, and reindex.
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.22
 Solving the recurrence
Use substitution to n −1

show that E[Yn] ≤ cn3 E [Yn ] = 4 ∑ E[Yk ]

n k =0
for some positive
constant c, which we
can pick sufficiently
large to handle the
initial conditions.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.23

 Solving the recurrence
Use substitution to n −1

show that E[Yn] ≤ cn3 E [Yn ] = 4 ∑ E[Yk ]

n k =0
for some positive n −1
constant c, which we ≤ 4 ∑ ck 3
can pick sufficiently n k =0
large to handle the
initial conditions. Substitution.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.24

 Solving the recurrence
n −1
Use substitution to
show that E[Yn] ≤ cn3 E [Yn ] = 4 ∑ E[Yk ]
n k =0
for some positive n −1
constant c, which we ≤ 4 ∑ ck 3
can pick sufficiently n k =0
large to handle the n 3
4 c
≤ ∫ x dx
initial conditions.
n 0
Integral method.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.25

 Solving the recurrence
Use substitution to n −1

show that E[Yn] ≤ cn3 E [Yn ] = 4 ∑ E[Yk ]

n k =0
for some positive n −1
constant c, which we ≤ 4 ∑ ck 3
can pick sufficiently n k =0
large to handle the n 3
4 c
≤ ∫ x dx
initial conditions.
n 0
4 c
=   n 4
n 4
Solve the integral.
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.26
 Solving the recurrence
Use substitution to n −1

show that E[Yn] ≤ cn3 E [Yn ] = 4 ∑ E[Yk ]

n k =0
for some positive n −1
constant c, which we ≤ 4 ∑ ck 3
can pick sufficiently n k =0
large to handle the n 3
4 c
≤ ∫ x dx
initial conditions.
n 0
4 c
=    n 4
n 4
= cn3. Algebra.
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.27
 The grand finale
Putting it all together, we have
2E[Xn] ≤ E[2Xn ]
Jensen’s inequality, since
f(x) = 2x is convex.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.28

 The grand finale
Putting it all together, we have
2E[Xn] ≤ E[2Xn ]
= E[Yn]
Definition.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.29

 The grand finale
Putting it all together, we have
2E[Xn] ≤ E[2Xn ]
= E[Yn]
≤ cn3 .
What we just showed.

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.30

 The grand finale
Putting it all together, we have
2E[Xn] ≤ E[2Xn ]
= E[Yn]
≤ cn3 .
Taking the lg of both sides yields
E[Xn] ≤ 3 lg n +O(1) .

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.31

 Post mortem
Q. Does the analysis have to be this hard?

Q. Why bother with analyzing exponential

height?

Q. Why not just develop the recurrence on

Xn = 1 + max{Xk–1, Xn–k}
directly?

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.32

 Post mortem (continued)
A. The inequality
max{a, b} ≤ a + b .
provides a poor upper bound, since the RHS
approaches the LHS slowly as |a – b| increases.
The bound
max{2a, 2b} ≤ 2a + 2b
allows the RHS to approach the LHS far more
quickly as |a – b| increases. By using the
convexity of f(x) = 2x via Jensen’s inequality,
we can manipulate the sum of exponentials,
resulting in a tight analysis.
© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.33
 Thought exercises
• See what happens when you try to do the
analysis on Xn directly.
• Try to understand better why the proof
uses an exponential. Will a quadratic do?
• See if you can find a simpler argument.
(This argument is a little simpler than the
one in the book—I hope it’s correct!)

© 2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.34