Analysis of Algorithms 1

Growth Rates
Growth rates of
functions:
Linear ~ n
Quadratic ~ n
2
Cubic ~ n
3

In a log-log chart,
the slope of the line
corresponds to the
growth rate of the
function

1E+0
1E+2
1E+4
1E+6
1E+8
1E+10
1E+12
1E+14
1E+16
1E+18
1E+20
1E+22
1E+24
1E+26
1E+28
1E+30
1E+0 1E+2 1E+4 1E+6 1E+8 1E+10
n
T
(
n
)
Cubic
Quadratic
Linear
Analysis of Algorithms 2
Constant Factors
The growth rate is
not affected by
constant factors or
lower-order terms
Examples
10
2
n + 10
5
is a linear
function
10
5
n
2
+ 10
8
n is a
quadratic function

1E+0
1E+2
1E+4
1E+6
1E+8
1E+10
1E+12
1E+14
1E+16
1E+18
1E+20
1E+22
1E+24
1E+26
1E+0 1E+2 1E+4 1E+6 1E+8 1E+10
n
T
(
n
)
Quadratic
Quadratic
Linear
Linear
Analysis of Algorithms 3
Big-Oh Notation
Given functions f(n) and
g(n), we say that f(n) is
O(g(n)) if there are
positive constants
c and n
0
such that
f(n) s cg(n) for n > n
0
Example: 2n + 10 is O(n)
2n + 10 s cn
(c 2) n > 10
n > 10/(c 2)
Pick c = 3 and n
0
= 10

1
10
100
1,000
10,000
1 10 100 1,000
n
3n
2n+10
n
Analysis of Algorithms 4
Big-Oh Example
Example: the function
n
2
is not O(n)
n
2
s cn
n s c
The above inequality
cannot be satisfied
since c must be a
constant

1
10
100
1,000
10,000
100,000
1,000,000
1 10 100 1,000
n
n^2
100n
10n
n
Analysis of Algorithms 5
More Big-Oh Examples
7n-2

7n-2 is O(n)
need c > 0 and n
0
> 1 such that 7n-2 s cn for n > n
0

this is true for c = 7 and n
0
= 1


3n
3
+ 20n
2
+ 5

3n
3
+ 20n
2
+ 5 is O(n
3
)
need c > 0 and n
0
> 1 such that 3n
3
+ 20n
2
+ 5 s cn
3
for n > n
0

this is true for c = 4 and n
0
= 21
3 log n + log log n
3 log n + log log n is O(log n)
need c > 0 and n
0
> 1 such that 3 log n + log log n s clog n for n > n
0

this is true for c = 4 and n
0
= 2
Analysis of Algorithms 6
Big-Oh and Growth Rate
The big-Oh notation gives an upper bound on the
growth rate of a function
The statement f(n) is O(g(n)) means that the growth
rate of f(n) is no more than the growth rate of g(n)
We can use the big-Oh notation to rank functions
according to their growth rate
f(n) is O(g(n)) g(n) is O(f(n))
g(n) grows more Yes No
f(n) grows more No Yes
Same growth Yes Yes
Analysis of Algorithms 7
Big-Oh Rules
If is f(n) a polynomial of degree d, then f(n) is
O(n
d
), i.e.,
1. Drop lower-order terms
2. Drop constant factors
Use the smallest possible class of functions
Say 2n is O(n) instead of 2n is O(n
2
)
Use the simplest expression of the class
Say 3n + 5 is O(n) instead of 3n + 5 is O(3n)
Analysis of Algorithms 8
Relatives of Big-Oh
big-Omega
f(n) is O(g(n)) if there is a constant c > 0
and an integer constant n
0
> 1 such that
f(n) > cg(n) for n > n
0
big-Theta
f(n) is O(g(n)) if there are constants c > 0 and c > 0 and an
integer constant n
0
> 1 such that cg(n) s f(n) s cg(n) for n > n
0

little-oh
f(n) is o(g(n)) if, for any constant c > 0, there is an integer
constant n
0
> 0 such that f(n) < cg(n) for n > n
0
little-omega
f(n) is e(g(n)) if, for any constant c > 0, there is an integer
constant n
0
> 0 such that f(n) > cg(n) for n > n
0
Analysis of Algorithms 9
Intuition for Asymptotic
Notation
Big-Oh
f(n) is O(g(n)) if f(n) is asymptotically less than or equal to g(n)
big-Omega
f(n) is O(g(n)) if f(n) is asymptotically greater than or equal to g(n)

big-Theta
f(n) is O(g(n)) if f(n) is asymptotically equal to g(n)
little-oh
f(n) is o(g(n)) if f(n) is asymptotically strictly less than g(n)

little-omega
f(n) is e(g(n)) if is asymptotically strictly greater than g(n)
Analysis of Algorithms 10
Example Uses of the
Relatives of Big-Oh
f(n) is e(g(n)) if, for any constant c > 0, there is an integer constant n
0
>
0 such that f(n) > cg(n) for n > n
0
need 5n
0
2
> cn
0
given c, the n
0
that satifies this is n
0
> c/5 > 0
5n
2
is e(n)
f(n) is O(g(n)) if there is a constant c > 0 and an integer constant n
0
> 1
such that f(n) > cg(n) for n > n
0
let c = 1 and n
0
= 1

5n
2
is O(n)
f(n) is O(g(n)) if there is a constant c > 0 and an integer constant n
0
> 1
such that f(n) > cg(n) for n > n
0
let c = 5 and n
0
= 1
5n
2
is O(n
2
)
Analysis of Algorithms 11
Divide-and-Conquer
Divide-and conquer is a
general algorithm design
paradigm:
Divide: divide the input data S in
two or more disjoint subsets S
1
,
S
2
,
Recur: solve the subproblems
recursively
Conquer: combine the solutions
for S
1
, S
2
, , into a solution for S
The base case for the
recursion are subproblems of
constant size
Analysis can be done using
recurrence equations
Analysis of Algorithms 12
Merge-Sort Review
Merge-sort on an input
sequence S with n
elements consists of
three steps:
Divide: partition S into
two sequences S
1
and S
2

of about n/2 elements
each
Recur: recursively sort S
1

and S
2
Conquer: merge S
1
and
S
2
into a unique sorted
sequence
Algorithm mergeSort(S, C)
Input sequence S with n
elements, comparator C
Output sequence S sorted
according to C
if S.size() > 1
(S
1
, S
2
) partition(S, n/2)
mergeSort(S
1
, C)
mergeSort(S
2
, C)
S merge(S
1
, S
2
)
Analysis of Algorithms 13
Recurrence Equation
Analysis
The conquer step of merge-sort consists of merging two sorted
sequences, each with n/2 elements and implemented by means of
a doubly linked list, takes at most bn steps, for some constant b.
Likewise, the basis case (n < 2) will take at b most steps.
Therefore, if we let T(n) denote the running time of merge-sort:





We can therefore analyze the running time of merge-sort by
finding a closed form solution to the above equation.
That is, a solution that has T(n) only on the left-hand side.

> +
<
=
2 if ) 2 / ( 2
2 if
) (
n bn n T
n b
n T
Analysis of Algorithms 14
Iterative Substitution
In the iterative substitution, or plug-and-chug, technique, we
iteratively apply the recurrence equation to itself and see if we can
find a pattern:








Note that base, T(n)=b, case occurs when 2
i
=n. That is, i = log n.
So,

Thus, T(n) is O(n log n).
ibn n T
bn n T
bn n T
bn n T
bn n b n T
bn n T n T
i i
+ =
=
+ =
+ =
+ =
+ + =
+ =
) 2 / ( 2
...
4 ) 2 / ( 2
3 ) 2 / ( 2
2 ) 2 / ( 2
)) 2 / ( )) 2 / ( 2 ( 2
) 2 / ( 2 ) (
4 4
3 3
2 2
2
n bn bn n T log ) ( + =
COT 5407 15
Solving Recurrences by
Substitution: Guess-and-Test
Guess the form of the solution
(Using mathematical induction) find the constants and
show that the solution works
Example
T(n) = 2T(n/2) + n
Guess (#1) T(n) = O(n)
Need T(n) <= cn for some constant c>0
Assume T(n/2) <= cn/2 Inductive hypothesis
Thus T(n) <= 2cn/2 + n = (c+1) n
Our guess was wrong!!
COT 5407 16
Solving Recurrences by
Substitution: 2
T(n) = 2T(n/2) + n
Guess (#2) T(n) = O(n
2
)
Need T(n) <= cn
2
for some constant c>0
Assume T(n/2) <= cn
2
/4 Inductive hypothesis
Thus T(n) <= 2cn
2
/4 + n = cn
2
/2+ n
Works for all n as long as c>=2 !!
But there is a lot of slack
9/9/08 COT 5407 17
Solving Recurrences by
Substitution: 3
T(n) = 2T(n/2) + n
Guess (#3) T(n) = O(nlogn)
Need T(n) <= cnlogn for some constant c>0
Assume T(n/2) <= c(n/2)(log(n/2)) Inductive hypothesis
Thus T(n) <= 2 c(n/2)(log(n/2)) + n
<= cnlogn -cn + n <= cnlogn
Works for all n as long as c>=1 !!
This is the correct guess. WHY?
Show T(n) >= cnlogn for some constant c>0
Analysis of Algorithms 18
More Examples
In the guess-and-test method, we guess a closed form solution
and then try to prove it is true by induction:


Guess: T(n) < cn log n.






Wrong: we cannot make this last line be less than cn log n








n bn cn n cn
n bn n cn
n bn n n c
n bn n T n T
log log
log ) 2 log (log
log )) 2 / log( ) 2 / ( ( 2
log ) 2 / ( 2 ) (
+ =
+ =
+ =
+ =

> +
<
=
2 if log ) 2 / ( 2
2 if
) (
n n bn n T
n b
n T
Analysis of Algorithms 19
More Examples
Recall the recurrence equation:


Guess #2: T(n) < cn log
2
n.






if c > b.
So, T(n) is O(n log
2
n).
In general, to use this method, you need to have a good guess
and you need to be good at induction proofs.
n cn
n bn cn n cn n cn
n bn n cn
n bn n n c
n bn n T n T
2
2
2
2
log
log log 2 log
log ) 2 log (log
log )) 2 / ( log ) 2 / ( ( 2
log ) 2 / ( 2 ) (
s
+ + =
+ =
+ =
+ =

> +
<
=
2 if log ) 2 / ( 2
2 if
) (
n n bn n T
n b
n T
COT 5407 20
Solving Recurrences: Recursion-tree
method
Substitution method fails when a good guess is not available
Recursion-tree method works in those cases
Write down the recurrence as a tree with recursive calls as the
children
Expand the children
Add up each level
Sum up the levels
Useful for analyzing divide-and-conquer algorithms
Also useful for generating good guesses to be used by substitution
method
9/9/08 COT 5407 21
Copyright The McGraw-Hill Companies, Inc. Permission required for reproduction or display.

9/9/08 COT 5407 22
Copyright The McGraw-Hill Companies, Inc. Permission required for reproduction or display.

Analysis of Algorithms 23
Master Method
Many divide-and-conquer recurrence equations have
the form:



The Master Theorem:

> +
<
=
d n n f b n aT
d n c
n T
if ) ( ) / (
if
) (
. 1 some for ) ( ) / ( provided
)), ( ( is ) ( then ), ( is ) ( if 3.
) log ( is ) ( then ), log ( is ) ( if 2.
) ( is ) ( then ), ( is ) ( if 1.
log
1 log log
log log
< s
O O
O O
O
+
+

o o
c
c
n f b n af
n f n T n n f
n n n T n n n f
n n T n O n f
a
k a k a
a a
b
b b
b b