A Gentle Introduction to Algorithm Complexity Analysis

Dionysis "dionyziz" Zindros <dionyziz@gmail.com

<mailto:dionyziz@gmail.com>>
* English <http://discrete.gr/complexity/?en>
* < ttp://d
screte.gr/compex
ty/?e>
* < ttp://d
screte.gr/compex
ty/?m>
* spao < ttp://d
screte.gr/compex
ty/?es>
* Hep trasate <ma
to:d
oyz
z@gma
.com>

Itroduct
o
A ot of programmers t at mae some of t e cooest ad most usefu
software today, suc as may of t e stuff we see o t e Iteret or use
da
y, do't ave a t eoret
ca computer sc
ece bacgroud. T ey're
st
 pretty awesome ad creat
ve programmers ad we t a t em for w at
t ey bu
d.
However, t eoret
ca computer sc
ece as
ts uses ad app
cat
os ad
ca tur out to be qu
te pract
ca. I t
s art
ce, targeted at
programmers w o ow t e
r art but w o do't ave ay t eoret
ca
computer sc
ece bacgroud, I w
 preset oe of t e most pragmat
c
toos of computer sc
ece: B
g O otat
o ad agor
t m compex
ty
aays
s. As someoe w o as wored bot
 a computer sc
ece academ
c
sett
g ad
 bu
d
g product
o-eve software
 t e
dustry, t
s

s t e too I ave foud to be oe of t e truy usefu oes
 pract
ce,
so I ope after read
g t
s art
ce you ca appy
t
 your ow code
to mae
t better. After read
g t
s post, you s oud be abe to
uderstad a t e commo terms computer sc
et
sts use suc as "b
g O",
"asymptot
c be av
or" ad "worst-case aays
s".
T
s text
s aso targeted at t e ju
or
g sc oo ad
g sc oo
studets from Greece or ayw ere ese
terat
oay compet
g
 t e
Iterat
oa Oymp
ad
 Iformat
cs
< ttp://e.w

ped
a.org/w

/Iterat
oa_Oymp
ad_
_Iformat
cs>, a
agor
t ms compet
t
o for studets, or ot er s
m
ar compet
t
os. As
suc ,
t does ot ave ay mat emat
ca prerequ
s
tes ad w
 g
ve you
t e bacgroud you eed
 order to cot
ue study
g agor
t ms w
t a
f
rmer uderstad
g of t e t eory be
d t em. As someoe w o used to
compete
 t ese studet compet
t
os, I
g y adv
se you to read
t roug t
s w oe
troductory mater
a ad try to fuy uderstad
t,
because
t w
 be ecessary as you study agor
t ms ad ear more
advaced tec 
ques.
I be
eve t
s text w
 be epfu for
dustry programmers w o do't
ave too muc exper
ece w
t t eoret
ca computer sc
ece (
t
s a fact
t at some of t e most
sp
r
g software eg
eers ever wet to
coege). But because
t's aso for studets,
t may at t
mes soud a

tte b
t 
e a textboo. I add
t
o, some of t e top
cs
 t
s text
may seem too obv
ous to you; for exampe, you may ave see t em dur
g
your
g sc oo years. If you fee you uderstad t em, you ca s
p
t em. Ot er sect
os go
to a b
t more dept ad become s
g ty
t eoret
ca, as t e studets compet
g
 t
s compet
t
o eed to ow
more about t eoret
ca agor
t ms t a t e average pract
t
oer. But
t ese t
gs are st
 good to ow ad ot tremedousy ard to foow,
so
t's 
ey we wort your t
me. As t e or
g
a text was targeted
at
g sc oo studets, o mat emat
ca bacgroud
s requ
red, so
ayoe w
t some programm
g exper
ece (
.e.
f you ow w at recurs
o

s) w
 be abe to foow t roug w
t out ay probem.
T roug out t
s art
ce, you w
 f
d var
ous po
ters t at 
 you to

terest
g mater
a ofte outs
de t e scope of t e top
c uder
d
scuss
o. If you're a
dustry programmer,
t's 
ey t at you're
fam

ar w
t most of t ese cocepts. If you're a ju
or studet
part
c
pat
g
 compet
t
os, foow
g t ose 
s w
 g
ve you cues
about ot er areas of computer sc
ece or software eg
eer
g t at you
may ot ave yet expored w
c you ca oo at to broade your
terests.
B
g O otat
o ad agor
t m compex
ty aays
s
s somet
g a ot of

dustry programmers ad ju
or studets a
e f
d ard to uderstad,
fear, or avo
d atoget er as useess. But
t's ot as ard or as
t eoret
ca as
t may seem at f
rst. Agor
t m compex
ty
s just a way
to formay measure ow fast a program or agor
t m rus, so
t reay

s qu
te pragmat
c. Let's start by mot
vat
g t e top
c a 
tte b
t.
A screes ot of a art
f
c
a
te
gece c aracter
 Haf-
fe 2
*F
gure 1*: Art
f
c
a
te
gece c aracters
 v
deo games use
agor
t ms to avo
d obstaces w e av
gat
g
 t e v
rtua word

Mot
vat
o
We aready ow t ere are toos to measure ow fast a program rus.
T ere are programs caed /prof
ers/ w
c measure ru
g t
me

m

secods ad ca ep us opt
m
ze our code by spott
g botteecs.
W
e t
s
s a usefu too,
t
s't reay reevat to agor
t m
compex
ty. Agor
t m compex
ty
s somet
g des
ged to compare two
agor
t ms at t e
dea eve
gor
g ow-eve deta
s suc as t e

mpemetat
o programm
g aguage, t e ardware t e agor
t m rus o,
or t e
struct
o set of t e g
ve CPU. We wat to compare agor
t ms

 terms of just w at t ey are: Ideas of ow somet
g
s computed.
Cout
g m

secods wo't ep us
 t at. It's qu
te poss
be t at a
bad agor
t m wr
tte
 a ow-eve programm
g aguage suc as
assemby < ttp://e.w

ped
a.org/w

/Assemby_aguage> rus muc
qu
cer t a a good agor
t m wr
tte
 a
g -eve programm
g
aguage suc as Pyt o < ttp://www.pyt o.org/> or Ruby
< ttp://www.ruby-ag.org/e/>. So
t's t
me to def
e w at a "better
agor
t m" reay
s.
As agor
t ms are programs t at perform just a computat
o, ad ot
ot er t
gs computers ofte do suc as etwor
g tass or user
put
ad output, compex
ty aays
s aows us to measure ow fast a program

s w e
t performs computat
os. xampes of operat
os t at are purey
/computat
oa/
cude umer
ca foat
g-po
t operat
os
< ttp://e.w

ped
a.org/w

/Foat
g_po
t> suc as add
t
o ad
mut
p
cat
o; searc
g w
t
 a database t at f
ts
 RAM for a g
ve
vaue; determ

g t e pat a art
f
c
a-
te
gece c aracter w

wa t roug
 a v
deo game so t at t ey oy ave to wa a s ort
d
stace w
t
 t e
r v
rtua word (see *F
gure 1*); or ru
g a
reguar express
o < ttp://www.reguar-express
os.
fo/> patter matc
o a str
g. Ceary, computat
o
s ub
qu
tous
 computer programs.
Compex
ty aays
s
s aso a too t at aows us to expa
 ow a
agor
t m be aves as t e
put grows arger. If we feed
t a d
fferet

put, ow w
 t e agor
t m be ave? If our agor
t m taes 1 secod to
ru for a
put of s
ze 1000, ow w

t be ave
f I doube t e
put
s
ze? W

t ru just as fast, af as fast, or four t
mes sower? I
pract
ca programm
g, t
s
s
mportat as
t aows us to pred
ct ow
our agor
t m w
 be ave w e t e
put data becomes arger. For
exampe,
f we've made a agor
t m for a web app
cat
o t at wors
we w
t 1000 users ad measure
ts ru
g t
me, us
g agor
t m
compex
ty aays
s we ca ave a pretty good
dea of w at w
 appe
oce we get 2000 users
stead. For agor
t m
c compet
t
os, compex
ty
aays
s g
ves us
s
g t about ow og our code w
 ru for t e
argest testcases t at are used to test our program's correctess. So
f
we've measured our program's be av
or for a sma
put, we ca get a
good
dea of ow
t w
 be ave for arger
puts. Let's start by a
s
mpe exampe: F
d
g t e max
mum eemet
 a array.

Cout
g
struct
os
I t
s art
ce, I' use var
ous programm
g aguages for t e
exampes. However, do't despa
r
f you do't ow a part
cuar
programm
g aguage. S
ce you ow programm
g, you s oud be abe to
read t e exampes w
t out ay probem eve
f you are't fam

ar w
t
t e programm
g aguage of c o
ce, as t ey w
 be s
mpe ad I wo't
use ay esoter
c aguage features. If you're a studet compet
g

agor
t ms compet
t
os, you most 
ey wor w
t C++
< ttp://www.cpuspus.com/doc/tutor
a/>, so you s oud ave o probem
foow
g t roug . I t at case I recommed wor
g o t e exerc
ses
us
g C++ for pract
ce.
T e max
mum eemet
 a array ca be ooed up us
g a s
mpe p
ece of
code suc as t
s p
ece of Javascr
pt
< ttp://www.qu
rsmode.org/js/
tro. tm> code. G
ve a
put array A
of s
ze :
|1.||var| |M = A[ 0 ];|
|2.|
|3.||for| |( ||var| |
= 0;
< ; ++
) {|
|4.|| ||
f| |( A[
] >= M ) {|
|5.|| ||M = A[
];|
|6.|| ||}|
|7.||}|
Now, t e f
rst t
g we' do
s cout ow may /fudameta

struct
os/ t
s p
ece of code executes. We w
 oy do t
s oce ad

t wo't be ecessary as we deveop our t eory, so bear w
t me for a
few momets as we do t
s. As we aayze t
s p
ece of code, we wat to
brea
t up
to s
mpe
struct
os; t
gs t at ca be executed by t e
CPU d
recty - or cose to t at. We' assume our processor ca execute
t e foow
g operat
os as oe
struct
o eac :
* Ass
g
g a vaue to a var
abe
* Loo
g up t e vaue of a part
cuar eemet
 a array
* Compar
g two vaues
* Icremet
g a vaue
* Bas
c ar
t met
c operat
os suc as add
t
o ad mut
p
cat
o
We' assume brac
g (t e c o
ce betwee |
f| ad |ese| parts of code
after t e |
f| cod
t
o as bee evauated) occurs
staty ad wo't
cout t ese
struct
os. I t e above code, t e f
rst 
e of code
s:
|1.||var| |M = A[ 0 ];|
T
s requ
res 2
struct
os: Oe for oo
g up A[ 0 ] ad oe for
ass
g
g t e vaue to M (we're assum
g t at 
s aways at east 1).
T ese two
struct
os are aways requ
red by t e agor
t m, regardess
of t e vaue of . T e |for| oop

t
a
zat
o code aso as to aways
ru. T
s g
ves us two more
struct
os; a ass
gmet ad a compar
so:
|1.||
= 0;|
|2.||
< ;|
T ese w
 ru before t e f
rst |for| oop
terat
o. After eac |for|
oop
terat
o, we eed two more
struct
os to ru, a
cremet of

ad a compar
so to c ec
f we' stay
 t e oop:
|1.||++
;|
|2.||
< ;|
So,
f we
gore t e oop body, t e umber of
struct
os t
s
agor
t m eeds
s 4 + 2. T at
s, 4
struct
os at t e beg

g of
t e |for| oop ad 2
struct
os at t e ed of eac
terat
o of w
c
we ave . We ca ow def
e a mat emat
ca fuct
o f(  ) t at, g
ve
a , g
ves us t e umber of
struct
os t e agor
t m eeds. For a
empty |for| body, we ave f(  ) = 4 + 2.

Worst-case aays
s
Now, oo
g at t e |for| body, we ave a array ooup operat
o ad a
compar
so t at appe aways:
|1.||
f| |( A[
] >= M ) { ...|
T at's two
struct
os r
g t t ere. But t e |
f| body may ru or may
ot ru, deped
g o w at t e array vaues actuay are. If
t appes
to be so t at |A[
] >= M|, t e we' ru t ese two add
t
oa

struct
os a array ooup ad a ass
gmet:
|1.||M = A[
]|
But ow we ca't def
e a f(  ) as eas
y, because our umber of

struct
os does't deped soey o  but aso o our
put. For
exampe, for |A = [ 1, 2, 3, 4 ]| t e agor
t m w
 eed more

struct
os t a for |A = [ 4, 3, 2, 1 ]|. W e aayz
g agor
t ms,
we ofte cos
der t e worst-case scear
o. W at's t e worst t at ca
appe for our agor
t m? W e does our agor
t m eed t e most

struct
os to compete? I t
s case,
t
s w e we ave a array


creas
g order suc as |A = [ 1, 2, 3, 4 ]|. I t at case, M eeds to
be repaced every s
ge t
me ad so t at y
eds t e most
struct
os.
Computer sc
et
sts ave a facy ame for t at ad t ey ca
t
/worst-case aays
s/; t at's ot
g more t a just cos
der
g t e
case w e we're t e most uucy. So,
 t e worst case, we ave 4

struct
os to ru w
t
 t e |for| body, so we ave f(  ) = 4 + 2 +
4 = 6 + 4. T
s fuct
o f, g
ve a probem s
ze , g
ves us t e
umber of
struct
os t at woud be eeded
 t e worst-case.

Asymptot
c be av
or
G
ve suc a fuct
o, we ave a pretty good
dea of ow fast a
agor
t m
s. However, as I prom
sed, we wo't be eed
g to go t roug
t e ted
ous tas of cout
g
struct
os
 our program. Bes
des, t e
umber of actua CPU
struct
os eeded for eac programm
g aguage
statemet depeds o t e comp
er of our programm
g aguage ad o t e
ava
abe CPU
struct
o set (
.e. w et er
t's a AMD or a Ite
Pet
um o your PC, or a MIPS processor o your Paystat
o 2) ad we
sa
d we'd be
gor
g t at. We' ow ru our "f" fuct
o t roug a
"f
ter" w
c w
 ep us get r
d of t ose m
or deta
s t at computer
sc
et
sts prefer to
gore.
I our fuct
o, 6 + 4, we ave two terms: 6 ad 4. I compex
ty
aays
s we oy care about w at appes to t e
struct
o-cout
g
fuct
o as t e program
put () grows arge. T
s reay goes aog
w
t t e prev
ous
deas of "worst-case scear
o" be av
or: We're

terested
 ow our agor
t m be aves w e treated bady; w e
t's
c aeged to do somet
g ard. Not
ce t at t
s
s reay usefu w e
compar
g agor
t ms. If a agor
t m beats aot er agor
t m for a
arge
put,
t's most probaby true t at t e faster agor
t m rema
s
faster w e g
ve a eas
er, smaer
put. *From t e terms t at we are
cos
der
g, we' drop a t e terms t at grow sowy ad oy eep t e
oes t at grow fast as  becomes arger.* Ceary 4 rema
s a 4 as 
grows arger, but 6 grows arger ad arger, so
t teds to matter more
ad more for arger probems. T erefore, t e f
rst t
g we w
 do
s
drop t e 4 ad eep t e fuct
o as f(  ) = 6.
T
s maes sese
f you t
 about
t, as t e 4
s s
mpy a
"

t
a
zat
o costat". D
fferet programm
g aguages may requ
re a
d
fferet t
me to set up. For exampe, Java eeds some t
me to


t
a
ze
ts v
rtua mac
e
< ttp://e.w

ped
a.org/w

/Java_v
rtua_mac
e>. S
ce we're

gor
g programm
g aguage d
ffereces,
t oy maes sese to
gore
t
s vaue.
T e secod t
g we'
gore
s t e costat mut
p
er
 frot of ,
ad so our fuct
o w
 become f(  ) = . As you ca see t
s
s
mp
f
es t
gs qu
te a ot. Aga
,
t maes some sese to drop t
s
mut
p
cat
ve costat
f we t
 about ow d
fferet programm
g
aguages comp
e. T e "array ooup" statemet
 oe aguage may
comp
e to d
fferet
struct
os
 d
fferet programm
g aguages.
For exampe,
 C, do
g |A[
]| does ot
cude a c ec t at

s
w
t
 t e decared array s
ze, w
e
 Pasca
< ttp://e.w

ped
a.org/w

/Pasca_%28programm
g_aguage%29>
t
does. So, t e foow
g Pasca code:
|1.||M := A[
]|
Is t e equ
vaet of t e foow
g
 C:
|1.||
f| |(
>= 0 &&
<  ) {|
|2.|| ||M = A[
];|
|3.||}|
So
t's reasoabe to expect t at d
fferet programm
g aguages w

y
ed d
fferet factors w e we cout t e
r
struct
os. I our exampe

 w
c we are us
g a dumb comp
er for Pasca t at
s ob
v
ous of
poss
be opt
m
zat
os, Pasca requ
res 3
struct
os for eac array
access
stead of t e 1
struct
o C requ
res. Dropp
g t
s factor
goes aog t e 
es of
gor
g t e d
ffereces betwee part
cuar
programm
g aguages ad comp
ers ad oy aayz
g t e
dea of t e
agor
t m
tsef.
T
s f
ter of "dropp
g a factors" ad of "eep
g t e argest
grow
g term" as descr
bed above
s w at we ca /asymptot
c be av
or/.
So t e asymptot
c be av
or of f(  ) = 2 + 8
s descr
bed by t e
fuct
o f(  ) = . Mat emat
cay spea
g, w at we're say
g ere
s
t at we're
terested
 t e 
m
t of fuct
o f as  teds to
f

ty;
but
f you do't uderstad w at t at p rase formay meas, do't
worry, because t
s
s a you eed to ow. (O a s
de ote,
 a
str
ct mat emat
ca sett
g, we woud ot be abe to drop t e costats

 t e 
m
t; but for computer sc
ece purposes, we wat to do t at for
t e reasos descr
bed above.) Let's wor a coupe of exampes to
fam

ar
ze ourseves w
t t e cocept.
T e cub
c fuct
o,
 bue, overcomes t e 
ear fuct
o,
 red,
after  = 45
*F
gure 2*: T e ^3 fuct
o, draw
 bue, becomes arger t a t e
1999 fuct
o, draw
 red, after  = 45. After t at po
t
t rema
s
arger for ever.
Let us f
d t e asymptot
c be av
or of t e foow
g exampe fuct
os
by dropp
g t e costat factors ad by eep
g t e terms t at grow t e
fastest.
1.
f(  ) = 5 + 12 g
ves f(  ) = .
By us
g t e exact same reaso
g as above.
2.
f(  ) = 109 g
ves f(  ) = 1.
We're dropp
g t e mut
p
er 109 * 1, but we st
 ave to put a 1
ere to
d
cate t at t
s fuct
o as a o-zero vaue.
3.
f(  ) = ^2 + 3 + 112 g
ves f(  ) = ^2
Here, ^2 grows arger t a 3 for suff
c
ety arge , so we're
eep
g t at.

4.
f(  ) = ^3 + 1999 + 1337 g
ves f(  ) = ^3
ve t oug t e factor
 frot of 
s qu
te arge, we ca st

f
d a arge eoug  so t at ^3
s b
gger t a 1999. As we're

terested
 t e be av
or for very arge vaues of , we oy eep
^3 (See *F
gure 2*).
5.
f(  ) =  + sqrt(  ) g
ves f(  ) = 
T
s
s so because  grows faster t a sqrt(  ) as we
crease .
You ca try out t e foow
g exampes o your ow:

xerc
se 1
 1. f(  ) = ^6 + 3
2. f(  ) = 2^ + 12
3. f(  ) = 3^ + 2^
4. f(  ) = ^ + 
(Wr
te dow your resuts; t e sout
o
s g
ve beow)
If you're av
g troube w
t oe of t e above, pug
 some arge  ad
see w
c term
s b
gger. Pretty stra
g tforward, u ?

Compex
ty
So w at t
s
s te
g us
s t at s
ce we ca drop a t ese
decorat
ve costats,
t's pretty easy to te t e asymptot
c be av
or
of t e
struct
o-cout
g fuct
o of a program. I fact, ay program
t at does't ave ay oops w
 ave f(  ) = 1, s
ce t e umber of

struct
os
t eeds
s just a costat (uess
t uses recurs
o; see
beow). Ay program w
t a s
ge oop w
c goes from 1 to  w
 ave
f(  ) = , s
ce
t w
 do a costat umber of
struct
os before
t e oop, a costat umber of
struct
os after t e oop, ad a
costat umber of
struct
os w
t
 t e oop w
c a ru  t
mes.
T
s s oud ow be muc eas
er ad ess ted
ous t a cout
g
d
v
dua

struct
os, so et's tae a oo at a coupe of exampes to get
fam

ar w
t t
s. T e foow
g PHP < ttp://p p.et/> program c ecs
to see
f a part
cuar vaue ex
sts w
t
 a array A of s
ze :
|01.||<?p p|
|02.|| ||$ex
sts| |= fase;|
|03.|| ||for| |( ||$
| |= 0; ||$
| |< ; ++||$
| |) {|
|04.|| ||
f| |( ||$A||[ ||$
| |] == ||$vaue| |) {|
|05.|| ||$ex
sts| |= true;|
|06.|| ||brea||;|
|07.|| ||}|
|08.|| ||}|
|09.||?>|
T
s met od of searc
g for a vaue w
t
 a array
s caed /
ear
searc /. T
s
s a reasoabe ame, as t
s program as f(  ) = 
(we' def
e exacty w at "
ear" meas
 t e ext sect
o). You may
ot
ce t at t ere's a "brea" statemet ere t at may mae t e program
term
ate sooer, eve after a s
ge
terat
o. But reca t at we're

terested
 t e worst-case scear
o, w
c for t
s program
s for t e
array A to ot cota
 t e vaue. So we st
 ave f(  ) = .

xerc
se 2

Systemat
cay aayze t e umber of
struct
os t e above PHP program

eeds w
t respect to 
 t e worst-case to f
d f(  ), s
m
ary to
ow we aayzed our f
rst Javascr
pt program. T e ver
fy t at,
asymptot
cay, we ave f(  ) = .
Let's oo at a Pyt o program w
c adds two array eemets toget er to
produce a sum w
c
t stores
 aot er var
abe:
|1.||v ||=| |a[ ||0| |] ||+| |a[ ||1| |]|
Here we ave a costat umber of
struct
os, so we ave f(  ) = 1.
T e foow
g program
 C++ c ecs to see
f a vector (a facy array)
amed A of s
ze  cota
s t e same two vaues ayw ere w
t

t:
|01.||boo| |dup
cate = ||fase||;|
|02.||for| |( ||
t| |
= 0;
< ; ++
) {|
|03.|| ||for| |( ||
t| |j = 0; j < ; ++j ) {|
|04.|| ||
f| |(
!= j && A[
] == A[ j ] ) {|
|05.|| ||dup
cate = ||true||;|
|06.|| ||brea||;|
|07.|| ||}|
|08.|| ||}|
|09.|| ||
f| |( dup
cate ) {|
|10.|| ||brea||;|
|11.|| ||}|
|12.||}|
As ere we ave two ested oops w
t
 eac ot er, we' ave a
asymptot
c be av
or descr
bed by f(  ) = ^2 .
*Rue of t umb*: S
mpe programs ca be aayzed by cout
g t e ested
oops of t e program. A s
ge oop over 
tems y
eds f(  ) = . A
oop w
t
 a oop y
eds f(  ) = ^2 . A oop w
t
 a oop w
t
 a
oop y
eds f(  ) = ^3 .
If we ave a program t at cas a fuct
o w
t
 a oop ad we ow t e
umber of
struct
os t e caed fuct
o performs,
t's easy to
determ
e t e umber of
struct
os of t e w oe program. Ideed, et's
tae a oo at t
s C exampe:
|1.||
t| |
;|
|2.||for| |(
= 0;
< ; ++
) {|
|3.|| ||f(  );|
|4.||}|
If we ow t at |f(  )|
s a fuct
o t at performs exacty 

struct
os, we ca t e ow t at t e umber of
struct
os of t e
w oe program
s asymptot
cay ^2 , as t e fuct
o
s caed exacty
 t
mes.
*Rue of t umb*: G
ve a ser
es of for oops t at are sequet
a, t e
sowest of t em determ
es t e asymptot
c be av
or of t e program. Two
ested oops foowed by a s
ge oop
s asymptot
cay t e same as t e
ested oops aoe, because t e ested oops /dom
ate/ t e s
mpe oop.
Now, et's sw
tc over to t e facy otat
o t at computer sc
et
sts
use. W e we've f
gured out t e exact suc f asymptot
cay, we' say
t at our program
s ( f(  ) ). For exampe, t e above programs are (
1 ), ( ^2 ) ad ( ^2 ) respect
vey. (  )
s proouced "t eta of
". Somet
mes we say t at f(  ), t e or
g
a fuct
o cout
g t e

struct
os
cud
g t e costats,
s ( somet
g ). For exampe, we
may say t at f(  ) = 2
s a fuct
o t at
s (  ) ot
g ew
ere. We ca aso wr
te 2 (  ), w
c
s proouced as "two 
s
t eta of ". Do't get cofused about t
s otat
o: A
t's say
g
s
t at
f we've couted t e umber of
struct
os a program eeds ad
t ose are 2, t e t e asymptot
c be av
or of our agor
t m
s descr
bed
by , w
c we foud by dropp
g t e costats. G
ve t
s otat
o, t e
foow
g are some true mat emat
ca statemets:
1. ^6 + 3 ( ^6 )
 2. 2^ + 12 ( 2^ )
3. 3^ + 2^ ( 3^ )
4. ^ +  ( ^ )
By t e way,
f you soved xerc
se 1 from above, t ese are exacty t e
aswers you s oud ave foud.
*We ca t
s fuct
o,
.e. w at we put w
t
 ( ere ), t e /t
me
compex
ty/ or just /compex
ty/ of our agor
t m.* So a agor
t m w
t
(  )
s of compex
ty . We aso ave spec
a ames for ( 1 ), ( 
), ( ^2 ) ad ( og(  ) ) because t ey occur very ofte. We say t at
a ( 1 ) agor
t m
s a /costat-t
me agor
t m/, (  )
s /
ear/,
( ^2 )
s /quadrat
c/ ad ( og(  ) )
s /ogar
t m
c/ (do't worry

f you do't ow w at ogar
t ms are yet we' get to t at
 a m
ute).
*Rue of t umb*: Programs w
t a b
gger ru sower t a programs w
t
a smaer .
A exampe of surfaces
dde
 a v
deo game
*F
gure 3*: A payer t at
s ocated
 t e yeow dot w
 ot see t e
s adowed areas. Sp
tt
g t e word
 sma fragmets ad sort
g t em
by t e
r d
stace to t e payer
s oe way to sove t e v
s
b

ty probem.

B
g-O otat
o
Now,
t's somet
mes true t at
t w
 be ard to f
gure out exacty t e
be av
or of a agor
t m
 t
s fas
o as we d
d above, espec
ay for
more compex exampes. However, we w
 be abe to say t at t e be av
or
of our agor
t m w
 ever exceed a certa
 boud. T
s w
 mae 
fe
eas
er for us, as we wo't ave to spec
fy exacty ow fast our
agor
t m rus, eve w e
gor
g costats t e way we d
d before. A
we' ave to do
s f
d a certa
 boud. T
s
s expa
ed eas
y w
t
a exampe.
A famous probem computer sc
et
sts use for teac
g agor
t ms
s t e
/sort
g probem/. I t e sort
g probem, a array A of s
ze 
s g
ve
(souds fam

ar?) ad we are ased to wr
te a program t at sorts t
s
array. T
s probem
s
terest
g because
t
s a pragmat
c probem

rea systems. For exampe, a f
e exporer eeds to sort t e f
es
t
d
spays by ame so t at t e user ca av
gate t em w
t ease. Or, as
aot er exampe, a v
deo game may eed to sort t e 3D objects d
spayed

 t e word based o t e
r d
stace from t e payer's eye
s
de t e
v
rtua word
 order to determ
e w at
s v
s
be ad w at
s't,
somet
g caed t e V
s
b

ty Probem
< ttp://e.w

ped
a.org/w

/H
dde_surface_determ
at
o> (see *F
gure
3*). T e objects t at tur out to be cosest to t e payer are t ose
v
s
be, w
e t ose t at are furt er may get
dde by t e objects

frot of t em. Sort
g
s aso
terest
g because t ere are may
agor
t ms to sove
t, some of w
c are worse t a ot ers. It's aso
a easy probem to def
e ad to expa
. So et's wr
te a p
ece of code
t at sorts a array.
Here
s a
eff
c
et way to
mpemet sort
g a array
 Ruby. (Of
course, Ruby supports sort
g arrays us
g bu
d-
 fuct
os w
c you
s oud use
stead, ad w
c are certa
y faster t a w at we' see
ere. But t
s
s ere for
ustrat
o purposes.)
|01.||b = []|
|02.||.t
mes ||do|
|03.|| ||m = a[ ||0| |]|
|04.|| ||m
= ||0|
|05.|| ||a.eac _w
t _
dex ||do| ||eemet,
||
|06.|| ||
f| |eemet < m|
|07.|| ||m = eemet|
|08.|| ||m
=
|
|09.|| ||ed|
|10.|| ||ed|
|11.|| ||a.deete_at( m
)|
|12.|| ||b << m|
|13.||ed|
T
s met od
s caed seect
o sort
< ttp://e.w

ped
a.org/w

/Seect
o_sort>. It f
ds t e m

mum of
our array (t e array
s deoted a above, w
e t e m

mum vaue
s
deoted m ad m

s
ts
dex), puts
t at t e ed of a ew array (

our case b), ad removes
t from t e or
g
a array. T e
t f
ds t e
m

mum betwee t e rema

g vaues of our or
g
a array, appeds t at
to our ew array so t at
t ow cota
s two eemets, ad removes
t
from our or
g
a array. It cot
ues t
s process ut
 a
tems ave
bee removed from t e or
g
a ad ave bee
serted
to t e ew
array, w
c meas t at t e array as bee sorted. I t
s exampe, we
ca see t at we ave two ested oops. T e outer oop rus  t
mes, ad
t e
er oop rus oce for eac eemet of t e array a. W
e t e
array a

t
ay as 
tems, we remove oe array
tem
 eac

terat
o. So t e
er oop repeats  t
mes dur
g t e f
rst
terat
o
of t e outer oop, t e | - 1| t
mes, t e | - 2| t
mes ad so fort ,
ut
 t e ast
terat
o of t e outer oop dur
g w
c
t oy rus oce.
It's a 
tte arder to evauate t e compex
ty of t
s program, as we'd
ave to f
gure out t e sum 1 + 2 + ... + ( - 1) + . But we ca for
sure f
d a "upper boud" for
t. T at
s, we ca ater our program
(you ca do t at
 your m
d, ot
 t e actua code) to mae
t
*worse* t a
t
s ad t e f
d t e compex
ty of t at ew program t at
we der
ved. If we ca f
d t e compex
ty of t e worse program t at
we've costructed, t e we ow t at our or
g
a program
s at most
t at bad, or maybe better. T at way,
f we f
d out a pretty good
compex
ty for our atered program, w
c
s worse t a our or
g
a, we
ca ow t at our or
g
a program w
 ave a pretty good compex
ty
too e
t er as good as our atered program or eve better.
Let's ow t
 of t e way to ed
t t
s exampe program to mae
t
eas
er to f
gure out
ts compex
ty. But et's eep
 m
d t at we ca
oy mae
t worse,
.e. mae
t tae up more
struct
os, so t at our
est
mate
s mea
gfu for our or
g
a program. Ceary we ca ater
t e
er oop of t e program to aways repeat exacty  t
mes
stead
of a vary
g umber of t
mes. Some of t ese repet
t
os w
 be useess,
but
t w
 ep us aayze t e compex
ty of t e resut
g agor
t m.
If we mae t
s s
mpe c age, t e t e ew agor
t m t at we've
costructed
s ceary ( ^2 ), because we ave two ested oops w ere
eac repeats exacty  t
mes. If t at
s so, we say t at t e or
g
a
agor
t m
s O( ^2 ). O( ^2 )
s proouced "b
g o of  squared".
W at t
s says
s t at our program
s asymptot
cay o worse t a ^2 .
It may eve be better t a t at, or
t may be t e same as t at. By t e
way,
f our program
s
deed ( ^2 ), we ca st
 say t at
t's O(
^2 ). To ep you rea
ze t at,
mag
e ater
g t e or
g
a program

 a way t at does't c age
t muc , but st
 maes
t a 
tte worse,
suc as add
g a mea
gess
struct
o at t e beg

g of t e
program. Do
g t
s w
 ater t e
struct
o-cout
g fuct
o by a
s
mpe costat, w
c
s
gored w e
t comes to asymptot
c be av
or.
So a program t at
s ( ^2 )
s aso O( ^2 ).
But a program t at
s O( ^2 ) may ot be ( ^2 ). For exampe, ay
program t at
s (  )
s aso O( ^2 )
 add
t
o to be
g O(  ). If
we
mag
e t e t at a (  ) program
s a s
mpe |for| oop t at repeats
 t
mes, we ca mae
t worse by wrapp
g
t
 aot er |for| oop w
c
repeats  t
mes as we, t us produc
g a program w
t f(  ) = ^2 . To
geera
ze t
s, ay program t at
s ( a )
s O( b ) w e b
s worse
t a a. Not
ce t at our aterat
o to t e program does't eed to g
ve
us a program t at
s actuay mea
gfu or equ
vaet to our or
g
a
program. It oy eeds to perform more
struct
os t a t e or
g
a
for a g
ve . A we're us
g
t for
s cout
g
struct
os, ot
actuay sov
g our probem.
So, say
g t at our program
s O( ^2 )
s be
g o t e safe s
de: We've
aayzed our agor
t m, ad we've foud t at
t's ever worse t a ^2 .
But
t coud be t at
t's
 fact ^2 . T
s g
ves us a good est
mate of
ow fast our program rus. Let's go t roug a few exampes to ep you
fam

ar
ze yoursef w
t t
s ew otat
o.

xerc
se 3
F
d out w
c of t e foow
g are true:
1. A (  ) agor
t m
s O(  )
2. A (  ) agor
t m
s O( ^2 )
3. A ( ^2 ) agor
t m
s O( ^3 )
4. A (  ) agor
t m
s O( 1 )
5. A O( 1 ) agor
t m
s ( 1 )
6. A O(  ) agor
t m
s ( 1 )

Sout
o
1. We ow t at t
s
s true as our or
g
a program was (  ). We ca
ac
eve O(  ) w
t out ater
g our program at a.
2. As ^2
s worse t a , t
s
s true.
3. As ^3
s worse t a ^2 , t
s
s true.
4. As 1
s ot worse t a , t
s
s fase. If a program taes 

struct
os asymptot
cay (a 
ear umber of
struct
os), we
ca't mae
t worse ad ave
t tae oy 1
struct
o
asymptot
cay (a costat umber of
struct
os).
5. T
s
s true as t e two compex
t
es are t e same.
6. T
s may or may ot be true deped
g o t e agor
t m. I t e
geera case
t's fase. If a agor
t m
s ( 1 ), t e
t
certa
y
s O(  ). But
f
t's O(  ) t e
t may ot be ( 1 ).
For exampe, a (  ) agor
t m
s O(  ) but ot ( 1 ).

xerc
se 4

Use a ar
t met
c progress
o sum to prove t at t e above program
s ot

oy O( ^2 ) but aso ( ^2 ). If you do't ow w at a ar
t met
c
progress
o
s, oo
t up o W

ped
a
< ttp://e.w

ped
a.org/w

/1_%2B_2_%2B_3_%2B_4_%2B_%2%80%A6>
t's
easy.
Because t e O-compex
ty of a agor
t m g
ves a /upper boud/ for t e
actua compex
ty of a agor
t m, w
e g
ves t e actua compex
ty
of a agor
t m, we somet
mes say t at t e g
ves us a /t
g t boud/.
If we ow t at we've foud a compex
ty boud t at
s ot t
g t, we ca
aso use a ower-case o to deote t at. For exampe,
f a agor
t m
s
(  ), t e
ts t
g t compex
ty
s . T e t
s agor
t m
s bot O( 
) ad O( ^2 ). As t e agor
t m
s (  ), t e O(  ) boud
s a t
g t
oe. But t e O( ^2 ) boud
s ot t
g t, ad so we ca wr
te t at t e
agor
t m
s o( ^2 ), w
c
s proouced "sma o of  squared" to

ustrate t at we ow our boud
s ot t
g t. It's better
f we ca
f
d t
g t bouds for our agor
t ms, as t ese g
ve us more
format
o
about ow our agor
t m be aves, but
t's ot aways easy to do.

xerc
se 5
Determ
e w
c of t e foow
g bouds are t
g t bouds ad w
c are
ot t
g t bouds. C ec to see
f ay bouds may be wrog. Use o(
otat
o ) to
ustrate t e bouds t at are ot t
g t.
1. A (  ) agor
t m for w
c we foud a O(  ) upper boud.
2. A ( ^2 ) agor
t m for w
c we foud a O( ^3 ) upper boud.
3. A ( 1 ) agor
t m for w
c we foud a O(  ) upper boud.
4. A (  ) agor
t m for w
c we foud a O( 1 ) upper boud.
5. A (  ) agor
t m for w
c we foud a O( 2 ) upper boud.

Sout
o
1. I t
s case, t e compex
ty ad t e O compex
ty are t e same, so
t e boud
s t
g t.
2. Here we see t at t e O compex
ty
s of a arger scae t a t e
compex
ty so t
s boud
s ot t
g t. Ideed, a boud of O( ^2 )
woud be a t
g t oe. So we ca wr
te t at t e agor
t m
s o( ^3 ).
3. Aga
 we see t at t e O compex
ty
s of a arger scae t a t e
compex
ty so we ave a boud t at
s't t
g t. A boud of O( 1 )
woud be a t
g t oe. So we ca po
t out t at t e O(  ) boud
s
ot t
g t by wr
t
g
t as o(  ).
4. We must ave made a m
stae
 cacuat
g t
s boud, as
t's
wrog. It's
mposs
be for a (  ) agor
t m to ave a upper boud
of O( 1 ), as 
s a arger compex
ty t a 1. Remember t at O g
ves
a upper boud.
5. T
s may seem 
e a boud t at
s ot t
g t, but t
s
s ot
actuay true. T
s boud
s
 fact t
g t. Reca t at t e
asymptot
c be av
or of 2 ad  are t e same, ad t at O ad are
oy cocered w
t asymptot
c be av
or. So we ave t at O( 2 ) =
O(  ) ad t erefore t
s boud
s t
g t as t e compex
ty
s t e
same as t e .
*Rue of t umb*: It's eas
er to f
gure out t e O-compex
ty of a
agor
t m t a
ts -compex
ty.
You may be gett
g a 
tte overw emed w
t a t
s ew otat
o by
ow, but et's
troduce just two more symbos before we move o to a
few exampes. T ese are easy ow t at you ow , O ad o, ad we wo't
use t em muc ater
 t
s art
ce, but
t's good to ow t em ow t at
we're at
t. I t e exampe above, we mod
f
ed our program to mae
t
worse (
.e. ta
g more
struct
os ad t erefore more t
me) ad
created t e O otat
o. O
s mea
gfu because
t tes us t at our
program w
 ever be sower t a a spec
f
c boud, ad so
t prov
des
vauabe
format
o so t at we ca argue t at our program
s good
eoug . If we do t e oppos
te ad mod
fy our program to mae
t *better*
ad f
d out t e compex
ty of t e resut
g program, we use t e
otat
o . t erefore g
ves us a compex
ty t at we ow our program
wo't be better t a. T
s
s usefu
f we wat to prove t at a program
rus sowy or a agor
t m
s a bad oe. T
s ca be usefu to argue
t at a agor
t m
s too sow to use
 a part
cuar case. For exampe,
say
g t at a agor
t m
s ( ^3 ) meas t at t e agor
t m
s't
better t a ^3 . It m
g t be ( ^3 ), as bad as ( ^4 ) or eve
worse, but we ow
t's at east somew at bad. So g
ves us a /ower
boud/ for t e compex
ty of our agor
t m. S
m
ary to , we can write
if e kno that our bound isn't tight. For example, a ( n^3 )
algorithm is ( n^4 ) and ( n^2 ). (  )
s proouced "b
g omega of
", w
e ( n ) is pronounced "small omega of n".

Exercise 6
For the folloing complexities rite don a tight and a non-tight O
bound, and a tight and non-tight boud of your c o
ce, prov
d
g t ey
ex
st.
1. ( 1 )
2. ( sqrt(  ) )
3. (  )
4. ( ^2 )
5. ( ^3 )

Sout
o
T
s
s a stra
g t-forward app
cat
o of t e def

t
os above.
1. T e t
g t bouds w
 be O( 1 ) ad ( 1 ). A o-t
g t O-boud
woud be O(  ). Reca t at O g
ves us a upper boud. As 
s of
arger scae t a 1 t
s
s a o-t
g t boud ad we ca wr
te
t as
o(  ) as we. But we caot f
d a o-t
g t boud for , as we
ca't get ower t a 1 for t ese fuct
os. So we' ave to do w
t
t e t
g t boud.
2. T e t
g t bouds w
 ave to be t e same as t e compex
ty, so
t ey are O( sqrt(  ) ) ad ( sqrt(  ) ) respect
vey. For
o-t
g t bouds we ca ave O(  ), as 
s arger t a sqrt(  )
ad so
t
s a upper boud for sqrt(  ). As we ow t
s
s a
o-t
g t upper boud, we ca aso wr
te
t as o(  ). For a ower
boud t at
s ot t
g t, we ca s
mpy use ( 1 ). As we ow t at
t
s boud
s ot t
g t, we ca aso wr
te
t as ( 1 ).
3. The tight bounds are O( n ) and (  ). Two o-t
g t bouds coud
be ( 1 ) and o( n^3 ). These are in fact pretty bad bounds, as they
are far from the original complexities, but they are still valid
using our definitions.
4. The tight bounds are O( n^2 ) and ( ^2 ). For o-t
g t bouds we
coud aga
 use ( 1 ) and o( n^3 ) as in our previous example.
5. The tight bounds are O( n^3 ) and ( ^3 ) respect
vey. Two
o-t
g t bouds coud be ( sqrt( n ) n^2 ) and o( sqrt( n ) n^3 ).
Although these bounds are not tight, they're better than the ones e
gave above.
The reason e use O and
stead of eve t oug O ad ca aso g
ve
t
g t bouds
s t at we may ot be abe to te
f a boud we've foud

s t
g t, or we may just ot wat to go t roug t e process of
scrut

z
g
t so muc .
If you do't fuy remember a t e d
fferet symbos ad t e
r uses,
do't worry about
t too muc r
g t ow. You ca aways come bac ad
oo t em up. T e most
mportat symbos are O ad .
Aso ote t at at oug g
ves us a ower-boud be av
or for our
fuct
o (
.e. we've
mproved our program ad made
t perform ess

struct
os) we're st
 referr
g to a "worst-case" aays
s. T
s
s
because we're feed
g our program t e worst poss
be
put for a g
ve 
ad aayz
g
ts be av
or uder t
s assumpt
o.
T e foow
g tabe
d
cates t e symbos we just
troduced ad t e
r
correspodece w
t t e usua mat emat
ca symbos of compar
sos t at
we use for umbers. T e reaso we do't use t e usua symbos ere ad
use Gree etters
stead
s to po
t out t at we're do
g a asymptot
c
be av
or compar
so, ot just a s
mpe compar
so.
Asymptot
c compar
so operator Numer
c compar
so operator
Our agor
t m
s *o*( somet
g ) A umber
s *<* somet
g
Our agor
t m
s *O*( somet
g ) A umber
s ** somet
g
Our agor
t m
s **( somet
g ) A umber
s *=* somet
g
Our agor
t m
s **( somet
g ) A umber
s ** somet
g
Our agor
t m
s **( something ) A number is *>* something
*Rule of thumb*: While all the symbols O, o, , and are useful at
times, O is the one used more commonly, as it's easier to determine than
and more practically useful than .
T e og fuct
o
s muc ower t a t e square root fuct
o, w
c ,

tur,
s muc ower t a t e 
ear fuct
o eve for sma 
*F
gure 4*: A compar
so of t e fuct
os , sqrt(  ), ad og(  ).
Fuct
o , t e 
ear fuct
o, draw
 gree at t e top, grows muc
faster t a t e square root fuct
o, draw
 red
 t e m
dde, w
c ,

 tur, grows muc faster t a t e og(  ) fuct
o draw
 bue at
t e bottom of t
s pot. ve for sma  suc as  = 100, t e
d
fferece
s qu
te proouced.

Logar
t ms
If you ow w at ogar
t ms are, fee free to s
p t
s sect
o. As a
ot of peope are ufam

ar w
t ogar
t ms, or just ave't used t em
muc recety ad do't remember t em, t
s sect
o
s ere as a

troduct
o for t em. T
s text
s aso for youger studets t at
ave't see ogar
t ms at sc oo yet. Logar
t ms are
mportat because
t ey occur a ot w e aayz
g compex
ty. A /ogar
t m/
s a
operat
o app
ed to a umber t at maes
t qu
te smaer muc 
e a
square root of a umber. So
f t ere's oe t
g you wat to remember
about ogar
t ms
s t at t ey tae a umber ad mae
t muc smaer
t a t e or
g
a (See *F
gure 4*). Now,
 t e same way t at square
roots are t e
verse operat
o of squar
g somet
g, ogar
t ms are
t e
verse operat
o of expoet
at
g somet
g. T
s
s't as ard as

t souds. It's better expa
ed w
t a exampe. Cos
der t e equat
o:
2^x = 1024
We ow w
s to sove t
s equat
o for x. So we as ourseves: W at
s
t e umber to w
c we must ra
se t e base 2 so t at we get 1024? T at
umber
s 10. Ideed, we ave 2^10 = 1024, w
c
s easy to ver
fy.
Logar
t ms ep us deote t
s probem us
g ew otat
o. I t
s case,
10
s t e ogar
t m of 1024 ad we wr
te t
s as og( 1024 ) ad we read

t as "t e ogar
t m of 1024". Because we're us
g 2 as a base, t ese
ogar
t ms are caed base 2 ogar
t ms. T ere are ogar
t ms
 ot er
bases, but we' oy use base 2 ogar
t ms
 t
s art
ce. If you're a
studet compet
g

terat
oa compet
t
os ad you do't ow about
ogar
t ms, I
g y recommed t at you pract
ce your ogar
t ms
< ttp://tutor
a.mat .amar.edu/Casses/Ag/LogFuct
os.aspx> after
compet
g t
s art
ce. I computer sc
ece, base 2 ogar
t ms are muc
more commo t a ay ot er types of ogar
t ms. T
s
s because we ofte
oy ave two d
fferet et
t
es: 0 ad 1. We aso ted to cut dow oe
b
g probem
to aves, of w
c t ere are aways two. So you oy eed
to ow about base-2 ogar
t ms to cot
ue w
t t
s art
ce.

xerc
se 7
Sove t e equat
os beow. Deote w at ogar
t m you're f
d
g
 eac
case. Use oy ogar
t ms base 2.
1. 2^x = 64
2. (2^2 )^x = 64
3. 4^x = 4
4. 2^x = 1
5. 2^x + 2^x = 32
6. (2^x ) * (2^x ) = 64

Sout
o
T ere
s ot
g more to t
s t a appy
g t e
deas def
ed above.
1. By tr
a ad error we ca f
d t at x = 6 ad so og( 64 ) = 6.
2. Here we ot
ce t at (2^2 )^x , by t e propert
es of expoets, ca
be wr
tte as 2^2x . So we ave t at 2x = 6 because og( 64 ) = 6
from t e prev
ous resut ad t erefore x = 3.
3. Us
g our owedge from t e prev
ous equat
o, we ca wr
te 4 as
2^2 ad so our equat
o becomes (2^2 )^x = 4 w
c
s t e same as
2^2x = 4. T e we ot
ce t at og( 4 ) = 2 because 2^2 = 4 ad
t erefore we ave t at 2x = 2. So x = 1. T
s
s read
y observed
from t e or
g
a equat
o, as us
g a expoet of 1 y
eds t e
base as a resut.
4. Reca t at a expoet of 0 y
eds a resut of 1. So we ave og( 1
) = 0 as 2^0 = 1, ad so x = 0.
5. Here we ave a sum ad so we ca't tae t e ogar
t m d
recty.
However we ot
ce t at 2^x + 2^x
s t e same as 2 * (2^x ). So we've
mut
p
ed
 yet aot er two, ad t erefore t
s
s t e same as 2^x
+ 1 ad ow a we ave to do
s sove t e equat
o 2^x + 1 = 32. We
f
d t at og( 32 ) = 5 ad so x + 1 = 5 ad t erefore x = 4.
6. We're mut
py
g toget er two powers of 2, ad so we ca jo
 t em
by ot
c
g t at (2^x ) * (2^x )
s t e same as 2^2x . T e a we
eed to do
s to sove t e equat
o 2^2x = 64 w
c we aready
soved above ad so x = 3.
*Rue of t umb*: For compet
t
o agor
t ms
mpemeted
 C++, oce
you've aayzed your compex
ty, you ca get a roug est
mate of ow
fast your program w
 ru by expect
g
t to perform about 1,000,000
operat
os per secod, w ere t e operat
os you cout are g
ve by t e
asymptot
c be av
or fuct
o descr
b
g your agor
t m. For exampe, a
(  ) agor
t m taes about a secod to process t e
put for  =
1,000,000.
factor
a( 5 ) -> factor
a( 4 ) -> factor
a( 3 ) -> factor
a( 2 ) ->
factor
a( 1 )
*F
gure 5*: T e recurs
o performed by t e factor
a fuct
o.

Recurs
ve compex
ty
Let's ow tae a oo at a recurs
ve fuct
o. A /recurs
ve fuct
o/
s
a fuct
o t at cas
tsef. Ca we aayze
ts compex
ty? T e
foow
g fuct
o, wr
tte
 Pyt o, evauates t e factor
a
< ttp://e.w

ped
a.org/w

/Factor
a> of a g
ve umber. T e
factor
a of a pos
t
ve
teger umber
s foud by mut
py
g
t w
t
a t e prev
ous pos
t
ve
tegers toget er. For exampe, t e factor
a
of 5
s 5 * 4 * 3 * 2 * 1. We deote t at "5!" ad proouce
t "f
ve
factor
a" (some peope prefer to proouce
t by scream
g
t out aoud

e "FIV!!!")
|1.||def| |factor
a(  ):|
|2.|| ||
f| | ||=||=| |1||:|
|3.|| ||retur| |1|
|4.|| ||retur| | ||*| |factor
a(  ||-| |1| |)|
Let us aayze t e compex
ty of t
s fuct
o. T
s fuct
o does't
ave ay oops

t, but
ts compex
ty
s't costat e
t er. W at we
eed to do to f
d out
ts compex
ty
s aga
 to go about cout
g

struct
os. Ceary,
f we pass some  to t
s fuct
o,
t w

execute
tsef  t
mes. If you're usure about t at, ru
t "by ad"
ow for  = 5 to va
date t at
t actuay wors. For exampe, for  =
5,
t w
 execute 5 t
mes, as
t w
 eep decreas
g  by 1
 eac
ca. We ca see t erefore t at t
s fuct
o
s t e (  ).
If you're usure about t
s fact, remember t at you ca aways f
d t e
exact compex
ty by cout
g
struct
os. If you w
s , you ca ow try
to cout t e actua
struct
os performed by t
s fuct
o to f
d a
fuct
o f(  ) ad see t at
t's
deed 
ear (reca t at 
ear
meas (  )).
See *F
gure 5* for a d
agram to ep you uderstad t e recurs
os
performed w e factor
a( 5 )
s caed.
T
s s oud cear up w y t
s fuct
o
s of 
ear compex
ty.
B
ary searc
g
 a array
*F
gure 6*: T e recurs
o performed by b
ary searc . T e A argumet for
eac ca
s
g 
g ted
 bac. T e recurs
o cot
ues ut
 t e
array exam
ed cos
sts of oy oe eemet. Courtesy of Lue Frac.

Logar
t m
c compex
ty

Oe famous probem
 computer sc
ece
s t at of searc
g for a vaue
w
t
 a array. We soved t
s probem ear
er for t e geera case.
T
s probem becomes
terest
g
f we ave a array w
c
s sorted ad
we wat to f
d a g
ve vaue w
t

t. Oe met od to do t at
s caed
/b
ary searc /. We oo at t e m
dde eemet of our array: If we f
d

t t ere, we're doe. Ot erw
se,
f t e vaue we f
d t ere
s b
gger
t a t e vaue we're oo
g for, we ow t at our eemet w
 be o
t e eft part of t e array. Ot erw
se, we ow
t' be o t e r
g t
part of t e array. We ca eep cutt
g t ese smaer arrays
 aves
ut
 we ave a s
ge eemet to oo at. Here's t e met od us
g
pseudocode:
|01.||def| |b
arySearc ( A, , vaue ):|
|02.|| ||
f| | ||=| |1||:|
|03.|| ||
f| |A[ ||0| |] ||=| |vaue:|
|04.|| ||retur| |true|
|05.|| ||ese||:|
|06.|| ||retur| |fase|
|07.|| ||
f| |vaue < A[  ||/| |2| |]:|
|08.|| ||retur| |b
arySearc ( A[ ||0.||..(  ||/| |2| |-| |1|
|) ],  ||/| |2| |-| |1||, vaue )|
|09.|| ||ese| |
f| |vaue > A[  ||/| |2| |]:|
|10.|| ||retur| |b
arySearc ( A[ (  ||/| |2| |+| |1| |)... ],
 ||/| |2| |-| |1||, vaue )|
|11.|| ||ese||:|
|12.|| ||retur| |true|
T
s pseudocode
s a s
mp
f
cat
o of t e actua
mpemetat
o. I
pract
ce, t
s met od
s eas
er descr
bed t a
mpemeted, as t e
programmer eeds to tae care of some
mpemetat
o
ssues. T ere are
off-by-oe errors ad t e d
v
s
o by 2 may ot aways produce a

teger vaue ad so
t's ecessary to foor() or ce
() t e vaue. But
we ca assume for our purposes t at
t w
 aways succeed, ad we'
assume our actua
mpemetat
o
 fact taes care of t e off-by-oe
errors, as we oy wat to aayze t e compex
ty of t
s met od. If
you've ever
mpemeted b
ary searc before, you may wat to do t
s

 your favour
te programm
g aguage. It's a truy e
g te
g edeavor.
See *F
gure 6* to ep you uderstad t e way b
ary searc operates.
If you're usure t at t
s met od actuay wors, tae a momet ow to
ru
t by ad
 a s
mpe exampe ad cov
ce yoursef t at
t
actuay wors.
Let us ow attempt to aayze t
s agor
t m. Aga
, we ave a recurs
ve
agor
t m
 t
s case. Let's assume, for s
mp
c
ty, t at t e array
s
aways cut
 exacty a af,
gor
g just ow t e + 1 ad - 1 part

t e recurs
ve ca. By ow you s oud be cov
ced t at a 
tte c age
suc as
gor
g + 1 ad - 1 wo't affect our compex
ty resuts. T
s

s a fact t at we woud ormay ave to prove
f we wated to be
prudet from a mat emat
ca po
t of v
ew, but pract
cay
t
s

tu
t
vey obv
ous. Let's assume t at our array as a s
ze t at
s a
exact power of 2, for s
mp
c
ty. Aga
 t
s assumpt
o does't c age
t e f
a resuts of our compex
ty t at we w
 arr
ve at. T e
worst-case scear
o for t
s probem woud appe w e t e vaue we're
oo
g for does ot occur
 our array at a. I t at case, we'd start
w
t a array of s
ze 
 t e f
rst ca of t e recurs
o, t e get a
array of s
ze  / 2
 t e ext ca. T e we' get a array of s
ze 
/ 4
 t e ext recurs
ve ca, foowed by a array of s
ze  / 8 ad
so fort . I geera, our array
s sp
t
 af
 every ca, ut
 we
reac 1. So, et's wr
te t e umber of eemets
 our array for every ca:
1. 0^t
terat
o: 
2. 1^st
terat
o:  / 2
3. 2^d
terat
o:  / 4
4. 3^rd
terat
o:  / 8
5. ...
6.
^t
terat
o:  / 2^

7. ...
8. ast
terat
o: 1
Not
ce t at
 t e
-t
terat
o, our array as  / 2^
eemets. T
s

s because
 every
terat
o we're cutt
g our array
to af, mea
g
we're d
v
d
g
ts umber of eemets by two. T
s trasates to
mut
py
g t e deom
ator w
t a 2. If we do t at
t
mes, we get  /
2^
. Now, t
s procedure cot
ues ad w
t every arger
we get a
smaer umber of eemets ut
 we reac t e ast
terat
o
 w
c we
ave oy 1 eemet eft. If we w
s to f
d
to see
 w at
terat
o
t
s w
 tae pace, we ave to sove t e foow
g equat
o:
1 =  / 2^

T
s w
 oy be true w e we ave reac ed t e f
a ca to t e
b
arySearc () fuct
o, ot
 t e geera case. So sov
g for
ere
w
 ep us f
d
 w
c
terat
o t e recurs
o w
 f

s .
Mut
py
g bot s
des by 2^
we get:
2^
= 
Now, t
s equat
o s oud oo fam

ar
f you read t e ogar
t ms
sect
o above. Sov
g for
we ave:

= og(  )
T
s tes us t at t e umber of
terat
os requ
red to perform a b
ary
searc
s og(  ) w ere 
s t e umber of eemets
 t e or
g
a array.
If you t
 about
t, t
s maes some sese. For exampe, tae  = 32,
a array of 32 eemets. How may t
mes do we ave to cut t
s
 af
to get oy 1 eemet? We get: 32 16 8 4 2 1. We d
d t
s 5
t
mes, w
c
s t e ogar
t m of 32. T erefore, t e compex
ty of b
ary
searc
s ( og(  ) ).
T
s ast resut aows us to compare b
ary searc w
t 
ear searc ,
our prev
ous met od. Ceary, as og(  )
s muc smaer t a ,
t
s
reasoabe to cocude t at b
ary searc
s a muc faster met od to
searc w
t
 a array t e 
ear searc , so
t may be adv
sabe to
eep our arrays sorted
f we wat to do may searc es w
t
 t em.
*Rue of t umb*: Improv
g t e asymptot
c ru
g t
me of a program
ofte tremedousy
creases
ts performace, muc more t a ay smaer
"tec 
ca" opt
m
zat
os suc as us
g a faster programm
g aguage.

Opt
ma sort
g
*Cogratuat
os.* You ow ow about aayz
g t e compex
ty of
agor
t ms, asymptot
c be av
or of fuct
os ad b
g-O otat
o. You
aso ow ow to
tu
t
vey f
gure out t at t e compex
ty of a
agor
t m
s O( 1 ), O( og(  ) ), O(  ), O( ^2 ) ad so fort . You
ow t e symbos o, O, , ad ad w at worst-case aays
s meas. If
you've come t
s far, t
s tutor
a as aready served
ts purpose.
T
s f
a sect
o
s opt
oa. It
s a 
tte more
voved, so fee
free to s
p
t
f you fee overw emed by
t. It w
 requ
re you to
focus ad sped some momets wor
g t roug t e exerc
ses. However,
t
w
 prov
de you w
t a very usefu met od
 agor
t m compex
ty
aays
s w
c ca be very powerfu, so
t's certa
y wort uderstad
g.
We ooed at a sort
g
mpemetat
o above caed a seect
o sort. We
met
oed t at seect
o sort
s ot opt
ma. A /opt
ma agor
t m/
s
a agor
t m t at soves a probem
 t e best poss
be way, mea
g
t ere are o better agor
t ms for t
s. T
s meas t at a ot er
agor
t ms for sov
g t e probem ave a worse or equa compex
ty to
t at opt
ma agor
t m. T ere may be may opt
ma agor
t ms for a
probem t at a s are t e same compex
ty. T e sort
g probem ca be
soved opt
may
 var
ous ways. We ca use t e same
dea as w
t
b
ary searc to sort qu
cy. T
s sort
g met od
s caed /mergesort/.
To perform a mergesort, we w
 f
rst eed to bu
d a eper fuct
o
t at we w
 t e use to do t e actua sort
g. We w
 mae a |merge|
fuct
o w
c taes two arrays t at are bot aready sorted ad merges
t em toget er
to a b
g sorted array. T
s
s eas
y doe:
|01.||def| |merge( A, B ):|
|02.|| ||
f| |empty( A ):|
|03.|| ||retur| |B|
|04.|| ||
f| |empty( B ):|
|05.|| ||retur| |A|
|06.|| ||
f| |A[ ||0| |] < B[ ||0| |]:|
|07.|| ||retur| |cocat( A[ ||0| |], merge( A[ ||1.||..A_ ], B ) )|
|08.|| ||ese||:|
|09.|| ||retur| |cocat( B[ ||0| |], merge( A, B[ ||1.||..B_ ] ) )|
T e cocat fuct
o taes a
tem, t e " ead", ad a array, t e "ta
",
ad bu
ds up ad returs a ew array w
c cota
s t e g
ve " ead"

tem as t e f
rst t
g
 t e ew array ad t e g
ve "ta
"
tem as
t e rest of t e eemets
 t e array. For exampe, cocat( 3, [ 4, 5, 6
] ) returs [ 3, 4, 5, 6 ]. We use A_ ad B_ to deote t e s
zes of
arrays A ad B respect
vey.

xerc
se 8
Ver
fy t at t e above fuct
o actuay performs a merge. Rewr
te
t

your favour
te programm
g aguage
 a
terat
ve way (us
g |for|
oops)
stead of us
g recurs
o.
Aayz
g t
s agor
t m reveas t at
t as a ru
g t
me of (  ),
w ere 
s t e egt of t e resut
g array ( = A_ + B_).

xerc
se 9

Ver
fy t at t e ru
g t
me of |merge|
s (  ).

Ut

z
g t
s fuct
o we ca bu
d a better sort
g agor
t m. T e

dea
s t e foow
g: We sp
t t e array
to two parts. We sort eac
of t e two parts recurs
vey, t e we merge t e two sorted arrays
to
oe b
g array. I pseudocode:
|1.||def| |mergeSort( A,  ):|
|2.|| ||
f| | ||=| |1||:|
|3.|| ||retur| |A ||#
t
s aready sorted|
|4.|| ||m
dde ||=| |foor(  ||/| |2| |)|
|5.|| ||eftHaf ||=| |A[ ||1.||..m
dde ]|
|6.|| ||r
g tHaf ||=| |A[ ( m
dde ||+| |1| |)... ]|
|7.|| ||retur| |merge( mergeSort( eftHaf, m
dde ), mergeSort(
r
g tHaf,  ||-| |m
dde ) )|
T
s fuct
o
s arder to uderstad t a w at we've goe t roug
prev
ousy, so t e foow
g exerc
se may tae you a few m
utes.

xerc
se 10
Ver
fy t e correctess of |mergeSort|. T at
s, c ec to see
f
|mergeSort| as def
ed above actuay correcty sorts t e array
t
s
g
ve. If you're av
g troube uderstad
g w y
t wors, try
t w
t
a sma exampe array ad ru
t "by ad". W e ru
g t
s fuct
o
by ad, mae sure eftHaf ad r
g tHaf are w at you get
f you cut
t e array approx
matey
 t e m
dde;
t does't ave to be exacty

t e m
dde
f t e array as a odd umber of eemets (t at's w at
|foor| above
s used for).
As a f
a exampe, et us aayze t e compex
ty of |mergeSort|. I
every step of |mergeSort|, we're sp
tt
g t e array
to two aves of
equa s
ze, s
m
ary to |b
arySearc |. However,
 t
s case, we
ma
ta
 bot aves t roug out execut
o. We t e appy t e agor
t m
recurs
vey
 eac af. After t e recurs
o returs, we appy t e
|merge| operat
o o t e resut w
c taes (  ) t
me.
So, we sp
t t e or
g
a array
to two arrays of s
ze  / 2 eac . T e
we merge t ose arrays, a operat
o t at merges  eemets ad t us
taes (  ) t
me.
Tae a oo at *F
gure 7* to uderstad t
s recurs
o.
N sp
ts
to N / 2 ad N / 2. ac of t ose sp
ts
to N / 4 ad N /
4, ad t e process cot
ues ut
 we ave cas of s
ze 1.
*F
gure 7*: T e recurs
o tree of merge sort.
Let's see w at's go
g o ere. ac c
rce represets a ca to t e
|mergeSort| fuct
o. T e umber wr
tte
 t e c
rce
d
cates t e
s
ze of t e array t at
s be
g sorted. T e top bue c
rce
s t e
or
g
a ca to |mergeSort|, w ere we get to sort a array of s
ze .
T e arrows
d
cate recurs
ve cas made betwee fuct
os. T e or
g
a
ca to |mergeSort| maes two cas to |mergeSort| o two arrays, eac
of s
ze  / 2. T
s
s
d
cated by t e two arrows at t e top. I tur,
eac of t ese cas maes two cas of
ts ow to |mergeSort| two arrays
of s
ze  / 4 eac , ad so fort ut
 we arr
ve at arrays of s
ze 1.
T
s d
agram
s caed a /recurs
o tree/, because
t
ustrates ow
t e recurs
o be aves ad oos 
e a tree (t e /root/
s at t e top
ad t e /eaves/ are at t e bottom, so
 rea
ty
t oos 
e a

versed tree).
Not
ce t at at eac row
 t e above d
agram, t e tota umber of
eemets
s . To see t
s, tae a oo at eac row
d
v
duay. T e
f
rst row cota
s oy oe ca to |mergeSort| w
t a array of s
ze ,
so t e tota umber of eemets
s . T e secod row as two cas to
|mergeSort| eac of s
ze  / 2. But  / 2 +  / 2 =  ad so aga


t
s row t e tota umber of eemets
s . I t e t
rd row, we ave 4
cas eac of w
c
s app
ed o a  / 4-s
zed array, y
ed
g a tota
umber of eemets equa to  / 4 +  / 4 +  / 4 +  / 4 = 4 / 4 = .
So aga
 we get  eemets. Now ot
ce t at at eac row
 t
s d
agram
t e caer w
 ave to perform a |merge| operat
o o t e eemets
retured by t e caees. For exampe, t e c
rce
d
cated w
t red
coor as to sort  / 2 eemets. To do t
s,
t sp
ts t e  / 2-s
zed
array
to two  / 4-s
zed arrays, cas |mergeSort| recurs
vey to sort
t ose (t ese cas are t e c
rces
d
cated w
t gree coor), t e
merges t em toget er. T
s merge operat
o requ
res to merge  / 2
eemets. At eac row
 our tree, t e tota umber of eemets merged

s . I t e row t at we just expored, our fuct
o merges  / 2
eemets ad t e fuct
o o
ts r
g t (w
c
s
 bue coor) aso as
to merge  / 2 eemets of
ts ow. T at y
eds  eemets
 tota t at
eed to be merged for t e row we're oo
g at.
By t
s argumet, t e compex
ty for eac row
s (  ). We ow t at
t e umber of rows
 t
s d
agram, aso caed t e /dept / of t e
recurs
o tree, w
 be og(  ). T e reaso
g for t
s
s exacty t e
same as t e oe we used w e aayz
g t e compex
ty of b
ary searc .
We ave og(  ) rows ad eac of t em
s (  ), t erefore t e
compex
ty of |mergeSort|
s (  * og(  ) ). T
s
s muc better t a
( ^2 ) w
c
s w at seect
o sort gave us (remember t at og(  )
s
muc smaer t a , ad so  * og(  )
s muc smaer t a  *  =
^2 ). If t
s souds comp
cated to you, do't worry: It's ot easy t e
f
rst t
me you see
t. Rev
s
t t
s sect
o ad reread about t e
argumets ere after you
mpemet mergesort
 your favour
te
programm
g aguage ad va
date t at
t wors.
As you saw
 t
s ast exampe, compex
ty aays
s aows us to
compare agor
t ms to see w
c oe
s better. Uder t ese
c
rcumstaces, we ca ow be pretty certa
 t at merge sort w

outperform seect
o sort for arge arrays. T
s cocus
o woud be
ard to draw
f we d
d't ave t e t eoret
ca bacgroud of agor
t m
aays
s t at we deveoped. I pract
ce,
deed sort
g agor
t ms of
ru
g t
me (  * og(  ) ) are used. For exampe, t e L
ux ere
uses a sort
g agor
t m caed eapsort
< ttp://xr.free-eectros.com/source/
b/sort.c>, w
c as t e same
ru
g t
me as mergesort w
c we expored ere, amey (  og(  ) )
ad so
s opt
ma. Not
ce t at we ave ot prove t at t ese sort
g
agor
t ms are opt
ma. Do
g t
s requ
res a s
g ty more
voved
mat emat
ca argumet, but rest assured t at t ey ca't get ay better
from a compex
ty po
t of v
ew.
Hav
g f

s ed read
g t
s tutor
a, t e
tu
t
o you deveoped for
agor
t m compex
ty aays
s s oud be abe to ep you des
g faster
programs ad focus your opt
m
zat
o efforts o t e t
gs t at reay
matter
stead of t e m
or t
gs t at do't matter, ett
g you wor
more product
vey. I add
t
o, t e mat emat
ca aguage ad otat
o
deveoped
 t
s art
ce suc as b
g-O otat
o
s epfu

commu
cat
g w
t ot er software eg
eers w e you wat to argue about
t e ru
g t
me of agor
t ms, so opefuy you w
 be abe to do t at
w
t your ewy acqu
red owedge.

About
T
s art
ce
s 
cesed uder Creat
ve Commos 3.0 Attr
but
o
< ttp://creat
vecommos.org/
ceses/by/3.0/>. T
s meas you ca
copy/paste
t, s are
t, post
t o your ow webs
te, c age
t, ad
geeray do w atever you wat w
t
t, prov
d
g you met
o my ame.
At oug you do't ave to,
f you base your wor o m
e, I ecourage
you to pub
s your ow wr
t
gs uder Creat
ve Commos so t at
t's
eas
er for ot ers to s are ad coaborate as we. I a s
m
ar
fas
o, I ave to attr
bute t e wor I used ere. T e 
fty
cos t at
you see o t
s page are t e fugue
cos
< ttp://p.yusueam
yamae.com/>. T e beaut
fu str
ped patter t at you
see
 t
s des
g was created by Lea Verou
< ttp://eaverou.me/css3patters/>. Ad, more
mportaty, t e
agor
t ms I ow so t at I was abe to wr
te t
s art
ce were taug t
to me by my professors N
os Papaspyrou
< ttp://www.softab.tua.gr/%7
c
e/> ad D
m
tr
s Fota
s
< ttp://www.softab.tua.gr/%7fota
s/>.
I'm currety a cryptograp y P D cad
date at t e U
vers
ty of At es
< ttp://d
.uoa.gr/>. W e I wrote t
s art
ce I was a udergraduate
ectr
ca ad Computer g
eer
g < ttp://ece.tua.gr/> udergraduate
at t e Nat
oa Tec 
ca U
vers
ty of At es < ttp://tua.gr/>
master
g
 software < ttp://www.csab.tua.gr/> ad a coac at t e
Gree Compet
t
o
 Iformat
cs < ttp://pdp.gr/>. Idustry-w
se I've
wored as a member of t e eg
eer
g team t at bu
t dev
atART
< ttp://www.dev
atart.com/>, a soc
a etwor for art
sts, at t e
secur
ty teams of Googe < ttps://www.googe.com/> ad Tw
tter
< ttps://tw
tter.com/>, ad two start-ups, Z
o ad Kam
bu w ere we d
d
soc
a etwor
g ad v
deo game deveopmet respect
vey. Foow me o
Tw
tter < ttp://www.tw
tter.com/d
oyz
z> or o G
tHub
< ttp://g
t ub.com/d
oyz
z>
f you ejoyed t
s, or ma
 me
<ma
to:d
oyz
z@gma
.com>
f you wat to get
 touc . May youg
programmers do't ave a good owedge of t e g
s aguage. -ma

me
f you wat to trasate t
s art
ce
to your ow at
ve aguage
so t at more peope ca read
t.
*T as for read
g.* I d
d't get pa
d to wr
te t
s art
ce, so
f you

ed
t, sed me a e-ma
 <ma
to:d
oyz
z@gma
.com> to say eo. I
ejoy rece
v
g p
ctures of paces aroud t e word, so fee free to
attac a p
cture of yoursef
 your c
ty!

Refereces
1. Corme, Le
serso, R
vest, Ste
. Itroduct
o to Agor
t ms
< ttp://www.amazo.co.u/Itroduct
o-Agor
t ms-T-Corme/dp/0262533057/ref=
sr_1_1?
e=UTF8&q
d=1341414466&sr=8-1>,
MIT Press.
2. Dasgupta, Papad
m
tr
ou, Vaz
ra
. Agor
t ms
< ttp://www.amazo.co.u/Agor
t ms-Sajoy-Dasgupta/dp/0073523402/ref=sr_1_1
?s=boos&
e=UTF8&q
d=1341414505&sr=1-1>,
McGraw-H
 Press.
3. Fota
s. Course of D
screte Mat emat
cs < ttp://d
screte.gr/> at t e
Nat
oa Tec 
ca U
vers
ty of At es.
4. Fota
s. Course of Agor
t ms ad Compex
ty
< ttp://www.coreab.ece.tua.gr/courses/agor
t ms/> at t e Nat
oa
Tec 
ca U
vers
ty of At es.
Tweet
< ttps://tw
tter.com/s are> A b
tco

Doate v
a b
tco
 < ttp://b
tco
.org/>:
Creat
ve Commos 3.0
< ttp://creat
vecommos.org/
ceses/by/3.0/>
Lov
gy made
 At es c
ty by d
oyz
z < ttp://tw
tter.com/d
oyz
z>.