Growth Rate of Functions and the Complexity of

Algorithms

CS/MA 109
Wayne Snyder
Fall, 2017

Today:

Summary: Graphing functions

Applications to Computer Science:

Complexity Theory and Growth Rate of Algorithms

Complexity classes: Logarithmic, Linear, Polynomial, Exponential

Computer Science
 Review: Functions and Graphs
Computer Science

f(x) = x2

A Graph is a drawing of the values of a

function, usually in two-dimensions in an X-Y
Plane with 4 quadrants.
 Review: Functions and Graphs
Computer Science

f(x) = x3 f(x) = x2

A Graph is a drawing of the values of a

function, usually in two-dimensions in an X-Y
Plane with 4 quadrants.
In math, we need all four quadrants because
values may be negative.
 Complexity of Algorithms
Computer Science

A Graph is a drawing of the values of a

function, usually in two-dimensions in an X-Y
Plane with 4 quadrants. f(x) = x3 f(x) = x2

In math, we need all four quadrants because

values may be negative.
But in computer science, we are mostly
interesting in the
Note: Algorithm is a fancy
Complexity of Algorithms
way of saying computer
which means program or app

How much time does the algorithm

take to complete its task?
We graph such behavior as a function
y = f(x)
where
y is the number of steps required
x is the size of the problem
 Complexity of Algorithms
Computer Science

A Graph is a drawing of the values of a

function, usually in two-dimensions in an X-Y f(x) = x3
Plane with 4 quadrants. f(x) = x2

In math, we need all four quadrants because

values may be negative.
But in computer science, we are mostly
interesting in the
Complexity of Algorithms
which means
How much time does the algorithm
take to complete its task?
We graph such behavior as a function
y = f(x)
where
y is the number of steps required
x is the size of the problem
Since the size and the number of steps is
always positive, we only need the upper right
quadrant.
 Complexity of Algorithms
Computer Science

We are going to examine various classes of f(x) = x3 f(x) = x2

functions which take similar amounts of
time, these are called
Complexity Classes of Algorithms
This is how computer scientists under the
big picture of how to solve problems and
what problems can possibly be solved.
 Complexity of Algorithms
Computer Science

We are going to examine various classes of

functions which take similar amounts of f(x) = x
time, these are called
Complexity Classes of Algorithms
This is how computer scientists under the
big picture of how to solve problems and
what problems can possibly be solved.

The first class is called

Linear Time Algorithms
and the functions describing their behavior
have the form

y = f(x) = a*x + b
The simplest such function is just

f(x) = x
 Complexity of Algorithms
Computer Science

The first class is called

f(x) = x
Linear Time Algorithms
and the functions describing their behavior have
the form

y = f(x) = a*x + b
Note: this is the equation of a line, well simplify
a bit and assume that a = 1 and b = 0:

f(x) = x

Example of Linear Time Algorithm:

(Algorithm 1 from last time)
Finding your homework paper in an
unordered pile of homeworks.
x = how many papers in pile
f(x) = how many papers you have
to look at in worst case.
 Complexity of Algorithms
Computer Science

The first class is called

f(x) = x
Linear Time Algorithms
and the functions describing their behavior have
the form

y = f(x) = a*x + b
Note: this is the equation of a line, well simplify Brief Digression: To be a computer
a bit and assume that a = 1 and b = 0:
scientist, you have to be a real pessimist!
f(x) = x You always want to know what is the
absolutely worst thing that can happen?
The worst thing in looking for your paper is
Example of Linear Time Algorithm: that it is in the last place you look. No
matter how to look at the pile, it MAY be
Finding your homework paper in an the last one you look at.
unordered pile of homeworks.
x = how many papers in pile
f(x) = how many papers you have
to look at in worst case.
 Complexity of Algorithms
Computer Science

Example of Linear Time Algorithm:

f(x) = x f(x) = 0.5 * x + 0.5
Finding your homework paper in an
unordered pile of homeworks.
x = how many papers in pile
f(x) = how many papers you have
to look at in worst case.
In the average case, some times it
will be the first one, sometimes the
last one, could be any number from 1
to X, where the pile has X papers in it
(1 + 2 + 3 + ... + (X-1) + X)
X
=*X+ You have to look
about half way down on average!
Still Linear Time!
 Complexity of Algorithms
Computer Science

Example of Linear Time Algorithm:

Finding your homework paper in an
unordered pile of homeworks.
x = how many papers in pile
f(x) = how many papers you have
to look at in worst case.
In the average case, some times it
will be the first one, sometimes the
last one, could be any number from 1
to X, where the pile has X papers in it
f(x) = x f(x) = 0.5 * x + 0.5
Important Note: This is sometimes called
Asymptotic Complexity, because we are
interested in what happens as X gets
larger and larger. What is the shape of the
function? This characterizes the
complexity class.

(1 + 2 + 3 + ... + (X-1) + X) Linear Time algorithms have a complexity

X function which is a line!

=*X+ You have to look

about half way down on average!
Still Linear Time!
 Complexity of Algorithms
Computer Science

Logarithmic Algorithms
Another class of algorithms have a
complexity function which is a f(x) = x
logarithm of x
f(x) = log2(x)
This is the complexity of Algorithm 2:
Binary Search, e.g., searching for a
word in a dictionary, or for a number
in an ordered list:
f(x) = log2(x)
 Complexity of Algorithms
Computer Science

Logarithmic Algorithms
Another class of algorithms have a Zoom Out:
complexity function which is a
f(x) = x
logarithm of x
f(x) = log2(x)
This is the complexity of Algorithm 2:
Binary Search, e.g., searching for a
word in a dictionary, or for a number
in an ordered list:
f(x) = log2(x)
 Complexity of Algorithms
Computer Science

Logarithmic Algorithms
Another class of algorithms have a
Zoom Out:
complexity function which is a
f(x) = x
logarithm of x
f(x) = log2(x)
This is the complexity of Algorithm 2:
Binary Search, e.g., searching for a
word in a dictionary, or for a number
in an ordered list:
f(x) = log2(x)
Complexity of Algorithms
Computer Science

Polynomial Algorithms f(x) = x3 f(x) = x2

Another class of algorithms have a

complexity function which is a
polynomial in x:
f(x) c1xn + c2xn-1 ... + cnx + b
For example,
f(x) = x2 or f(x) = x3
or
f(x) = * x2 - *x

Algorithm 4 (Bubble Sort) has a

complexity of
f(x) = x2
 Complexity of Algorithms
Computer Science

Polynomial Algorithms f(x) = x2 Zoom Out:

f(x) = x
Another class of algorithms have a
complexity function which is a
polynomial in x:

f(x) c1xn + c2xn-1 ... + cnx + b

For example,
f(x) = x2 or f(x) = x3

These take longer to run than linear

time algorithms as the size of the
problem gets bigger, as we can see
from the graph.
Punchline: If you have a choice,
choose a linear time algorithm!
 Complexity of Algorithms
Computer Science

f(x) = 2x
Exponential Algorithms
f(x) = * x2 - *x f(x) = x
These have a complexity function
which is NOT a polynomial, because
it has X in the exponent.
Example:
f(x) = 2x f(x) = Xx

These functions grow so fast, they

exceed any polynomial algorithm, and
are only usable for relatively small X.
Algorithm 5 (Guess a password) has
a complexity of
f(x) = 70x

where x = length of the password.

Complexity of Algorithms
Computer Science
f(x) = 2x
f(x) = x2 f(x) = x2

f(x) = log2(x)
Complexity of Algorithms
Computer Science
f(x) = 2x
f(x) = x2 f(x) = x2

f(x) = log2(x)
 Exponential Problem: Partition Problem
Computer Science

Algorithm: Try every possible subset!

Input: A set of numbers.
Input: { 1, 2, 7, 3 }
Output: A separation of these numbers into
to two subsets (a partition) such that the {} { 1, 2, 7, 3 } X
sum of each subset is the same, or output {3} { 1, 2, 7 } X
No such partition is possible. {7} { 1, 2, 3 } X
{ 7,3 } { 1, 2 } X
{2} { 1, 7, 3 } X
Example1: Input: { 2, 5, 8, 11, 12 } { 2, 3 } { 1, 7 } X
{ 2, 7 } { 1, 3 } X
Output: { 2, 5, 12 } { 8, 11 } { 2, 7, 3 } { 1 } X
{1} { 2, 7, 3 } X
Example2: Input: { 1, 2, 7, 3 } { 1, 3 } { 2, 7 } X
{ 1, 7 } { 2, 3 } X
Output: No such partition { 1, 7, 3 } { 2 } X
possible. { 1, 2 } { 7, 3 } X
{ 1, 2, 3 } { 7 } X
How to solve it? { 1, 2, 7 } { 3 } X
{ 1, 2, 7, 3 } { } X

Complexity: f(x) = 2x
 Exponential Problem: Guessing Passwords
Computer Science

Suppose you want to break into my computer account, and you

know my password is 8 characters long, and could contain letters
(upper and lower case) and digits.

How many passwords would you have to try in the worst case?

Guess first character: 26+26+10 = 62 choices

Guess second character: 62*62 choices

..

Guess 8th character: 628 = 218,340,105,584,896

 Exponential Problem: Paritition Problem
Computer Science

Algorithm: Try every possible subset!

Input: A set of numbers.
Input: { 1, 2, 7, 3 }
Output: A separation of these numbers into
to two subsets (a partition) such that the {} { 1, 2, 7, 3 } X 0000
sum of each subset is the same, or output {3} { 1, 2, 7 } X 0001
No such partition is possible. {7} { 1, 2, 3 } X 0010
{ 7,3 } { 1, 2 } X 0011
{2} { 1, 7, 3 } X 0100
Example1: Input: { 2, 5, 8, 11, 12 } { 2, 3 } { 1, 7 } X 0101
{ 2, 7 } { 1, 3 } X 0110
Output: { 2, 5, 12 } { 8, 11 } { 2, 7, 3 } { 1 } X 0111
{1} { 2, 7, 3 } X 1000
Example2: Input: { 1, 2, 7, 3 } { 1, 3 } { 2, 7 } X 1001
{ 1, 7 } { 2, 3 } X 1010
Output: No such partition { 1, 7, 3 } { 2 } X 1011
possible. { 1, 2 } { 7, 3 } X 1100
{ 1, 2, 3 } { 7 } X 1101
How to solve it? { 1, 2, 7 } { 3 } X 1110
{ 1, 2, 7, 3 } { } X 1111

Complexity: f(x) = 2x
Exponential Problem: Traveling Salesman
Computer Science

Input: A set of cities with roads between some

of them, where the length off each road is
known. You can get to any city from any other
city over some combination of roads.

A tour is a trip which hits each city exactly

once, arriving back at the starting city. The
cost is the sum of the lengths of all roads
traversed.

Example: A Traveling Salesman wants to

make a tour of all the cities to sell his wares,
but is on a tight budget. He needs to know the
shortest tour.

Output: The shortest possible tour of all

the cities.

Total Length = 97
 Complexity of Algorithms
Computer Science

Example: Traveling Salesman

Problem: Find the shortest circuit f(x) = xx
f(x) = x2 f(x) = x
which touches each node exactly
once.
Algorithm: Try all possible circuits and
pick the smallest. (Really, this is the
best we know how to do!)

There are N! = N*(N-1)* ... *2*1

possible circuits, this is approximately

f(x) = xx
which grows very fast!!
Example: Exponential Growth of Algorithms
Computer Science

f(x) = x
Example: Exponential Growth of Algorithms
Computer Science

f(x) = x2
Example: Exponential Growth of Algorithms
Computer Science

f(x) = xx
Example: Exponential Growth of Algorithms
Computer Science