Design and Analysis of Algorithms

Prepared by: Marvi Bongo

INTRODUCTION
An algorithm, named after the ninth century scholar Abu Jafar Muhammad Ibn Musu Al-Khowarizmi, is defined as
follows: Roughly speaking:
An algorithm is a set of rules for carrying out calculation either by hand or on a machine.
An algorithm is a finite step-by-step procedure to achieve a required result.
An algorithm is a sequence of computational steps that transform the input into the output.
An algorithm is a sequence of operations performed on data that have to be organized in data structures.
An algorithm is an abstraction of a program to be executed on a physical machine (model of Computation).

The Classic Multiplication Algorithm

1. Multiplication, the American way:
Multiply the multiplicand one after another by each digit of the multiplier taken from right to left.
2. Multiplication, the English way:
Multiply the multiplicand one after another by each digit of the multiplier taken from left to right.

Algorithmic is a branch of computer science that consists of designing and analyzing computer algorithms

1. The design pertain to

i. The description of algorithm at an abstract level by means of a pseudo language, and
ii. Proof of correctness that is, the algorithm solves the given problem in all cases.
2. The analysis deals with performance evaluation (complexity analysis).

Algorithm's Performance
Two important ways to characterize the effectiveness of an algorithm:
Space complexity - a measure of the amount of working storage an algorithm needs
Time complexity - an algorithm concerns determining an expression of the number of steps needed
as a function of the problem size

The performance evaluation of an algorithm is obtained by totaling the number of occurrences of each operation
when running the algorithm. The performance of an algorithm is evaluated as a function of the input size n and is to
be considered modulo a multiplicative constant.

Asymptotic Notations
These are commonly use notations in performance analysis and used to characterize the complexity of an
algorithm.
1. O (Big Oh) notation
2. (Big Omega)
3. (Big Theta)

-Notation (Upper Bound)

This notation gives an upper bound for a function to within a constant factor. We write f(n) = O(g(n)) if there
are positive constants n0 and c such that to the right of n0, the value of f(n) always lies on or below c g(n).

-Notation (Lower Bound)

This notation gives a lower bound for a function to within a constant factor. We write f(n) = (g(n)) if there
are positive constants n0 and c such that to the right of n0, the value of f(n) always lies on or above c g(n).

-Notation (Same order)

This notation bounds a function to within constant factors. We say f(n) = (g(n)) if there exist positive
constants n0, c1 and c2 such that to the right of n0 the value of f(n) always lies between c1 g(n) and c2 g(n) inclusive.
Algorithm Analysis
The complexity of an algorithm is a function g(n) that gives the upper bound of the number of operation (or
running time) performed by an algorithm when the input size is n.

There are two interpretations of upper bound.

Worst-case Complexity
The running time for any given size input will be lower than the upper bound except possibly for
some values of the input where the maximum is reached.
Average-case Complexity
The running time for any given size input will be the average number of operations over all
problem instances for a given size.
Optimality
An algorithm for a given problem is optimal if its complexity reaches the lower bound over all the algorithms
solving this problem.
Reduction
Another technique for estimating the complexity of a problem is the transformation of problems, also called
problem reduction.

1. GREEDY ALGORITHMS
Greedy algorithms are simple and straightforward. They are shortsighted in their approach in the sense that
they take decisions on the basis of information at hand without worrying about the effect these decisions
may have in the future. They are easy to invent, easy to implement and most of the time quite efficient.
Many problems cannot be solved correctly by greedy approach. Greedy algorithms are used to solve
optimization problems
Greedy Approach
Greedy Algorithm works by making the decision that seems most promising at any moment; it never
reconsiders this decision, whatever situation may arise later.
Characteristics and Features of Problems solved by Greedy Algorithms
To construct the solution in an optimal way. Algorithm maintains two sets. One contains chosen items and
the other contains rejected items.
The greedy algorithm consists of four (4) function.
o A function that checks whether chosen set of items provide a solution.
o A function that checks the feasibility of a set.
o The selection function tells which of the candidates is the most promising.
o An objective function, which does not appear explicitly, gives the value of a solution.
Structure Greedy Algorithm
Initially the set of chosen items is empty i.e., solution set.
At each step
o item will be added in a solution set by using selection function.
o IF the set would no longer be feasible
reject items under consideration (and is never consider again).
o ELSE IF set is still feasible THEN
add the current item.
Definitions of feasibility
A feasible set (of candidates) is promising if it can be extended to produce not merely a solution, but an
optimal solution to the problem. In particular, the empty set is always promising why? (because an optimal
solution always exists)
Unlike Dynamic Programming, which solves the subproblems bottom-up, a greedy strategy usually
progresses in a top-down fashion, making one greedy choice after another, reducing each problem to a
smaller one.
Greedy-Choice Property
The "greedy-choice property" and "optimal substructure" are two ingredients in the problem that lend to a
greedy strategy.
It says that a globally optimal solution can be arrived at by making a locally optimal choice.

A. Knapsack Problem
There are two versions of problem
o Fractional knapsack problem
The setup is same, but the thief can take fractions of items, meaning that the items can be
broken into smaller pieces so that thief may decide to carry only a fraction of xi of item i,
where 0 xi 1.
Exhibit greedy choice property.
Greedy algorithm exists.
Exhibit optimal substructure property.
o 0-1 knapsack problem
The setup is the same, but the items may not be broken into smaller pieces, so thief may
decide either to take an item or to leave it (binary choice), but may not take a fraction of an
item.
Exhibit No greedy choice property.
No greedy algorithm exists.
Exhibit optimal substructure property.
Only dynamic programming algorithm exists.
B. Huffmans Codes
Huffman code is a technique for compressing data.
Huffman's greedy algorithm look at the occurrence of each character and it as a binary string in an optimal
way.
Based on lengths of assigned codes based on frequencies
Variable Length Codes are known as Prefix Codes
How to generate a Prefix Codes?
o The Huffman coding algorithm does this by constructing a binary tree
o By strategically assigning bits string to symbols based on their frequency, we generate a prefix code
that also bring about compression
Constructing a Huffman code
A greedy algorithm that constructs an optimal prefix code called a Huffman code. The algorithm builds the
tree T corresponding to the optimal code in a bottom-up manner. It begins with a set of |c| leaves and
perform |c|-1 "merging" operations to create the final tree.

Data Structure used: Priority queue = Q

Huffman (c)
n = |c|
Q=c
for i =1 to n-1
do z = Allocate-Node ()
x = left[z] = EXTRACT_MIN(Q)
y = right[z] = EXTRACT_MIN(Q)
f[z] = f[x] + f[y]
INSERT (Q, z)
return EXTRACT_MIN(Q)
 Analysis
Q implemented as a binary heap.
line 2 can be performed by using BUILD-HEAP (P. 145; CLR) in O(n) time.
FOR loop executed |n| - 1 times and since each heap operation requires O(lg n) time.
=> the FOR loop contributes (|n| - 1) O(lg n)
=> O(n lg n)
Thus the total running time of Huffman on the set of n characters is O(nlg n).

C. Minimum Spanning Tree

A spanning tree is a connected graph using all vertices in which there is no cycle
A spanning tree of a graph is a tree that has all the vertices of the graph connected by some edges
A graph can have one or more number of spanning trees
If the graph has N vertices then the spanning tree will have N-1 edges
Greedy spanning tree algorithm
One of the most elegant spanning tree algorithm that I know of is as follows:
Examine the edges in graph in any arbitrary sequence.
Decide whether each edge will be included in the spanning tree.
Minimum Spanning Tree Algorithm
Perform the spanning tree algorithm (above) by examining the edges is order of non decreasing
weight (smallest first, largest last). If two or more edges have the same weight, order them
arbitrarily.
Maximum Spanning Tree Algorithm
Perform the spanning tree algorithm (above) by examining the edges in order of non increasing
weight (largest first, smallest last). If two or more edges have the same weight, order them
arbitrarily.
Minimum Spanning Tree
A minimum spanning tree (MST) of a weighted graph G is a spanning tree of G whose edges sum is minimum
weight.
In other words, a MST is a tree formed from a subset of the edges in a given undirected graph, with two
properties:
it spans the graph, i.e., it includes every vertex of the graph.
it is a minimum, i.e., the total weight of all the edges is as low as possible.

D. Kruskals Algorithm
In kruskal's algorithm the selection function chooses edges in increasing order of length without worrying
too much about their connection to previously chosen edges, except that never to form a cycle
The result is a forest of trees that grows until all the trees in a forest (all the components) merge in a single
tree.

E. Prims Algorithm
This algorithm was first propsed by Jarnik, but typically attributed to Prim.
It starts from an arbitrary vertex (root) and at each stage, add a new branch (edge) to the tree already
constructed; the algorithm halts when all the vertices in the graph have been reached. This strategy is
greedy in the sense that at each step the partial spanning tree is augmented with an edge that is the
smallest among all possible adjacent edges.
MST-PRIM
Input: A weighted, undirected graph G=(V, E, w)
Output: A minimum spanning tree T.
T={}
 Let r be an arbitrarily chosen vertex from V.
U = {r}
WHILE | U| < n
DO
Find u in U and v in V-U such that the edge (u, v) is a smallest edge between U-V.
T = TU{(u, v)}
U= UU{v}
Analysis
The algorithm spends most of its time in finding the smallest edge. So, time of the algorithm basically
depends on how do we search this edge.
Straightforward method
Just find the smallest edge by searching the adjacency list of the vertices in V. In this case, each iteration
costs O(m) time, yielding a total running time of O(mn).
Binary heap
By using binary heaps, the algorithm runs in O(m log n).
Fibonacci heap
By using Fibonacci heaps, the algorithm runs in O(m + n log n) time.

F. Dijkstra's Algorithm (Shortest Path)

Problem: Determine the length of the shortest path from the source to each of the other nodes of the
graph. This problem can be solved by a greedy algorithm often called Dijkstra's algorithm.

The algorithm maintains two sets of vertices, S and C. At every stage the set S contains those vertices that have
already been selected and set C contains all the other vertices. Hence we have the invariant property V=S U C. When
algorithm starts Delta contains only the source vertex and when the algorithm halts, Delta contains all the vertices of
the graph and problem is solved. At each step algorithm choose the vertex in C whose distance to the source is least
and add it to S.

2. DIVIDE AND CONQUER

A top-down technique for designing algorithms that consists of dividing the problem into smaller
subproblems hoping that the solutions of the subproblems are easier to find and then composing the partial
solutions into the solution of the original problem
Little more formally, divide-and-conquer paradigm consists of following major phases:
Breaking the problem into several sub-problems that are similar to the original problem but smaller
in size,
Solve the sub-problem recursively (successively and independently), and then
Combine these solutions to subproblems to create a solution to the original problem.
Binary search algorithm
Given A[1n] & key K call BinSrch(A,1,n,K)
Algorithm BinSrch(A, i, j, K)
if i = j then
if (A[i] = K then return I;
else return -1;
else
mid (i+j) / 2
if A[mid] = K return mid;
else if A[mid] > K then
Return BinSrch(A, i, mid 1, K);
Else return BinSrch(A, mid + 1, j, K);
Iterative version of binary search
Algorithm BinSrch(A[1..n], K0
Low 1; high n;
While (low < high) do
mid (low + high) / 2;
if A[mid] > K then
high mid 1;
else if A[mid] < K then
low mid + 1
else return mid;
return -1;

3. DYNAMIC PROGRAMMING
Dynamic programming is a fancy name for using divide-and-conquer technique with a table.
2 Big ideas of Dynamic Program
Recursive substructure
o T(n) = T(n-1) + T(n-2)
o The optimization problem that were trying to solve has some sort of a recursive substructure. The
solution of the problem of size n can be expressed as some function of the solution of smaller
problems.
Memoization
o Keep track of intermediate results, and going 1 step further, we solve the intermediate problems in a
specific order to maximize efficiency
The dynamic programming is a paradigm of algorithm design in which an optimization problem is solved by a
combination of caching subproblem solutions and appealing to the "principle of optimality.
Optimal Substructure simply means solving larger problems given the solutions to its smaller sub problems
There are three basic elements that characterize a dynamic programming algorithm:
1. Substructure
Decompose the given problem into smaller (and hopefully simpler) subproblems. Express the solution of the
original problem in terms of solutions for smaller problems. Note that unlike divide-and-conquer problems,
it is not usually sufficient to consider one decomposition, but many different ones.
2. Table-Structure
After solving the subproblems, store the answers (results) to the subproblems in a table. This is done
because (typically) subproblem solutions are reused many times, and we do not want to repeatedly solve
the same problem over and over again.
3. Bottom-up Computation
Using table (or something), combine solutions of smaller subproblems to solve larger subproblems, and
eventually arrive at a solution to the complete problem. The idea of bottom-up computation is as follow:
Bottom-up means
Start with the smallest subproblems.
Combining theirs solutions obtain the solutions to subproblems of increasing size.
Until arrive at the solution of the original problem.

A. Matrix-Chain Multiplication
The chain matrix multiplication problem is perhaps the most popular example of dynamic programming used
in the upper undergraduate course (or review basic issues of dynamic programming in advanced algorithm's
class).
 The chain matrix multiplication problem involves the question of determining the optimal sequence for
performing a series of operations. This general class of problem is important in complier design for code
optimization and in databases for query optimization.
Dynamic Programming Approach
Step 1: Check if the problem has Optimal Substructure
If we have an optimal solution for Ai.j
Assume the solution has the following parentheses: (Aik)(Ak+1j)
If there is a better way to multiply (Aik), then we would have a more optimal solution.
This would be a contradiction, as we already stated that we have the optimal solution for Ai.j
Therefore this problem has optimal substructure
Recursive Formula
M[i,j] = 0 if i=j
mini<k<j { M[i,j] =M[i,k] + M[k+1,j] + pi-1pkpj }

Matrix-Chain DP Algorithm
Matrix-Chain(array p[1 .. n], int n) {
Array s[1 .. n 1, 2 .. n];
FOR i = 1 TO n DO m[i, i] = 0; // initialize
FOR L = 2 TO n DO { // L=length of subchain
FOR i = 1 TO n L + 1 do {
j = i + L 1;
m[i, j] = infinity;
FOR k = i TO j 1 DO { // check all splits
q = m[i, k] + m[k + 1, j] + p[i 1] p[k] p[j];
IF (q < m[i, j]) {
m[i, j] = q;
s[i, j] = k;
}
}
}
}
return m[1, n](final cost) and s (splitting markers);
}

B. Knapsack Problem DP Solution

Dynamic-0-1-knapsack (v, w, n, W)
FOR w = 0 TO W
DO c[0, w] = 0
FOR i=1 to n
DO c[i, 0] = 0
FOR w=1 TO W
DO IFf wi w
THEN IF vi + c[i-1, w-wi]
THEN c[i, w] = vi + c[i-1, w-wi]
ELSE c[i, w] = c[i-1, w]
ELSE
c[i, w] = c[i-1, w]
C. Activity Selection Problem DP Solution
An activity-selection is the problem of scheduling a resource among several competing activity.
Problem Statement Given a set S of n activities with and start time, Si and fi, finish time of an ith activity.
Find the maximum size set of mutually compatible activities.
Compatible Activities
Activities i and j are compatible if the half-open internal [si, fi) and [sj, fj) do not overlap, that is, i and j are
compatible if si fj and sj fi
Dynamic-Programming Algorithm
The finishing time are in a sorted array f[i] and the starting times are in array s[i]. The array m[i] will store the
value mi, where mi is the size of the largest of mutually compatible activities among activities {1, 2, . . . , i}.
Let BINARY-SEARCH(f, s) returns the index of a number i in the sorted array f such that f(i) s f[i + 1].
for i =1 to n
do m[i] = max(m[i-1], 1+ m [BINARY-SEARCH(f, s[i])])
We have P(i] = 1 if activity i is in optimal selection, and P[i] = 0
otherwise
i=n
while i > 0
do if m[i] = m[i-1]
then P[i] = 0
i=i-1
else
i = BINARY-SEARCH (f, s[i])
P[i] = 1
Analysis
The running time of this algorithm is O(n lg n) because of the binary search which takes lg(n) time as
opposed to the O(n) running time of the greedy algorithm. This greedy algorithm assumes that the activities
already sorted by increasing time.

Source:
http://www.personal.kent.edu/~rmuhamma/Algorithms/MyAlgorithms/