Amit Shekhar ID: 108266469 NetID: ashekhar CSE 548 Analysis of Algorithms Solution to Assignment #2 Due Thursday, October

r 27, 2011 Discussed at Oce hours and with Ayon Chakraborty and Amit Arya

Problem 1 In class we saw how to construct circuits for addition of two n-bit numbers that have depth of O(log n) and O(1). In this problem we look at how to design circuits for multiplying. Note: You may assume that the size of an addition circuit (both for standard and nonunique representation) is (n). (a) Design a na circuit that has depth O(log2 n). What is the size of this circuit? Explain ve how to compute the size. Solution: From the class, we establish that circuit for Addition of two n-bit numbers has depth of log n using standard representation. The recurrence relations for n-bit addition: = O(log n) Depth: T (n) = T ( n ) + 1 2 n Size: S(n) = 3S( 2 ) + n = ( 3 )log n n 2 = O(nlog 3 ) where n is the task done at Multiplexer and 3 came from the decision to compute more signicant half twice. Once when C n = 1 and another time when C n = 0 2 2

Here, we seek multiplication of two n-bit numbers, which is denitely an addition of n numbers of 2n bits each. This is an adaptation of the simple mathematical multiplication. Here is how it is so. The length of each number becomes 2n because after each bit is multiplied with the multiplicand, the resultant (intermediate) number is shifted towards left by one bit. Since we are multiplying n bits, we must have shifted one of the intermediate result to n bits to the left. Thereby, making the length of each number to be added to 2n. Please note the extra bits can be padded. Note that we have already established that circuit for Addition of two n-bit numbers has depth of (log n) Method: Now we take each intermediate result in pair and add them. repeat this process till we get the nal value. This is analogous to MERGE part of a divide and conquer algorithm. This gives the depth of circuit is log n for this part. Hence, the depth of entire circuit becomes O(log2 n) Size Computation:

If we assume O(nlog 3 ) to be equal to (n) then Size would be O(n2 ) as there can be n nodes for the tree shown. Each such node requires (n) of circuit size. (b) Design a better circuit that has depth O(log n). What is the size of this circuit? Explain how to compute the size. Solution: From the class, we establish that circuit for Addition of two n-bit numbers has constant depth using Non Unique representation. The recurrence relations for n-bit addition: Depth: T (n) = O(1) Size: T (n) = O(n) Here, we seek multiplication of two n-bit numbers, which is denitely an addition of n numbers of 2n bits each. This is an adaptation of the simple mathematical multiplication. Here is how it is so. The length of each number becomes 2n because after each bit is multiplied with the multiplicand, the resultant (intermediate) number is shifted towards left by one bit. Since we are multiplying n bits, we must have shifted one of the intermediate result to n bits to the left. Thereby, making the length of each number to be added to 2n. Please note the extra bits can be padded. we have already established that circuit for Addition of two n-bit numbers has constant depth. Method: Now we take each intermediate result in pair and add them. and repeat this process till we get the nal value. This is analogous to MERGE part of a divide and conquer algorithm. This gives the depth of circuit is log n for this part. Hence, the depth of entire circuit becomes O(log n) Size Computation: for n nodes once again we get the circuit size as O(n2 )

Problem 2 In this problem, we continue working on multiplication circuits. The objective is to reduce the overall size of the multiplication circuits to something that is o(n2 ). Hint: please use Problem 5 of Problem Set 1 as a guide. (a) Please describe your construction algorithm and describe the depth of the circuit. Solution: The number can be written as: X = 2m a + b and Y = 2m c + d Multiplication requires: Z = 22m ac + 2m (bc + ad) + bd We see that (bc + ad) can be represented by (a + b)(c + d) (ac + bd). This reduces the number of multiplication circuit by 1. As multiplication can be performed by using circuits that can perform multiplication of (a + b)(c + d), ac, bd only. The image below shows the abstracted circuit of multiplication in this way:

(b) What is the recurrence relation for the size of your circuit? Solution: The recurrence relation: T (n) = 3T ( n ) + O(n) 2 (c) Please solve this recurrence relation. Solution: From Recursion Tree:

Work at level 1 n n Work at level 2 31 21 . . . . . . Work at level n 3n 2n n

Depth of the tree = log2 n Therefore, total work = n ( 3 )i i= 0 . . . log2 n 2 The terms are in Increasing geometric series hence, the last term would dominate. This also means contribution from leaves will dominate. At the leaf level contribution is T (1) = (1) (by each leaf). The number of leaves is 3log2 n nlog2 3 Therefore, T (n) = (nlog2 3 ) Alternatively, This recurrence relation falls under case 1 of Master Theorem where T (n) = aT ( n ) + f (n) b with a = 3, b = 2, = 1, and f (n) = n logb a ) The solution is: T (n) = O(n = O(nlog2 3 )

Problem 3 Imagine an innitely large array A[0 . . . ]. In this array, A[0] = A[1] = . . . = A[n 1] = 0, and A[n] = A[n + 1] = A[n + 2] = . . . = 1. That is, the rst n elements of the array are 0 and all subsequent elements are 1. In constant time we can query the array to nd the value of an array element A[i]. We do not know the value of n, the index of the rst nonzero element. A = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1... (1) Describe an algorithm to nd the value of n. This algorithm should make at most O(log n) queries. Solution: We query elements whose index is given by 2i where i ranges from 0 . . . log n. Here, its possible that we may bump into the part where there are only 1s. In that case, we should return back to immediately previous point [the index at 2( logn 1) in the array] and then we should start repeating above thing from that index till we reach the rst 1. Please note that rst occurrence of 1 is at 2( i 1) < n < 2i and i = log n At max O(log n) queries. (2) Describe an algorithm for nding any index m, where A[m] = 1 and m < nc for some constant c > 1. Show that this algorithm requires O(log log n) array queries. Solution: we solve this problem in the same way as above except that we move forward in the array at a step of 2( 2i ) where ranges from 0 . . . log log n Proof for the number of queries required: i Here, we step by 22 to reach to the part where there are all 1s. Hence, to reach n we step log log n and query at each step whether we have reached at the required place. (3) Describe an algorithm for nding any m, where A[m] = 1. How fast can you make this algorithm? Solution: i 22 Once again, we can go forward to raise the step. The step can be any suitable 22 based upon inniteness of the size of array. Note that the level of step can be chosen based upon the value of n. If at all, it approaches to innity then we can increase the step value to a big number. We can do this in O(log n) and some constant time where we return back and check for the rst 1.

Problem 4 Consider the knapsack problem discussed in class: For set S = {s1 , s2 , , sn } of integers and target integer K, is there a subset T S, such that ti = K?
ti T

(1) Recall the algorithm from class. Provide the subproblems, then present pseudocode from class. Solution: Sub problem: For every possible value of target integer K, we are going to check what subset of S can t in properly. In order to check for any intermediate K from 0 . . . K we try the entire set S. Though, DP technique helps us to compute most of the bigger values using solution to smaller subproblem. for all k = 0 . . . K for all i = 1 . . . N Knapsack problem for set {s1 , s2 , , si } Pseudocode: Initialization: for all k = 0 . . . K B[0, k] = 0 B[0, 0] = 0 Loop Assignment: for all i = 1 . . . N for all k = 0 . . . K B[i, k] = B[i 1, k] B[i 1, k si ] Goal Cell: Return B[n, k]

(2) The algorithm from (1) only answers the question with a yes or no. What part of the algorithm would we modify so that we can compute the set T? What are the modications? Solution: Pseudocode: Initialization: for all k = 0 . . . K B[0, k] = 0 T AB[0, k] = 0 table for encoding regions for all i = 0 . . . n T AB[i, 0] = 0

B[0, 0] = 0 for all i = 1 . . . N for all k = 0 . . . K if(B[i 1, k si ] > B[i 1, k]) B[i, k] = B[i 1, k si ] T AB[i, k] = 1 else B[i, k] = B[i 1, k] T AB[i, k] = 0 The construction of subset from the Tab Matrix. for all i = n . . . 1 if(T AB[i, K] == 1) Print i th element from set. K = K si Return B[n, k]

Problem 5 Problem from Steve Skiena Consider the problem of storing n books on shelves in a library. The order of the books is xed by the cataloging system and so cannot be rearraged. Therefore, we can speak of a book bi , where 1 i n, that has a thickness ti and height hi . The length of each bookshelf at this library is L. Suppose all the books have the same height h (i.e. h = hi = hj for all i, j) and the shelves are all separated by a distance of greater than h, so any book ts on any shelf. The greedy algorithm would ll the rst shelf with as many books as we can until we get the smallest i such that bi does not t, and then repeat with subsequent shelves. Show that the greedy algorithm always nds the optimal shelf placement, and analyze the time complexity. Solution: Problem : Show that the greedy algorithm always nds the optimal shelf placement Proof by contradiction. There are n books to be stored. Book bi has a thickness ti , where 1 i n, . All the books have the same height h (i.e. h = hi = hj for all i, j). The length of each book shelf at the library is L and the shelves are all separated by a distance of greater than h, so any book ts on any shelf. Assume, there exists an optimal way other than greedy, and we can arrange the books such that more number of books t in a shelf. This infers, there should be some space left by the greedy method because height is constant and order of books cannot be changed. So, even the alternate solution ends up lling the rst shelf with as many books as it can until it get the smallest i such that bi does not t, and then repeat with subsequent shelves. Again because of the condition that height is constant and order of books cannot be changed. Hence we conclude that greedy method is the same as the alternate optimal one. The Recurrence relations is as follows : T(n) = = T (n 1) + c = (T (n 2) + c) + c = ((T (n 3) + c) + c) + c . . . = T (n k) + k c T (n) is constant for n = 1 after a depth of n 1. Hence, the running time = O(n)

Problem 6 Problem from Steve Skiena This is a generalization of the previous problem. Now consider the case where the height of the books is not constant, but we have the freedom to adjust the height of each shelf to that of the tallest book on the shelf. Thus the cost of a particular layout is the sum of the heights of the largest book on each shelf. (1) Give an example to show that the greedy algorithm of stung each shelf as full as possible does not always give the minimum overall height. Solution: Consider books a,b,c,d having widths 3,6,3,6,3 and heights 11,14,17,5,6 respectively. Let the width of the shelf be 13. In such a case, the Greedy Algorithm will order the books as: Shelf 1: a, b max height = 14 Shelf 2: c, d max height = 17 Shelf 3: e max height = 6 Shelf height will be = 37. However, if we order the books as: Shelf 1: a max height = 11 Shelf 2: b, c max height = 17 Shelf 2: d, e max height = 6 Hence, total height of all shelves = 34, which is better than the answer given by Greedy Algorithm. Thus, Greedy Algorithm does not always give optimal solution for minimizing the overall height of the shelves. (2) What technique should we use to solve this problem? Solution: Using Technique of dynamic programming, we can solve this problem. (3) What are the subproblems? Solution: For a height, all the books that can t in one self. Here, a possible set can start anywhere in the list of books. So, a subproblem may contain a list of books bi , bi + 1 . . . bj i < jandi, j n (4) How many subproblems are there? Solution: Upper limit for the number of subproblems can be nL. For every Shelf, we may have n sub problems. (5) Give an algorithm for this problem, and analyze its time complexity. Solution: Pseudo code: Initialize: OPT[][] be the DP memoization matrix initialized with a large value. Loop and sub problem solving using the previously solved smaller problem

for all l = 0 . . . L for all k = 1 . . . n 1 if ((summation of thickness of all books from k . . . k + l) Book Shelf length) OP T [k, k + l] = max{OP T (k, k + l 1), hk+l } else Current subproblem doesnt t in. OP T [k, k + l] = very large value. Goal Cell: OP T [1, n] Time complexity O(nL)

Problem 7 Suppose you are given three strings of characters: X, Y, and Z, where |X| = n, |Y| = m, and |Z| = n+m. Z is said to be a shue of X and Y i Z can be formed by interleaving the characters from X and Y in a way that maintains the left-to-right ordering of the characters from each string. (a) Show that cchocohilaptes is a shue of chocolate and chips, but chocochilatspe is not. Solution:

(b) Give an ecient dynamic-programming algorithm that determines whether Z is a shue of X and Y. Solution: When we try to make the underlying DAG (Directed acyclic Graph) here, we see that each node is being approached from either of the two ways. For any (i, j)th element it is reachable either from (i, j 1)th node or (i 1, j)th node. If we start traversing from (0, 0) to the end (m, n) by above mentioned ways and we actually reach to the end without violating any rule, then were done and the 3rd string is a shue. Otherwise, the 3rd string is not a Shue. Pseudo code: A is the matrix that stores solution to previously computed smaller subproblems for all i = 0 . . . M and j = 0 . . . N if (X[i] == Z[i + j]) A[i][j] = A[i][j 1] i1+1 else if (Y [j] == Z[i + j]) A[i][j] = A[i 1][j] j j+1 else Return NOT A SHUFFLE! Return A SHUFFLE!

Hint: The values the dynamic programming matrix you construct should be Boolean, not numeric.

Problem 8 Fun, challenging problem There are N soldiers lined up, about to execute a hapless prisoner. The lieutenant, who begins the ring process, is located at one end of the line. The soldiers all want to re simultaneously. Unfortunately, the soldiers can only talk to their immediate neighbors on the left and right. In addition, these soldiers are not very intelligent and only have constant memory, independent of N . Notice that constant memory is very little. It is not enough even to count up to N ; this requires log(n) bits. It is not enough to have a name, because unique names also require log(n) bits. Devise an algorithm that allows these soldiers to re in unison. The algorithm should have running time O(n). Solution: Here, the soldiers can only talk to their immediate neighbors on the left and right. This fact gives us a way to solve. And we must devise a way such that everyone gets to hear the signal before shooting. Every time, a soldier at the left most position can denitely talk with someone in the mid of the row. Firstly, devise a way how a soldier can talk to another soldier that is in the middle of the row i.e, how to identify mid of the row. Here we can use basic reection property. Say, soldier on the left sends two signals towards right. One of the signal travels 3x faster than another. So, when the slower signal reaches the mid, faster signal must also be reaching the mid point after traveling to the end. So, this gives us the mid point for any row of any length. Now, In order to enable everyone to signal we must start with the communication with rst element on the left and the mid of the row. The above mechanism can be employed to do this. Next, we have two sub-list of equal size Divided by the mid that we found just now. Now, by treating the new sub-arrays the same way as we did to the original row we can make communication happen between 1st soldier on the left and the soldier standing in the mid of 1st sub-array. When we do it recursively to the end when each soldier talks to only its neighbor, we are in a position to make each soldier talk in Unison. Now the running cost analysis: The rst level communication requires 2n time The next level communication requires 2n time 2 . . . The subsequent level communication requires 2n time 2l Hence the total work required still remains linear i.e, O(n)

Note that they move and interact at exactly the same speed; i.e., they are synchronized. Hint: Use divide and conquer. You should assume soldiers all operate at the same speed.

Problem 9 In class we saw how to analyze divide and conquer matrix multiplication of two N N matrices. Now well show how to lay out the matrix to optimize for memory transfers (both using blocking and divide and conquer). Well use the same DAM model that we described in class and in the last problem set. (a) Suppose that we divide the matrix into blocks of size B (i.e., B by B). Prove that we can achieve O(N 3 /B 3/2 ) memory transfers for a multiplication. Solution: The recurrence relation for the following method is: T (n) = 8T ( n ) + O(n2 ) 2 Solving the above recurrence relation, we get T(n) = O(n3 ) Now, a Block of size B can hold ( B B) elements. And a block B can be considered now as a square sub matrix. 2 For a matrix of size N N , there can be ( N ) such blocks as B is the unit of transfer and the B N N counting transfers. This also infers that the N N matrix can now be considered as ( B B ) square matrix. Therefore, as per our solution to the recurrence relation, the required number of N multiplication will be: ( B )3 .
B2 (b) Suppose that we divide the matrix into blocks of size M (i.e., M by M ). Prove that we can achieve O(N 3 /B M ) memory transfers for a multiplication. Solution: Here, a Block of size M can hold ( M M ) elements. And the total number of multiplication N will be ( M )3 For each ( M M ) elements there will be M blocks. And the memory transfers will be B N O(( M )3 M ) B N Upon simplication this becomes: O( BM )
3

In Big Oh notation itll be O( N 3 )

Problem 10 (challenging problem, but good practice for understanding divide and conquer well) Now lets explore divide and conquer solutions for matrix multiplication (as we discussed in class). We want to multiply A and B (each of size N N ) and the product will be stored in C: A= A1 A2 A3 A4 B= B1 B2 B3 B4 C= C1 C2 C3 C4

The divide and conquer solution from class is: A1 A2 A3 A4 B1 B2 B3 B4 = A1 B1 + A2 B3 A1 B2 + A2 B4 A3 B1 + A4 B3 A3 B2 + A4 B4

To ensure good memory locality, well store the matrix in the same divide and conquer order, e.g., A1, A2, A3, A4. Thus the matrix is stored in a zig-zag manner in memory. (a) What is the recursion relation for the cost of a multiplication? Solution: The recurrence relation for the cost of multiplication: T (n) = 8T ( n ) + O(n2 ) 2 (b) Solve the recursion relation and prove that the divide-and-conquer multiplication achieves O(N 3 /B M ) memory transfers. Solution: Solving the recurrence relation, we get T(n) = O(n3 ) Proof: Now, a Block of size B can hold ( B B) elements. And a block B can be considered now as a square sub matrix. 2 For a matrix of size N N , there can be ( N ) such blocks as B is the unit of transfer and the B N N counting transfers. This also infers that the N N matrix can now be considered as ( B B ) square matrix. Therefore, as per our solution to the recurrence relation, the required number of 3 N multiplication will be: ( B )3 . In Big Oh notation itll be O( N 3 ) B2 Similarly, a Block of size M can hold ( M M ) elements. And the total number of multi N plication will be ( M )3 For each ( M M ) elements there will be M blocks. And the memory B N transfers will be O(( M )3 M ) B
N Upon simplication this becomes: O( BM )
3