CSE 548: Analysis of Algorithms

Lecture 3
( Divide-and-Conquer Algorithms:
Matrix Multiplication )

Rezaul A. Chowdhury
Department of Computer Science
SUNY Stony Brook
Fall 2012
Iterative Matrix Multiplication
n
zij = ∑x
k =1
ik y kj

Iter-MM ( Z, X, Y ) { X, Y, Z are n × n matrices,

where n is a positive integer }
1. for i ← 1 to n do
2. for j ← 1 to n do
3. Z[ i ][ j ] ← 0

4. for k ← 1 to n do
5. Z[ i ][ j ] ← Z[ i ][ j ] + X[ i ][ k ] ⋅ Y[ k ][ j ]
Recursive ( Divide & Conquer ) Matrix Multiplication
Z X Y
n/2 n/2 n/2

n/2 Z11 Z12 n/2 X11 X12 n/2 Y11 Y12

n = n
× n

Z21 Z22 X21 X22 Y21 Y22

n n n

n/2

n/2 X11 Y11 + X12 Y21 X11 Y12 + X12 Y22

= n

X21 Y11 + X22 Y21 X21 Y12 + X22 Y22

n
Recursive ( Divide & Conquer ) Matrix Multiplication
Rec-MM ( X, Y ) { X and Y are n × n matrices,
where n = 2k for integer k ≥ 0 }

1. Let Z be a new n × n matrix

2. if n = 1 then
3. Z←X⋅Y
4. else

5. Z11 ← Rec-MM ( X11, Y11 ) + Rec-MM ( X12, Y21 )

6. Z12 ← Rec-MM ( X11, Y12 ) + Rec-MM ( X12, Y22 )

7. Z21 ← Rec-MM ( X21, Y11 ) + Rec-MM ( X22, Y21 )

8. Z22 ← Rec-MM ( X21, Y12 ) + Rec-MM ( X22, Y22 )

9. endif

10. return Z

# recursive matrix products: 8 Θ 1 ,    1,

  
# matrix sums: 4 8
 Θ  , .
2
 Θ 
Strassen’s Algorithms for Matrix Multiplication ( MM )

In 1968 Volker Strassen came up with a recursive MM algorithm that

runs asymptotically faster than the classical Ο  algorithm.
In each level of recursion the algorithm uses:
7 recursive matrix multiplications ( instead to 8 ), and
18 matrix additions ( instead of 4 ).
Strassen’s MM: 10 Matrix Additions/Subtractions

    

     

               

    

     

               

 Strassen’s MM: 7 Matrix Products

X11 X12 X21 X22 X11 X12 X21 X22 X11 X12 X21 X22
Y11 Y11 Y11 + _
Y21 Y21 Y21
Y12 +_ Y12 Y12 + _
Y22 Y22 ++ Y22

   ∙     ∙     ∙ 

X11 X12 X21 X22
Y11 + +
Y21
Y12
Y22 + +
X11 X12 X21 X22 X11 X12 X21 X22 X11 X12 X21 X22    ∙ 
Y11
_+ Y11 ++ Y11
Y21 Y21 Y21 + _
Y12 Y12 Y12
Y22 Y22 Y22 + _
   ∙     ∙     ∙ 
Strassen’s MM: 8 More Matrix Additions/Subtractions

Z11 Z12 X11 X12 Y11 Y12

= ×
Z21 Z22 X21 X22 Y21 Y22

X11 Y11 + X12 Y21 X11 Y12 + X12 Y22

=
X21 Y11 + X22 Y21 X21 Y12 + X22 Y22

( Pd1 - Pr1 )
Pr1 + P11
– ( P22 – Pc2 )
= ( Pd1 – Pr2 )
Pr2 – P22
+ ( P11 – Pc1 )
 Strassen’s Matrix Multiplication

P11 P22 Pr1 Pr2 Pc1 Pc2 Pd1

X11 X12 X21 X22
_ +
_ +
Y11

Z11
Y21 + +
Y12
Y22 __ + _ + +
X11 X12 X21 X22
Y11
Y21
Z12
_+
Y12
Y22 ++
X11 X12 X21 X22
Y11
_ ++
Z21
Y21 +
Y12
Y22

X11 X12 X21 X22

Y11
__ _ + + +
Y21
Z22 _
Y12
Y22
_+ +
+ +
 Strassen’s Matrix Multiplication

Sums:
         
           
           
         
         
Running Time:
Products:
Θ 1 ,    1,
   ∙     ∙ 
  
7
 Θ  , .
   ∙     ∙  2
   ∙     ∙ 
 Θ !"#$ %  Ο  .&
   ∙ 
 Deriving Strassen’s Algorithm
' ( 0 0  0
' (  0 1 ) * 0 0 -  

) * -    ⇒ 0 0 ' ( 1
0 0 ) * .
 
2
, / 3

We will try to minimize the number of multiplications needed to

evaluate 4 using special matrix products that are easy to compute.
Type Product #Mults
' (  '  (-

(·) ) * - )  *- 4

' '  ' -


(A) ' ' - ' -
1

' '  ' -


' ' - '   -
(B) 1

' 0  '
 '  ( -  
'( ( -
(C) 2

' ('  '   -  (

- 
(D) 0 ( ( 2
 Deriving Strassen’s Algorithm
' ( 0 0 ( ( 0 0 '( 0 0 0
) * 0 0  ( ( 0 0  )( *( 0 0
0 0 ' ( 0 0 0 0 0 0 ' (
0 0 ) * 0 0 0 0 0 0 ) *
5678 9 :  ;<=> ? ∆@
0 0 0 0 '( 0 0 0
∆  0 0 0 0  )( *( 0 0
0 0 ) ) 0 0 '  ) (  )
0 0 ) ) 0 0 0 *)
5678 9 :  ;<=> ? ∆$
0 0 0 0 '( 0 0 0
∆  )  ( 0 0 )  (  0 *( 0 ()
 )( 0 0  )( )  ( 0 ') 0
0 0 0 0 0 0 0 *)
5678 A :  ;<=> ? ∆B

'  ( 0 0 0 0 0 0 0
0 0 0 0  0 *( 0 *)  *(
∆ 
'  (  '  ) 0 '  ) 0 0 0 0 0
0 0 0 0 0 0 0 *  )
5678 C : ;<=> ? 5678 D : ;<=> ?
 Algorithms for Multiplying Two n×
×n Matrices
A recursive algorithm based on multiplying two I J I matrices
using K multiplications will yield an Ο !"#L M algorithm.
To beat Strassen’s algorithm: log Q K R log 7⇒ K R I!"#$ % .
So, for a 3 J 3 matrix, we must have: K R 3!"#$ % R 22.
But the best known algorithm uses 23 multiplications!
Inventor Year Complexity
Classical ― Θ 
Volker Strassen 1968 Θ  .&E%
Victor Pan
( multiply two 70 × 70 matrices using 1978 Θ  .%FG
143,640 multiplications )

Θ  .%%
Don Coppersmith & Shmuel Winograd
1990
( arithmetic progressions )
Andrew Stothers 2010 Θ  .%H
Virginia Williams 2011 Θ  .% %

Lower bound: Ω  ( why? )