Академический Документы
Профессиональный Документы
Культура Документы
LU Factorization
Kristin Davies
Peter He
Feng Xie
Hamid Ghaffari
April 4, 2007
Outline
Introduction to LU Factorization (Kristin)
LU Update (Hamid)
Introduction – Transformations – Sparsity – Simplex – Implementation
A = LU
Example for a 3x3 matrix:
Transformation Algorithms
Modified form of Gaussian elimination
Transformations – Doolittle
Doolittle factorization – L has 1’s on its diagonal
General algorithm – determine rows of U from top to
bottom; determine columns of L from left to right
for i=1:n
for j=i:n
LikUkj=Aij gives row i of U
end
for j=i+1:n
LjkUki=Aji gives column i of L
end
end
Introduction – Transformations – Sparsity – Simplex – Implementation
l31 l32
1 0 0 u33 − 4 − 2 8 end
u11
(1 0 0) 0 = 2 ⇒ u11 = 2
0
Similarly
1 0 0 u11 u12 u13 2 − 1 − 2
l 0 0 u 22 u23 = − 4 6 3
21 1
l31 l321 0 0 u33 − 4 − 2 8
⇒ u12 = −1 , u13 = −2
Introduction – Transformations – Sparsity – Simplex – Implementation
2
(l21 1 0) 0 = −4 ⇒ 2l21 = −4 ⇒ l21 = −2
0
Similarly
1 0 0 2 − 1 − 2 2 − 1 − 2
l 0 0 u 22 u23 = − 4 6 3
21 1
l31 l321 0 0 u33 − 4 − 2 8
⇒ l31 = −2
Introduction – Transformations – Sparsity – Simplex – Implementation
3rd row of U
1st row of U
1 0 0 2 − 1 − 2 2 − 1 − 2 1 0 0 2 − 1 − 2 2 − 1 − 2
l 0 0 u22 u23 = − 4 6 3 − 2 1 0 0 4 − 1 = − 4 6
21 1 3
l31 l32 1 0 0 u33 − 4 − 2 8 − 2 − 1 1 0 0 u33 − 4 − 2 8
2nd coln of L
1st coln of L
1 0 0 2 − 1 − 2 2 − 1 − 2 1 0 0 2 − 1 − 2 2 − 1 − 2
− 2 1 0 0 u22 u23 = − 4 6 3 − 2 1 0 0 4 − 1 = − 4 6 3
− 2 l32 1 0 0 u33 − 4 − 2 8 − 2 l32 1 0 0 u33 − 4 − 2 8
2nd row of U
Introduction – Transformations – Sparsity – Simplex – Implementation
Transformations – Crout
Crout factorization – U has 1’s on its diagonal
General algorithm – determine columns of L from left to
right; determine rows of U from top to bottom (same!?)
for i=1:n
for j=i:n
LjkUki=Aji gives column i of L
end
for j=i+1:n
LikUkj=Aij gives row i of U
end
end
Introduction – Transformations – Sparsity – Simplex – Implementation
1
(l11 0 0 ) 0 = 2 ⇒ l11 = 2
0
Similarly
l11 0 0 1 u12 u13 2 − 1 − 2
l 0 0 1 u 23 = − 4 6 3
21 l22
l31 l32l33 0 0 1 − 4 − 2 8
⇒ l21 = −4 , l31 = −4
Introduction – Transformations – Sparsity – Simplex – Implementation
Transformations – Solution
Once the L and U matrices have been found, we can
easily solve our system
AX=B AUX=B
LY=B UX=Y
l11 0 0 y1 b1 u11 u12 u13 x1 y1
l 0 y2 = b2 0 u u23 x2 = y2
21 l22 22
row 1 1 2 2 4 4
list = column 3 5 3 4 2 3
value 3 4 5 7 2 6
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of
Sparse Matrix (square)
A ↔ G(A) ↔ Adj(G)
i
Adj(G) ∈ {0,1}
j
Matrix P is of permutation and G(A) = G(PT AP)
Structure Decomposition of
Sparse Matrix (square)
Structure Decomposition of
Sparse Matrix (square)
Structure Decomposition of
Sparse Matrix (square)
Structure Decomposition of
Sparse Matrix (square)
t
= i1 i2t . . . t
in
t
{
(j ∈ St ⊂ 1 2 . . . n , }
St is closed on strong connection. )
Step 2: Choose the j1-th column ( j1 ≠ j ),
j = j1 , t=t+1, returning step 1.
After p times,
{1,2,...,n} = S1 ∪ S2 ∪... ∪ Sp
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of
Sparse Matrix (square)
Aˆ 11 Aˆ 12 . . . Aˆ 1p
Aˆ Aˆ 22 . . . Aˆ 2p
= P adj(G)P = .
T 21
Â
. . . . . .
Aˆ
Aˆ pp
p1 . . . .
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of
Sparse Matrix (square)
B = (bij p×
)∈R ,p
= 0 i= j
bij = = 1 Aˆ ij ≠ 0. ⇒
= 0 Aˆ ij = 0
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of
Sparse Matrix (square)
Structure Decomposition of
Sparse Matrix (square)
Structure Decomposition of
Sparse Matrix (General)
Dulmage-Mendelsohn Decomposition
∃P,Q such that
A
h x x
PT AQ = As X
A v
Introduction – Transformations – Sparsity – Simplex – Implementation
Structure Decomposition of
Sparse Matrix (General)
LU Factorization Method :
Gilbert/Peierls
LU Factorization Method :
Gilbert/Peierls
x = b;
for j = 1:n
if ≠(x(j) 0) , x(j+1:n) =
x(j+1:n) - L(j+1:n,j) * x(j);
end; ---Total time: O(n+flops)
Introduction – Transformations – Sparsity – Simplex – Implementation
x j ≠ 0 ∧ lij ≠ 0 ⇒ x i ≠ 0
bi ≠ 0 ⇒ x i ≠ 0
LetG(L) j →an
have i edge lij ≠ 0
if β = {i b ≠ 0}
i
χ = {i x i ≠ 0}
Let χ = Re achG(L ) (β) and
Then
---Total time: O(flops)
Introduction – Transformations – Sparsity – Simplex – Implementation
References
Simplex Method
– Problem size
greenbea.mps
of NETLIB
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Computational Form, Basis
Basic variables
x1 x2 x3 x4
1 0 1 2
0 1 2 1
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Notations
β index set of basic variables
γ index set of non-basic variables
B:=Aβ basis
R:=Aγ non-basis (columns)
xβ cβ
A = [ B | R], x = , c =
xγ cγ
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Basic Feasible Solution
Ax = b Bxβ + Rxγ = b
xβ = B −1 (b − Rxγ )
Simplex Method
– Checking Optimality
Objective value : cT x = cβT xβ + cγT xγ
= c0 + d T xγ
Simplex Method
– Improving Basic Feasible Solution
⋮ ⋮
xn = 0 xn ↑
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Improving Basic Feasible Solution
Choose xq (incoming variable) s.t. dq < 0 Objective value
increase xq as much as possible c0 + d T xγ
Simplex Method
– Basis Updating
Simplex Method
– Basis Updating B = b1 ,..., bp ,..., bm
m
Write a = ∑ v i bi = Bv ( v = B −1 a ) B = [b1 ,..., a,..., bm ]
i =1
Simplex Method
– Basis Updating B = b1 ,..., bp ,..., bm
m
Write a = ∑ v i bi = Bv ( v = B −1 a ) B = [b1 ,..., a,..., bm ]
i =1
1 vi
bp = p a − ∑ p bi ( v p ≠ 0)
v i≠ p v
p −1 p +1 T
v1
v 1 v v m
bp = Bη , where η = − p ,⋯ , − p , p , − p , ⋯, − p
v v v v v
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Basis Updating B = b1 ,..., bp ,..., bm
m
Write a = ∑ v i bi = Bv ( v = B −1 a ) B = [b1 ,..., a,..., bm ]
i =1
1 vi
bp = p a − ∑ p bi
v i≠ p v
p −1 p +1 T
v1
v 1 v v m
bp = Bη , where η = − p ,⋯ , − p , p , − p , ⋯, − p
v v v v v
vp : pivot element
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Basis Updating B = b1 ,..., bp ,..., bm
m
Write a = ∑ v i bi = Bv ( v = B −1 a ) B = [b1 ,..., a,..., bm ]
i =1
1 vi
bp = p a − ∑ p bi
v i≠ p v
p −1 p +1 T
v v 1
1 v v m
bp = Bη , where η = − p ,⋯ , − p , p , − p , ⋯, − p
v v v v v
Simplex Method
– Basis Updating B = b1 ,..., bp ,..., bm
m
Write a = ∑ v i bi = Bv ( v = B −1 a ) B = [b1 ,..., a,..., bm ]
i =1
1 vi
bp = p a − ∑ p bi
v i≠ p v
p −1 p +1 T
v v 1
1 v v m
bp = Bη , where η = − p ,⋯ , − p , p , − p , ⋯, − p
v v v v v
B −1 = EB −1
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Basis Updating
E = e1 , ..., e p −1 ,η , e p +1 ,..., e m
1 η1
ETM
⋱ (Elementary Transformation Matrix)
= ηp
⋱
ηm 1
1 0 1 η1 1 0
⋱ ⋮
⋱ ⋮
⋱ ⋮
= ηp 1 ⋯ 1
⋮ ⋱ ⋮ ⋱ ⋮ ⋱
0 1 0 1 ηm 1
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Basis Updating
Basis tends to get denser after each update ( B −1 = EB −1 )
1 η1 w1 w1 + w pη 1
⋱ ⋮ ⋮
Ew = ηp w p = w pη p
⋱ ⋮ ⋮
ηm 1 wm wm + w pη m
Ew = w if wp = 0
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Algorithm
Simplex Method
– Algorithm
Simplex Method
– Algorithm
In practice, compromise.
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– Typical Operations in Simplex Method
Typical operations : B −1w, wT B −1
Two ways:
−1
Product form of the inverse ( B = Ek Ek −1 ⋯ E1 ) (obsolete)
LU factorization
Introduction – Transformations – Sparsity – Simplex – Implementation
Simplex Method
– LU Factorization
Reduce complexity using LU update
( B = BE , B −1 = EB −1 )
Refactorization
(reinstate efficiency and numerical accuracy)
Sparse LU Updates in Simplex Method
Hamid R. Ghaffari
LU Update Methods
Preliminaries
Bartels-Golub LU UPDATE
Sparse Bartels-Golub Method
Reid’s Method
The Forrest-Tomlin Method
Suhl-Suhl Method
More Details on the Topic
Revised Simplex Algorithm
The first variant of the Revised Simplex Method was the Bartels-Golub
Method.
Bartels-Golub Method
Algorithm
I Can we do better?
Bartels-Golub Method
Characteristics
Single-Entry-Eta Decomposition:
1 1 1 1
l21 1 l21 1 1 1
= · l ·
l 1 1 1 1
31 31
l41 1 1 1 l41 1
Sparse Bartels-Golub Method
eta matrices
Single-Entry-Eta Decomposition:
1 1 1 1
l21 1 l21 1 1 1
= · l ·
l 1 1 1 1
31 31
l41 1 1 1 l41 1
I If the spike always occurs in the first column and extends to the
bottom row, the Sparse Bartels-Golub Method becomes worse than
the Bartels-Golub Method.
Sparse Bartels-Golub Method Disadvantages
I If the spike always occurs in the first column and extends to the
bottom row, the Sparse Bartels-Golub Method becomes worse than
the Bartels-Golub Method.
I The upper-triangular matrix will always be fully-decomposed
resulting in huge amounts of fill-in;
Sparse Bartels-Golub Method Disadvantages
I If the spike always occurs in the first column and extends to the
bottom row, the Sparse Bartels-Golub Method becomes worse than
the Bartels-Golub Method.
I The upper-triangular matrix will always be fully-decomposed
resulting in huge amounts of fill-in;
I Large numbers of eta matrices;
Sparse Bartels-Golub Method Disadvantages
I If the spike always occurs in the first column and extends to the
bottom row, the Sparse Bartels-Golub Method becomes worse than
the Bartels-Golub Method.
I The upper-triangular matrix will always be fully-decomposed
resulting in huge amounts of fill-in;
I Large numbers of eta matrices;
I O(n3 )-cost decomposition;
Ried Suggestion on Sparse Bartels-Golub Method
Task: The task is to find a way to reduce that bump before attempting to
decompose it;
Row singleton: any row of the bump that only has one non-zero
entry.
Column singleton: any column of the bump that only has one
non-zero entry.
Method:
I When a column singleton is found, in a bump, it is moved to the top
left corner of the bump.
Advantages:
I It significantly reduces the growth of the number of eta matrices in
the Sparse Bartels-Golub Method
I So, the basis should not need to be decomposed nearly as often.
I So, Reid’s Method remains numerically less stable than the Sparse
Bartels-Golub Method.
The Forrest-Tomlin Method
The Forrest-Tomlin Method
I At most one row-eta matrix factor will occur for each iteration
where an unpredictable number occurred before.
I The code can take advantage of such knowledge for predicting
necessary storage space and calculations.
I Fill-in should also be relatively slow, since fill-in can only occur
within the spiked column.
Disadvantages:
I Sparse Bartels-Golub Method allowed LU-decomposition to pivot for
numerical stability, but Forrest-Tomlin Method makes no such
allowances.
I Therefore, severe calculation errors due to near-singular matrices are
more likely to occur.
Suhl-Suhl Method
This method is a modification of Forrest-Tomlin Method.
For More Detail