Академический Документы
Профессиональный Документы
Культура Документы
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
What This Field is All About ?
Linear
n
D n
Invertible Transforms
x
Linear
Unitary
Separable
Structured
n D n
x
We shall keep the linearity of the inverse-transform
As for the forward (computing from x), there are
infinitely many possible solutions
We shall seek the sparsest of all solutions – the one
with the fewest non-zeros
n D n
x
This makes the forward transform a highly non-linear
operation
The field of sparse and redundant representations
is all about defining clearly this transform, solving various
theoretical and numerical issues related to it, and
showing how to use it in practice
Sounds great !?
No, in fact, it sounds boring !!!!
Who cares about a new
transform?
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
What This Field is All About ?
CT
Radar Imaging
Heart Signal Traffic Information
Simplicity Reliability
8KB
How & why does it works?
Discrete
4KB
Cosine
Trans.
Problem
Model Signal
(Application)
Numerical
Scheme
Machine
Signal Learning
Processing Mathematics
Wavelet Approximation
Theory Theory
Multi-Scale Linear
Analysis Algebra
Signal Sparseland Optimization
Transforms Theory
Compression Inpainting
Blind Source Super-
Separation Resolution
Denoising Demosaicing
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
A Closer Look at the
Sparseland Model
0
Iteration 6
0
1
2
3
4
5
-1
-2
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
Who Works on This
and Who Are We ?
An extremely talented
mathematician and statistician
from the Stanford Statistics
Department
He is among the few who founded
this field sparse and redundant
representations and its
spin-off topic of
compressed sensing
In 2013 he won the Shaw
prize (“the Nobel of the
east”)
Computer-Science Department
Prof. Michael Elad The Technion
M. Elad published more than 70 journal
papers and a book in the field of sparse
& redundant representations, spanning
theory, algorithms, and applications
In the past 10 years, Michael Elad has
been teaching a course at the Technion,
similar to the one offered here
Michael Elad | The Computer-Science Department | The Technion
The Course’s Team: Teaching Assistant
# citations
1606
11190
From the 362
506
Technion 423
642
Israel 2988
.
.
Zhang Lei – Xidian Radar .
Zhang Lei – HK Polytech .
.
Zhang Li – Xidian EE 1128
Zhang Li – Tsinghua Univ. .
Zhang Long – Xidian .
.
Zhang Lan – Xi An Jiao Tong .
Zhang Liang – Chongqing .
18735
Zhang Liao – Xidian .
Zhang Lin – Beijing Univ. .
Zhang Lu – Capital Med. 16474
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
Several Examples:
Applications Leveraging
this Model
Source Outcome
Source
Initial dictionary
The obtained
Result 30.829dB (overcomplete DCT)
dictionary after
10 64×256
iterations
Noisy image 20
Original
15.81 13.89 6.60 JPEG
JPEEG-2000
14.67 12.41 5.49 Sparse-Based
Original
? 18.62 7.61 JPEG
JPEEG-2000
? 16.12 6.31 Sparse-Based
? 16.81
Results for 400 Bytes per each file
7.20
Ideal Image
SR Result
PSNR=16.95dB
Bicubic
interpolation
PSNR=14.68dB Given Image
An effective (yet
simple) model for Sparse and redundant
signals/images is representations and other
key in getting better Which model example-based modeling
algorithms for to choose? methods are drawing a
various applications considerable attention in
recent years
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
This Course:
Numerical Applications
Theory
Problems (image processing)
Part 1: Part 2:
Introduction to the Sparse Representations
Fundamentals of Sparse - From Theory to
Representations Practice
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
Underdetermined Linear
System of Equations
& Regularization
Underdetermined Linear System
n A n
b b
x x
Solving this linear system amounts to an
attempt to obtain x from b, a problem known
as Interpolation or Single-Image-Super-Resolution
Clearly, in such problems we desire a single
solution among the infinitely many that exist
Who is J(x) ?
Clearly, everything depends on the choice of this
function, so who is it? How should we choose it?
The answer to this question depends on your
needs, beliefs, mathematical proficiency, your field,
and more
The idea of regularization is central in sciences, and
it has been used in many fields and in many ways
Why is it called REGULARIZATION? Because it takes
a problem with a severe singularity (a.k.a. ill-posed
problem) and turns it to become “regular”
L2 -based
1
Jx
2
Bx for some chosen matrix B
2 2
1 2
min Bx s.t. A x b
x 2 2
1
1 1
T T T T
A x A B B A b A B B A b
A B B
1
1 1
T T T T
x B B A A b
1 2
min Bx s.t. A x b
x 2 2
A B B
1
1 1
T T T T
x opt B B A A b
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
The Temptation
of Convexity
Is There Life Beyond L2 ?
x1 x2 x1
x2
tf x 1 (1 t)f x 2
f x
Epigraph
x1 x x2
f t x 1 (1 t)x 2 tf x 1 (1 t)f x 2
Convex function
x1 x x2
Non-Convex function
x2 x2
x1 x1
f x 2 f x1 f x1
T
x 2 x1
f x
2
0
Example:
Is the following function (strictly) convex?
1
Jx
2
Bx
2 2
Answer:
The first derivative of this expression is given by
J x B B x
T
Lp-norm (p1): x x
2 p
2 k
x p
p
xk
k k
L: x
max x k
k
A norm is a function x :
m
with the properties:
o Non-negativity: x x 0 and x 0 x 0
o Homogeneity: x & c, c x c x
o Triangle-Inequality: x , y , x y x y
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
A Closer Look at
the L1 Minimization
Our Goal
P1 min x
x 1
s.t. A x b
Why are we so interested in this choice of J(x)?
SPARSE SOLUTIONS
m
Example: m in
n A n
A [1 1]
s.t. 2
b [2]
b
x
2
Sparse Optimal
Solutions solutions
h 0 | A h 0 Assume w.l.o.g.
that these are
supp h supp x * the first k
columns in A
sign(x ) x
* *
x * h 1
j j
h j
j 1
k k
sign(x )x sign(x j )h j x *
* * * T
j j h sign(x*)
1
j 1 j 1
T
h sign(x*) 0
x* 1
x * h 1
Conclusions:
x*+εh is in S as well (just as optimal as x*)
We can choose ε to null one entry in x*+εh,
getting a new optimal solution with k-1 non-zeros
We can proceed this way and null elements in the
optimal solution till we get n non-zeros, as claimed
b
x
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
Conversion of (P1) to
Linear Programming
Solving (P1)
x u v where x, u, v
m
and u 0 & v 0
Example:
o All the positive entries go to u
1 1 0
3
3
0 o All the negative ones are in v
0
0 0
o Both vectors are non-negative
4 0 4
x uv o A by-product of this split, since
1 0 1
0 0 0
their supports do not overlap,
0 0 0 we get
T
2 2 0 u v 0
b A x A u v
m
x 1
xk
k 1
m m
A A u
u k
v k v
k 1 k 1
1 1 u
T T u
and 0
v
v
1 T 1 T u s.t. b A A u u
min
(LP) & 0
u ,v v
v
v
Proof: 0 0 0
0 0 0
o Assume that the optimal solution violates uTv=0 2 2 0
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
Promoting Sparse
Solutions
What Have we Done so Far?
Ax b Regularize:
What
has infinitely Jx ?
to do? min J x s.t. A x b
many solutions x
Ax b Ax b
Ax b Ax b
x const x 2
const
1
o Symmetric (x)
o Monotonically x
non-decreasing for x>0
o Monotonic non-increasing derivative for x>0
Examples: x log 1 x
x
x 1 e
x
x
1 x
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
The L0-Norm and the
(P0) Problem
From Lp to L0
m
Definition: Given a vector x , its L0-norm is
defined as
m
I xk
p
x 0
lim x p
p0
k 1
I x
p
x 0
lim x p
k
p0
k 1
5 xy x y 5 5 xy x 0
y 6 4 xy x y 6
0 0 0 0 0 0 0 0
I x
p
x 0
lim x p
lim k
p0 p0
k 1
o inequality: Surprisingly
xy x 0
y
0 0
m
x0 Our field is not about covering all b n
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
A Signal Processing
Perspective
Mathematical Riddle ?
(P0 ) min x 0
s.t. A x b
x
So far, our discussion has been purely mathematical: We
are interested in better understanding the problem (P0)
that seeks the sparsest solution to a linear system
Piece-wise constant
signals can be described n A n
as a sparse combination
of atoms taken from the b
Heaviside dictionary m x
=
x5
+ x9
1
0 + x16