U

S

T
i
w
a
r
y
,

I
I
I
T

A
l
l
a
h
a
b
a
d

1
Image Filtering in the
Spatial Domain
Overview: Linear filtering
Linear filters
Applications
Denoising
Non-linear filters: median filter
Sharpening
Subsampling
Images as functions
Source: S. Seitz
We can think of an image as a function, f, from
R
2
to R:
f( x, y ) gives the intensity at position ( x, y )
Realistically, we expect the image only to be
defined over a rectangle, with a finite range:
f: [a,b] x [c,d] [0,1]

A color image is just three functions pasted
together. We can write this as a vector-valued
function:
Images as functions
( , )
( , ) ( , )
( , )
r x y
f x y g x y
b x y
(
(
=
(
(

Source: S. Seitz





Motivation 1: Noise reduction
Given a camera and a still scene,
how can you reduce noise?
Take lots of images and average them!
Whats the next best thing?
Source: S. Seitz
This image is
too big to fit
on the screen.
How can we
reduce it by a
factor of two?

How about
taking every
second pixel?

Motivation 2: Image half-sizing
Source: S. Seitz
First attempt at a solution
Lets replace each pixel with an
average of all the values in its
neighborhood
Source: S. Marschner
First attempt at a solution
Lets replace each pixel with an
average of all the values in its
neighborhood
Moving average in 1D:
Source: S. Marschner
Weighted Moving Average
Can add weights to our moving
average
Weights [1, 1, 1, 1, 1] / 5
Source: S. Marschner
Weighted Moving Average
Non-uniform weights [1, 4, 6, 4, 1] /
16
Source: S. Marschner
Moving Average In 2D
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Source: S. Seitz
Moving Average In 2D
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Source: S. Seitz
Moving Average In 2D
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Source: S. Seitz
Moving Average In 2D
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Source: S. Seitz
Moving Average In 2D
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Source: S. Seitz
Moving Average In 2D
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30 30 30 20 10
0 20 40 60 60 60 40 20
0 30 60 90 90 90 60 30
0 30 50 80 80 90 60 30
0 30 50 80 80 90 60 30
0 20 30 50 50 60 40 20
10 20 30 30 30 30 20 10
10 10 10 0 0 0 0 0
Source: S. Seitz
Lets replace each pixel with a weighted average
of its neighborhood
The weights are called the filter kernel
What are the weights for a 3x3 moving average?
Generalization of moving
average
1 1 1
1 1 1
1 1 1
box filter
Source: D. Lowe
Defining convolution

= -
l k
l k g l n k m f n m g f
,
] , [ ] , [ ] , )[ (
f
Let f be the image and g be the kernel. The
output of convolving f with g is denoted f * g.
Source: F. Durand
Convention: kernel is flipped
MATLAB: conv2 vs. filter2 (also imfilter)
Key properties
Linearity: filter(f
1
+ f
2
) = filter(f
1
)
+ filter(f
2
)
Shift invariance: same behavior
regardless of pixel location:
filter(shift(f)) = shift(filter(f))
Theoretical result: any linear shift-
invariant operator can be
represented as a convolution

Properties in more detail
Commutative: a * b = b * a
Conceptually no difference between filter
and signal
Associative: a * (b * c) = (a * b) * c
Often apply several filters one after
another: (((a * b
1
) * b
2
) * b
3
)
This is equivalent to applying one filter: a *
(b
1
* b
2
* b
3
)
Distributes over addition: a * (b + c) =
(a * b) + (a * c)
Scalars factor out: ka * b = a * kb = k
(a * b)
Identity: unit impulse e = [, 0, 0, 1, 0,
0, ],
a * e = a

details
What is the size of the output?
MATLAB: filter2(g, f, shape)
shape = full: output size is sum of sizes
of f and g
shape = same: output size is same as f
shape = valid:output size is difference of sizes of f and
g
f
g g
g g
f
g g
g
g
f
g g
g g
full
same
valid
some details
What about near the edge?
the filter window falls off the edge of the
image
need to extrapolate
methods:
clip filter (black)
wrap around
copy edge
reflect across edge
Source: S. Marschner
Few more details
What about near the edge?
the filter window falls off the edge of the
image
need to extrapolate
methods (MATLAB):
clip filter (black): imfilter(f, g, 0)
wrap around: imfilter(f, g, circular)
copy edge: imfilter(f, g,
replicate)
reflect across edge: imfilter(f, g,
symmetric)
Source: S. Marschner
Practice with linear filters
0 0 0
0 1 0
0 0 0
Original
?
Source: D. Lowe
Practice with linear filters
0 0 0
0 1 0
0 0 0
Original
Filtered
(no change)
Source: D. Lowe
Practice with linear filters
0 0 0
1 0 0
0 0 0
Original
?
Source: D. Lowe
Practice with linear filters
0 0 0
1 0 0
0 0 0
Original
Shifted left
By 1 pixel
Source: D. Lowe
Practice with linear filters
Original
?
1 1 1
1 1 1
1 1 1
Source: D. Lowe
Practice with linear filters
Original
1 1 1
1 1 1
1 1 1
Blur (with a
box filter)
Source: D. Lowe
Practice with linear filters
Original
1 1 1
1 1 1
1 1 1
0 0 0
0 2 0
0 0 0
-
?
(Note that filter sums to 1)
Source: D. Lowe
Practice with linear filters
Original
1 1 1
1 1 1
1 1 1
0 0 0
0 2 0
0 0 0
-
Sharpening filter
- Accentuates differences with
local average
Source: D. Lowe
Sharpening
Source: D. Lowe
Smoothing with box filter
revisited
Smoothing with an average actually doesnt
compare at all well with a defocused lens
Most obvious difference is that a single point of
light viewed in a defocused lens looks like a
fuzzy blob; but the averaging process would
give a little square
Source: D. Forsyth
Smoothing with box filter
revisited
Smoothing with an average actually doesnt
compare at all well with a defocused lens
Most obvious difference is that a single point of
light viewed in a defocused lens looks like a
fuzzy blob; but the averaging process would
give a little square
Better idea: to eliminate edge effects, weight
contribution of neighborhood pixels according
to their closeness to the center, like so:
fuzzy blob
Gaussian Kernel
Constant factor at front makes volume sum to 1 (can
be ignored, as we should re-normalize weights to
sum to 1 in any case)
0.003 0.013 0.022 0.013 0.003
0.013 0.059 0.097 0.059 0.013
0.022 0.097 0.159 0.097 0.022
0.013 0.059 0.097 0.059 0.013
0.003 0.013 0.022 0.013 0.003
5 x 5, o = 1
Source: C. Rasmussen
Choosing kernel width
Gaussian filters have infinite support,
but discrete filters use finite kernels
Source: K. Grauman
Choosing kernel width
Rule of thumb: set filter half-width to
about
3


Example: Smoothing with a
Gaussian
Mean vs. Gaussian filtering
Gaussian filters
Remove high-frequency
components from the image (low-
pass filter)
Convolution with self is another
Gaussian
So can smooth with small-width kernel,
repeat, and get same result as larger-
width kernel would have
Convolving two times with Gaussian
kernel of width is same as convolving
once with kernel of width sqrt(2)
Separable kernel
Factors into product of two 1D
Gaussians
Useful: can convolve all rows, then all
columns
How does this change the computational
complexity?
Source: K. Grauman
Separability of the Gaussian
filter

Source: D. Lowe
Separability example
*
*
=
=
2D convolution
(center location only)
Source: K. Grauman
The filter factors
into a product of 1D
filters:
Perform convolution
along rows:
Followed by convolution
along the remaining column:
Review: Linear filtering
What are the defining mathematical
properties of a convolution?
What is the difference between
blurring with a box filter and blurring
with a Gaussian?
What happens when we convolve a
Gaussian with another Gaussian?
What is separability?
How does separability affect
computational complexity?
Noise
Salt and pepper noise:
contains random
occurrences of black and
white pixels
Impulse noise: contains
random occurrences of
white pixels
Gaussian noise: variations
in intensity drawn from
a Gaussian
normal
distribution
Source: S. Seitz
Gaussian noise
Mathematical model: sum of many
independent factors
Good for small standard deviations
Assumption: independent, zero-
mean noise
Source: K. Grauman
Smoothing with larger standard deviations suppresses noise, but also
blurs the image
Reducing Gaussian noise
Reducing salt-and-pepper noise
Whats wrong with the results?
3x3 5x5 7x7
Alternative idea: Median filtering
A median filter operates over a
window by selecting the median
intensity in the window







Is median filtering linear?
Source: K. Grauman
Median filter
What advantage does median
filtering have over Gaussian
filtering?
Robustness to outliers
Source: K. Grauman
Median filter
Salt-and-pepper noise Median filtered
Source: K. Grauman
MATLAB: medfilt2(image, [h w])
Median vs. Gaussian filtering
3x3 5x5 7x7
Gaussian
Median
Sharpening revisited
What does blurring take away?
original
smoothed (5x5)

detail
=
sharpened
=
Lets add it back:
original detail
+
Unsharp mask filter
Gaussian
unit impulse
Laplacian of Gaussian
) ) 1 (( ) 1 ( ) ( g e f g f f g f f f + - = - + = - + o o o o
image blurred
image
unit impulse
(identity)
Recall
Original
1 1 1
1 1 1
1 1 1
0 0 0
0 2 0
0 0 0
-
Sharpening filter
- Accentuates differences with
local average
This image is
too big to fit
on the screen.
How can we
reduce it by a
factor of two?

How about
taking every
second pixel?

Image half-sizing
Source: S. Seitz
Image sub-sampling
Throw away every other row
and column to create a 1/2
size image
- called image sub-
sampling
1/4
1/8
Slide by Steve
Seitz
Aliasing
Not a good idea to shrink an image
by taking every second pixel
If we do, characteristic errors appear
Wagon wheels rolling the wrong way in
movies
Checkerboards disintegrate in ray
tracing
Striped shirts look funny on color
television

Source: D. Forsyth
Resample the
checkerboard by taking
one sample at each
circle. Top left and top
right: reasonable
results. Bottom left is
all black and bottom
right has checks that
are too big.
Example
Source: D. Forsyth
Problem: Undersampling
What if we missed things between
the samples?
Simple example: undersampling a
sine wave
unsurprising result: information is lost
Source: S.
Marschner
Problem: Undersampling
What if we missed things between
the samples?
Simple example: undersampling a
sine wave
unsurprising result: information is lost
surprising result: indistinguishable from
lower frequency

Source: S.
Marschner
Problem: Undersampling
What if we missed things between
the samples?
Simple example: undersampling a
sine wave
unsurprising result: information is lost
surprising result: indistinguishable from
lower frequency
Aliasing: signals traveling in
disguise as other frequencies
Source: S.
Marschner
Aliasing in video
Slide by Steve Seitz
Aliasing in graphics
Source: A. Efros
Sampling Theorem
When sampling a signal at discrete
intervals, the sampling frequency
must be greater than twice the
highest frequency of the input signal
in order to be able to reconstruct the
original perfectly from the sampled
version (Shannon, Nyquist,
Whittaker, Kotelnikov)
Source: F. Durand
Antialiasing
What can we do about aliasing?
Sample more often
But when we want to downsample a
given digital image, we cant increase
its original sampling resolution
Make the signal less wiggly
Get rid of all frequencies that are
greater than half the new sampling
frequency
Will lose information
But its better than aliasing


Source: A. Efros
Gaussian (low-pass) pre-
filtering
G 1/4
G 1/8
Gaussian 1/2
Solution: filter the image, then subsample
Filter size should double for each size reduction
Source: Steve
Seitz
Subsampling with Gaussian
pre-filtering
1/4 (2x zoom) 1/8 (4x zoom) 1/2
Source: Steve
Seitz
Without prefiltering:
1/4 (2x zoom) 1/8 (4x zoom) 1/2
Source: Steve
Seitz
Spatial Filtering
use filter (can also be called as
mask/kernel/template or window)
the values in a filter subimage are
referred to as coefficients, rather than
pixel.
our focus will be on masks of odd
sizes, e.g. 3x3, 5x5,
Spatial Filtering Process
simply move the filter mask from
point to point in an image.
at each point (x,y), the response of
the filter at that point is calculated
using a predefined relationship.

=
=
+ + + =
mn
i i
i i
mn mn
z w
z w z w z w R ...
2 2 1 1
Smoothing Spatial Filters
used for blurring and for noise
reduction
blurring is used in preprocessing
steps, such as
removal of small details from an image
prior to object extraction
bridging of small gaps in lines or curves
noise reduction can be accomplished
by blurring with a linear filter and also
by a nonlinear filter
Smoothing Linear Filters
output is simply the average of the
pixels contained in the neighborhood of
the filter mask.
called averaging filters or lowpass filters.
Smoothing Linear Filters
replacing the value of every pixel in an
image by the average of the gray levels in
the neighborhood will reduce the sharp
transitions in gray levels.
sharp transitions
random noise in the image
edges of objects in the image
thus, smoothing can reduce noises
(desirable) and blur edges (undesirable)

3x3 Smoothing Linear Filters
box filter weighted average
the center is the most important and other
pixels are inversely weighted as a function of
their distance from the center of the mask
Weighted average filter
the basic strategy behind weighting
the center point the highest and then
reducing the value of the coefficients
as a function of increasing distance
from the origin is simply an attempt
to reduce blurring in the
smoothing process.
Example
a). original image 500x500
pixel
b). - f). results of smoothing
with square averaging filter
masks of size n = 3, 5, 9, 15
and 35, respectively.
Note:
big mask is used to eliminate
small objects from an image.
the size of the mask establishes
the relative size of the objects
that will be blended with the
background.
a b
c d
e f
Order-Statistics Filters (Nonlinear
Filters)
the response is based on ordering
(ranking) the pixels contained in the
image area encompassed by the filter
example
median filter : R = median{z
k
|k =
1,2,,n x n}
max filter : R = max{z
k
|k = 1,2,,n x
n}
min filter : R = min{z
k
|k = 1,2,,n x n}
note: n x n is the size of the mask
Median Filters
replaces the value of a pixel by the
median of the gray levels in the
neighborhood of that pixel (the original
value of the pixel is included in the
computation of the median)
quite popular because for certain types of
random noise (impulse noise salt and
pepper noise) , they provide excellent
noise-reduction capabilities, with
considering less blurring than linear
smoothing filters of similar size.
83
Example : Median Filters
84
Sharpening Spatial Filters
to highlight fine detail in an image
or to enhance detail that has been
blurred, either in error or as a
natural effect of a particular method
of image acquisition.


85
Blurring vs. Sharpening
as we know that blurring can be done
in spatial domain by pixel averaging in
a neighbors
since averaging is analogous to
integration
thus, we can guess that the sharpening
must be accomplished by spatial
differentiation.
First and Second-order derivative
of f(x,y)
when we consider an image function
of two variables, f(x,y), at which
time we will dealing with partial
derivatives along the two spatial
axes.

y
y x f
x
y x f
y x
y x f
c
c
+
c
c
=
c c
c
= V
) , ( ) , ( ) , (
f
2
2
2
2
2
) , ( ) , (
y
y x f
x
y x f
f
c
c
+
c
c
= V
(linear operator)
Laplacian operator
Gradient operator
Result Laplacian mask
Laplacian mask implemented an
extension of diagonal neighbors
Correct the effect of featureless
background
easily by adding the original and
Laplacian image.
be careful with the Laplacian filter
used

V +
V
=
) , ( ) , (
) , ( ) , (
) , (
2
2
y x f y x f
y x f y x f
y x g
if the center coefficient
of the Laplacian mask is
negative
if the center coefficient
of the Laplacian mask is
positive
Example
a). image of the North
pole of the moon
b). Laplacian-filtered
image with




c). Laplacian image
scaled for display
purposes
d). image enhanced by
addition with original
image
1 1 1
1 -8 1
1 1 1
Mask of Laplacian + addition
to simply the computation, we can
create a mask which do both
operations, Laplacian Filter and
Addition the original image.
Mask of Laplacian + addition
0 -1 0
-1 5 -1
0 -1 0








)] 1 , ( ) 1 , (
) , 1 ( ) , 1 ( [ ) , ( 5
)] , ( 4 ) 1 , ( ) 1 , (
) , 1 ( ) , 1 ( [ ) , ( ) , (
+ + +
+ + =
+ + + +
+ + =
y x f y x f
y x f y x f y x f
y x f y x f y x f
y x f y x f y x f y x g
Example
Note
0 -1 0
-1 5 -1
0 -1 0
0 0 0
0 1 0
0 0 0

V +
V
=
) , ( ) , (
) , ( ) , (
) , (
2
2
y x f y x f
y x f y x f
y x g
=
+
0 -1 0
-1 4 -1
0 -1 0
0 -1 0
-1 9 -1
0 -1 0
0 0 0
0 1 0
0 0 0
=
+
0 -1 0
-1 8 -1
0 -1 0
Unsharp masking
subtracting a blurred version of an
image produces sharpening output
image.
) , ( ) , ( ) , ( y x f y x f y x f
s
=
sharpened image = original image blurred image
High-boost filtering
generalized form of Unsharp masking
A > 1
) , ( ) , ( ) , ( y x f y x Af y x f
hb
=
hb
s
f (x, y) (A 1)f (x, y) (f (x, y) f (x, y))
(A 1)f (x, y) f (x, y)
=
=
High-boost filtering
if we use Laplacian filter to create
sharpen image f
s
(x,y) with addition of
original image
) , ( ) , ( ) 1 ( ) , ( y x f y x f A y x f
s hb
=

V +
V
=
) , ( ) , (
) , ( ) , (
) , (
2
2
y x f y x f
y x f y x f
y x f
s
High-boost filtering
yields

V +
V
=
) , ( ) , (
) , ( ) , (
) , (
2
2
y x f y x Af
y x f y x Af
y x f
hb
if the center coefficient
of the Laplacian mask is
negative
if the center coefficient
of the Laplacian mask is
positive
High-boost Masks
A > 1
if A = 1, it becomes standard
Laplacian sharpening
Example
Gradient Operator
first derivatives are implemented
using the magnitude of the
gradient.
(
(
(
(

c
c
c
c
=
(

= V
y
f
x
f
G
G
y
x
f
2
1
2
2
2
1
2 2
] [ ) f (
(
(

|
|
.
|

\
|
c
c
+
|
.
|

\
|
c
c
=
+ = V = V
y
f
x
f
G G mag f
y x
the magnitude becomes nonlinear
y x
G G f + ~ V
commonly approx.
Gradient Mask
simplest approximation, 2x2
z
1
z
2
z
3
z
4
z
5
z
6
z
7
z
8
z
9
) ( and ) (
5 6 5 8
z z G z z G
y x
= =
2
1
2
5 6
2
5 8
2
1
2 2
] ) ( ) [( ] [ z z z z G G f
y x
+ = + = V
5 6 5 8
z z z z f + ~ V
Gradient Mask
Sobel operators,
3x3
z
1
z
2
z
3
z
4
z
5
z
6
z
7
z
8
z
9
) 2 ( ) 2 (
) 2 ( ) 2 (
7 4 1 9 6 3
3 2 1 9 8 7
z z z z z z G
z z z z z z G
y
x
+ + + + =
+ + + + =
y x
G G f + ~ V
the weight value 2 is to
achieve smoothing by
giving more important
to the center point
Note
the summation of coefficients in all
masks equals 0, indicating that they
would give a response of 0 in an area
of constant gray level.
Example
Example of Combining Spatial
Enhancement Methods
want to sharpen
the original image
and bring out more
skeletal detail.
problems: narrow
dynamic range of
gray level and high
noise content
makes the image
difficult to enhance
Example of Combining Spatial
Enhancement Methods
solve :
1. Laplacian to highlight fine
detail
2. gradient to enhance
prominent edges
3. gray-level transformation to
increase the dynamic range of
gray levels