Motion Estimation for compression

April, 29, 2009

‫שידור ווידיאו ואודיו על רשת האינטרנט‬

Dr. Ofer Hadar
Communication Systems Engineering Department
Ben-Gurion University of the Negev
URL: http://www.cse.bgu.ac.il/~hadar
Based on: 1 Lecture of Osnat Bar Nir from Scopus
2. Motion estimation lecture:
www-ict.its.tudelft.nl/html/education/courses/et10-38/hc/hc14
3. Image communication course, lec.10, by Girod
4. Block matching algorithms, from: http://atlantis.ucc.ie/dvideo/contents.html

Copyright @1999, O. Hadar

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬ 1

 ‫‪2‬‬

‫‪Block Matching Approaches‬‬

‫שידור ווידיאו ואודיו ברשת האינטרנט ‪2009 -‬‬

 3

What Is Motion Estimation ?

Example:

Frame 1 Frame 2

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 4

Block Matching
GOOD MATCH
Macro-Block to be coded

FAIR MATCH POOR MATCH

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 5

Residual Error Picture

Desired Picture Reference Picture Predicted Picture

The residual Error Picture

contains the differences
between the desired
picture and the
predicted one.

This is the picture to be Residual Error Picture

coded and transmitted.

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 6

Why Motion Estimation ?

The basic principle of motion estimation is that in

most cases, consecutive video frames will be
similar except for changes induced by objects
moving within the frames.

For each block in current frame find a matching block from

consecutive frame, so that only the (relatively small)
Residual Error Picture and its corresponding Motion Vectors
are encoded.

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 7

How Motion is estimated ?

The problem for motion estimation to solve is
how to effectively represent the changes
between two video frames.

Motion estimation applies a comprehensive two-

dimensional spatial search in order to find the
best match for each block.

MPEG does not define how this search should

be performed.

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 8

Block Matching - 1

 Block Matching
 Assumes the 2D translation motion model.
 Subdivide the frame (image information) into square (NXN) blocks.
 Image sequence model.

x(m, n, t )  x(m  d h , n  d v , t  1) for (m, n)  S

x(m, t )  x(m  d , t )
(d h ,d v)(m, n)

S
t-1 Observe
notation!
time
t

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 9

Criterions for Block Matching - 1

 For each NXN block S, a single motion vector (dh,dv) is found by

minimizing an error measure- finding the best “match” which
minimizes the error measure.
 Mean Square Error (MSE) match criterion-
M (d )   (
mS
x ( m, t )  x ( m  d , t  1)) 2

 Mean Absolute Difference (MAD):

M (d )   x(m, t )  x(m  d , t  1)
mS

 Full search block matcher calculates M(d) for all d within an

MXM search area.

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 10

Criterions for Block Matching - 2

 Pixel Difference Classification (PDC):

 Compares each pixel of the target block with its counterpart
in the candidate block and classifies each pixel pair as either
matching or not. “Match” if the difference is less than some
threshold (th). The grater the number of matching pixels, the
better the match is.

M (d )   ord ( x(m, t )  x(m  d , t  1)

mS
 t)

 1, if x is true 
ord(x)   
0, if x is false 

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 11

Criterions for Block Matching - 3

 Integral projection
 Sums the values of pixels from each column and each row of
the block.
K L L L K K
M (d )    x( p, q, t )   x( p, q, t  1)    x( p, q, t )   x( p, q, t  1)
p 1 q 1 q 1 q 1 p 1 p 1

K - no. of rows, L - no. of columns

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 12

Full Search
All blocks in the search window (part of reference frame) are
compared against current block (in currently coded frame), in
order to select the most representative one.
The pointer to the selected block is called Motion Vector.

currently coded frame reference frame

Usually requires long processing time, therefore not

applicable in real-time applications

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 13

Full search block matching - 1

Full search/brute force/exhaustive block matching
dv
M
dv,max
M
N
dh,max N dh

x(m,t)

 All possible displacements within the search range are

compared =>
 Computationally complex.
 Noise sensitive.
 Estimated vector field is not very smooth.

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 14
Sub-Optimal Motion Estimation

Motion estimation is very expensive computationally

DSP computation power is limited…

Full search is not

applicable

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 15
Sub Optimal Motion Estimation

For a search window of ±32 pixels, full search will

take more than 4000 points per MB.

DSP enables a budget of about 250 points per MB.

A way must be found to estimate motion in

much more efficient way:

A Sub-optimal Motion Estimation is needed

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 16

Sub-Optimal Motion Estimation

Main approaches to reduce computation of motion

estimation:
Sub-sampling of the motion displacement space

Pel sub-sampling: A sub-sample of the MB is

compared with sub-sampled blocks in search
window.
Adaptive search area

Predictive Motion Estimation

Hierarchical & multi-resolution motion estimation

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 17
Sub-sampling of the motion displacement
space

Compare a sub-set of blocks against

current MB

Algorithms based on a coarse-to-fine

approach: Step size of search is reduced
by ½ with every search step.

Many algorithms, using various searching

orders.
2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬
 Sub-sampling of the motion displacement 18

space
example: 3-Step Search:

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 19

Reduced complexity block matching - 1

 To reduce the complexity of full search block matching, various
restricted search patterns have been proposed.
 “One at a time search”:
 Horizontal stage: the point of minimum distortion on the
horizontal axis is found.
 Vertical stage: Starting with this point, the minimum
distortion in the vertical direction is found.
dv

dh

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 20

Reduced complexity block matching - 2

 N-step search or cross-search
 A number of positions around a center are tested. The position of
minimum distortion becomes the center of the next stage.
dv

dh

 Principle of locality: First, examine a number of sparsely spaced

candidate blocks, from the search area, and choose the best match
from these to be the center of a finer (denser) search.
 Special case: Quadrant Monotonic: candidate blocks close to the
optimal block have better matches and the value of distortion is a
function of the distance from the optimal position.
2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬
 21

Reduced complexity block matching - 3

 2D logarithmic search
 Iterative comparison of error measure values at 5
neighboring points
 Logarithmic refinement of the search pattern if:
 The best match is in the center of the 5-point pattern.

 The center of the search pattern touches the border of

the search range.
 Computational complexity:

Example: max. horizontal, vertical displacement = 6 integer-pix

accuracy. (a- for special vector, b- worst case).
2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬
 22

2-D Logarithmic Search

We will discuss two approaches here:
 This approach was published in a paper by Jain and Jain.
 Iterative comparison of the error measure (MAE) at five
neighboring points
 Logarithmic refinement (divide by 2) of the search pattern if
 Best match is in the center of the 5-point pattern (right figure)
 OR, center of search pattern touches the border of the search
range (left figure).
Search edge
4 4
3 3
5 5 5 5
5
3 3 4 3 3 5 4
2 2
5
2 2 2 2
1 1

1 1 1 1 1 1

1 1

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 23

Reduced complexity block matching - 5

 Orthogonal search algorithm (OSA):
 Hybrid of the 2D logarithmic search and the 3 step search:
 First stage- horizontal:
 Compute distortion in 3 points (0,0), (s,0), (-s,0), where s
is the step size.
 Position of minimum distortion becomes the center of the
vertical stage.
 Second stage- vertical:
 Two point- above and below the new center are
examined, and the value of the distortion function at
these 3 positions are compared.
 The position with the minimum distortion becomes the
center of the next stage.
 After one horizontal and vertical iteration, the step size s is
halved, if it is greater than 1, and the algorithm proceeds
from step one.

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 24

Reduced complexity block matching - 6

 Orthogonal search algorithm (OSA), cont.

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 25

Pel Subsampling

16x16 Pixel Array 2:1 Subsampling 4:1 Subsampling

Reduces the number of pels taken into account for the distance
calculation at every search position.

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 Adaptive search area
26

Example
Spiral Search:

Performs a Spiral Search for the matching block.

Search window size is data dependent, and set in run-time

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 27

Predictive Motion Estimation

Often there is a high correlation between

motion of neighboring blocks.

Motion vectors can be predicted a priori, in

order to gain an initial guess of the next motion
vector.

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 29

Hierarchal Block Matching - 1

 Start with estimation on low resolution frames.

 Propagate (intermediate) vectors to higher resolution levels.
 Allow only refinement of propagated motion vectors
 Limited complexity even for large search areas.
 Implicit smoothing of motion field.

Resolution level 1

Resolution level 2

Resolution level 3

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 30

Hierarchal Block Matching - 2

2.a. propagate
motion field
1. Block matcher (e.g. N-step) on small blocks

2.b. refine motion field

3.b. refine motion field

Frame t-1 Frame t

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 31

Hierarchal Block Matching - 3

Search area Found motion
vector

Resolution level 1

propagate Search area

(only
refinement
Resolution level 2 allowed)

Smoothness is obtained because the same propagated vector is used for

multiple blocks at higher levels.
2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬
 32

Block matching for other models

 For other than traditional models, block matching is still

possible, but becomes very computationally expensive.
 Example: Rotation

Ф=0 Ф=0.15 Ф=0.30 Ф=0.70

For a very large number of rotations, the match criterion has to be

calculated.

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬

 33

Summary
After reviewing the above sub-optimal motion
estimation method, we found out that the best
results were achieved while using an algorithm
which combines sub-sampling of the motion
displacement space with adaptive search area
and predictive motion vectors.

Results were close enough to the results

obtained by full search.

2009 - ‫שידור ווידיאו ואודיו ברשת האינטרנט‬