Вы находитесь на странице: 1из 36

# Computer Vision (EEE6503) Fall 2009, Yonsei Univ.

-
-
Lucas
Lucas
-
-
Tomasi
Tomasi
(KLT)
(KLT)
Feature Tracker
Feature Tracker
Computer Vision Lab.
Computer Vision Lab.
Jae Kyu Suhr
Jae Kyu Suhr
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Introduction
Introduction
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Motivation Motivation
Image sequence
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Motivation Motivation
Feature point detection
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Motivation Motivation
Feature point tracking
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Motivation Motivation
Tracking result
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
3D Reconstruction results 3D Reconstruction results
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Problem Statement
Problem Statement
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Problem Statement Problem Statement
I, J : 1
st
and 2
nd
grayscaledimages
I(x,y) : gray value of I at [x y]
T
Let u=[u
x
u
y
]
T
be a point on the 1
st
image I
The goal is to find v on J , where I(u) and J (v) are
similar. (v= u+d = [u
x
+d
x
u
y
+d
y
]
T
)
d=[d
x
d
y
]
T
is the image velocity at u
or the optical flow at u.
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Problem Statement Problem Statement
Definition of the residual function (d).

x
and
y
integration window size parameter
integration window size (2
x
+1)(2
y
+1).
(u
x
, u
y
)
w
x
w
y
I J
( - )
2
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Pyramid Implementation
Pyramid Implementation
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Why pyramid representation? Why pyramid representation?
Standard KLT algorithm can deal with small pixel displacement.
Solution for this is a pyramidal implementation.
Let I
0
=I be the 0
th
level image
The pyramid representation is built recursively.
I
0
I
1
I
2
I
3
I
Lm
(Lm: 2~4)
I
L-1
: the image at level L-1
I
L
: the image at level L
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Pyramidal feature tracking Pyramidal feature tracking
A given point u in I, find its corresponding location v=u+d.
Corresponding point of u(u
0
) on the pyramidal image I
L
is u
L
Simple overall pyramid tracking algorithm
d
Lm
is computed at the pyramid level L
m
d
Lm-1
is computed
with an initial guess of d
Lm
at L
m-1
This continues up to the level 0
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Let g
L
=[g
L
x
g
L
y
]
T
be an initial guess at level L
( g
L
is available from level L
m
to level L+1 )
Residual pixel displacement vector d
L
=[d
L
x
d
L
y
]
T
that minimizes the new image matching error function
L
Window size (2
x
+1)(2
y
+1) is constant for all pyramid
g
L
is used to pre-translate the image patch in 2
nd
image J
d
L
is small and therefore easy to compute using KLT algorithm
Pyramidal feature tracking Pyramidal feature tracking
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Pyramidal feature tracking Pyramidal feature tracking
Assume that d
L
is computed, new initial guess g
L-1
at level L-1
Initial guess for the deepest level L
m
The final optical flow solution d
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Standard KLT algorithm
Standard KLT algorithm
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Standard KLT optical flow computation Standard KLT optical flow computation
Goal is to find d
L
that minimizes the matching function
L
.
Same operation is performed for all levels L, drop the superscript L
and define the new images A and B
Let us change the displacement vector and the point vector
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Standard KLT optical flow computation Standard KLT optical flow computation
Goal is to find that minimizes the matching function
To find the optimum
Let us substitute by its 1
st
order Taylor expansion
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Standard KLT optical flow computation Standard KLT optical flow computation
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Standard KLT optical flow computation Standard KLT optical flow computation
( G must be invertible).
Standard LK is valid only the pixel displacement is small.
(Because of the first order Taylor approximation)
Therefore, iterative version of LK is necessary.
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Iterative KLT algorithm
Iterative KLT algorithm
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Iterative KLT optical flow computation Iterative KLT optical flow computation
k: iteration index

## : initial guess from the previous iteration 1,2,,k-1

B
k
: new translated image according to
The goal is to compute that minimize
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Iterative KLT optical flow computation Iterative KLT optical flow computation
One step LK optical flow computation is
I
x
, I
y
are computed only once at the beginning of the iteration.
G (22matrix) remains constant throughout the iteration loop.
Only I
k
needs to be recomputed at each iteration
Once is computed, a new pixel displacement guess
The iteration goes on until is smaller than a threshold
or reached at the maximum number of iteration.
(5 are enough to reach convergence)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Iterative KLT optical flow computation Iterative KLT optical flow computation
At 1
st
iteration, the initial guess
Assuming that K iterations are necessary to reach convergence,
the final solution for the optical flow vector
This overall procedure is repeated at all levels L-1, L-2, , 0
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Feature selection Feature selection
Goal is to find the location of v on J corresponding to u on I.
How to detect u on I
Find the point u whose G matrix is non-singular
The minimum eigenvalueof G must larger than a threshold.
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Feature selection Feature selection
1. Compute the G matrix and its minimum eigenvalue
m
at every pixel
in the image I.
2. Call
max
the maximum value of
m
over the whole image.
3. Retain the image pixels that have a
m
value larger than a threshold.
4. Retain the local maximum pixels (a pixel is kept if its
m
value is
larger than that of any other pixel in its 33 neighborhood).
5. Keep the subset of those pixels so that the minimum distance
between any pair of pixels is larger than a given threshold distance.
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Feature selection Feature selection
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Feature selection Feature selection
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Feature selection Feature selection
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Feature selection Feature selection
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Feature selection Feature selection
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Declaring a feature Declaring a feature Lost Lost
A feature point falls outside of the image.
An image patch around the tracked point varies too much
between image I and image J .
(A cost function (SSD) is larger than a threshold).
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Flow chart Flow chart
Input 1
st
image
Select feature points
Input 2
nd
image
Track feature points
Delete lost feature points
Save feature points
Save tracked points
Replace 1
st
image with 2
nd
image
Replace 2
st
image with next image
Select feature points
to replace lost feature points
Save feature points
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
References References
Bruce D. Lucas and Takeo Kanade. An Iterative Image Registration
Technique with an Application to Stereo Vision. IJ CAI, pages 674-679,
1981.
J . Shi and C. Tomasi, "Good Features to Track," CVPR'94
J ean-Yves Bouguet, Pyramidal Implementation of the Lucas Kanade
Feature Tracker Description of the algorithm, Intel Corporation
Microprocessor Research Labs.
http://www.ces.clemson.edu/~stb/klt/ C++code
KLT: An Implementation of the Kanade-Lucas-Tomasi Feature Tracker
OpenCV: CalcOpticalFlowLK, CalcOpticalFlowPyrLK