International Conference on Control, Automation and Systems 2007

Oct. 17-20, 2007 in COEX, Seoul, Korea

A Real-Time Color Feature Tracking System Using Color Histograms*

Jung Uk Cho 1, Seung Hun Jin 1, Xuan Dai Pham 1, D o n g k y u n Kim 1, and Jae W o o k Jeon 2
1School of Information and Communication Engineering, Sungkyunkwan University, Suwon, Korea
(Tel: +82-31-290-7237; E-mail: {ichead, coredev, pdxuan, bluster}@ece.skku.ac.kr)
2 School of Information and Communication Engineering, Sungkyunkwan University, Suwon, Korea
(Tel : +82-31-290-7129; E-mail: jwjeon@yurim.skku.ac.kr)
Abstract: Color feature tracking is based on pattern matching algorithms where the appearance of the target is
compared with a reference model in successive images and the position of the target is estimated. The major drawback
of these methods is that such operations require expensive computation power. It is bottleneck to implement real-time
color feature tracking system. The probabilistic tracking methods have been shown to be robust and versatile for a
modest computational cost. However, the probabilistic tracking methods break down easily when the object moves very
fast because these methods search only the regions of interest based on the probability density function to estimate the
position of the moving object. In this paper, we propose a real-time color feature tracking circuit. We propose a
window-based image processing structure to improve the processing speed of the tracking circuit. The tracking circuit
searches all regions of the image to perform a matching operation in order to estimate the position of the moving object.
The main results of our work are that we have designed and implemented a physically feasible hardware circuit to
improve the processing speed of the operations required for real-time color feature tracking. Therefore, this work has
resulted in the development of a real-time color feature tracking system employing an FPGA (Field Programmable Gate
Array) implemented circuit designed by VHDL (the VHSIC Hardware Description Language). Its performance has
been measured to compare with the equivalent software implementation.

Keywords: Color histogram,

color tracking, FPGA, VHDL.

tracking circuit searches all regions of the image to
perform a matching operation in order to estimate the
position of the moving object during one frame time
without any delay or loss of image data. The main
purpose of our work, described in this paper, has been to
design and implement physically feasible hardware
circuits to improve the processing speed of the
operations required for real-time color feature tracking.
Therefore, this work has resulted in the development of
a real-time color feature tracking system employing an
FPGA implemented circuit designed by VHDL. Its
performance has been measured and compared with the
equivalent software implementation.

1. I N T R O D U C T I O N
Tracking in image sequences has been used in the
fields of intelligent robots [1], surveillance [2], human
computer interface [3], smart room [4], vehicle tracking
[5], and video compression [6]. Color feature tracking
provides efficient and robust tracking performance in a
cluttered environment. Color feature tracking is based
on pattern matching algorithms where the appearance of
the samples is compared with a reference model in
successive images and the position of the target is
estimated. The major drawback of these methods is that
the appearance of many samples has to be compared
with the reference model for each and every new image.
Such operations require expensive computation power.
It is bottleneck to implement real-time color feature
tracking system. The main constraint of the tracking
system is computation time [7]. The probabilistic
tracking methods such as Kalman filtering, particle
filtering, and condensation algorithms have been shown
to be robust and versatile for a modest computational
cost [8]. However, the probabilistic tracking methods
break down easily when the object moves very fast
because these methods search only the regions of
interest based on the probability density function to
estimate the position of the moving object.
In this paper, we propose a real-time color feature
tracking system using color histograms. We propose a
window-based image processing structure to improve
the processing speed of the tracking circuit. The

2. T R A C K I N G U S I N G C O L O R
HISTOGRAMS

* This research was performed for the Intelligent Robotics

Development Program, one of the 21st Century Frontier R&D
Programs funded by the Ministry of Commerce, Industry and Energy
of Korea.
978-89-950038-6-2-98560/07/$15 @ICROS

1163

Color can provide a useful feature for tracking a

moving object in a cluttered environment. A color
histogram is a popular form of representation of the
object appearance in object tracking [9]. A histogram is
independent from scaling and rotation of the object and
robust to the partial occlusion and non rigidity of the
object [10][l 1]. The color histogram is a useful
technique for tracking. This method makes a model
color histogram according to the appearance of an
object in the image. The color histogram can be
calculated in RGB or any other color space, or simply in
a grey level space, depending on the input image
sequence. It matches the color histogram of the model
and the color histograms of the samples in the image
sequence. The object location can be estimated through
searching for the maximum value of the matching
operation. This algorithm has the advantage of fast

recognition and localization of the object in the image

sequence.
We propose a color feature tacking circuit using color
histograms. We propose a standard RGB (Red, Green,
and Blue) color model which is obtained by a histogram
technique in the RGB color space. This produces the
following specializations of the abstract measurement
step.
2.1 Create an appearance model (color histograms)
First, RGB images are captured from the camera. We
generate the appearance model (color histograms) of the
object within the patch size (Hx, Hy) around the position
of the object: Three 1D RGB histograms,
q R = {qrR : r = 1," , R} , q G = {q gG : g = 1," , G} , and

q B = {qbB : v = 1," , B} are designed as representation

space for the appearance of the object. Here, qrR , qgG ,

and qbB are the number of pixels in the patch (Hx, Hy)
around the position of the object where the red color
values are in the bin r, the green color values are in the
bin s, and the blue color values are in the bin b,
respectively. R, G, and B are the number of bins of red
color, green color, and blue color, respectively.
2.2 Find the best matching (position of the target)

( x, y )

We find the position

in the frame whose

histogram is the best match for the model histogram for

each frame. For each position (x, y) in the image, we
generate sample histograms pR, pG, and pB form the
patch (Hx, Hy) around the position (x, y). Here, prR , pgG ,
and pbB are the number of pixels in the patch (Hx, Hy)
around the position of the samples where the red color
values are in the bin r, the green color values are in the
bin s, and the blue color values are in the bin b,
respectively. R, G, and B are the number of bins of red
color, green color, and blue color, respectively. We
compare the color histogram of the model with the color
histogram of the samples. We use the popular
Bhattacharyya similarity measure to measure the
matching value.

( x, y ) = arg max ( x, y )

(3)

( x, y )

3. THE CIRCUIT USED FOR COLOR

FEATURE TRACKING USING COLOR
HISTOGRAMS
We propose a real-time color feature tracking circuit
using color histograms. Figure 1 shows the block
diagram of the circuit used for color feature tracking
using color histograms. It consists of five modules: the
Digital Camera Controller, Position Counter, Color
Histogram Based Tracker, Trajectory Presenter, and
Digital Video Output. It is designed using VHDL and
implemented in an FPGA, in order to perform color
feature tracking using color histograms in real-time.
3.1 Digital Camera Controller
In the Digital Camera Controller module, the Camera
Link specification is applied to this module to grab the
image from the digital color camera. The image sync
signals, Sync, including the vertical sync signal,
horizontal sync signal, and pixel clock, and digital color
image data, RGB, including red, green, and blue colors,
are generated by this module. They are used in other
modules of the color feature tracking circuit.
3.2 Position Counter
Every pixel has its position (pixel, line, and frame) in
the image. The position of a pixel, Position, is
calculated by the Position Counter module. The frame
of the image is identified using a vertical sync signal
and the line of the image is identified using a horizontal
sync signal. The pixel of the image is divided by the
pixel clock. This position is used as the coordinate on
the image to track an object in other modules of the
color feature tracking circuit.
3.3 Color Histogram Based Tracker
The Color Histogram Based Tracker module is
designed according to abovementioned color histogram
based tracking algorithm. This module consists of
Window Based Processing, Histogram Generation, and

R ( x, y ) = prR ( x, y ) qrR
r =1

G ( x, y ) = pgG ( x, y ) qgG

(1)

g =1

B ( x, y ) = pbB ( x, y ) qbB
b =1

Thus, the more similar a sample appears to the model,

the larger it becomes. These three similarities are
blended to get a unified similarity.
( x, y ) = R ( x, y ) + G ( x, y ) + (1 ) B ( x, y ) (2)
The ( x , y ) is the position in the frame whose
histograms give the maximal similarity measure to the
model.

1164

Fig. 1. Block diagram of the circuit used for color feature

tracking using color histograms.

Fig. 2. Block diagram of the color histogram based tracker

module.

Histogram Matching blocks as indicated in Fig. 2. The

color histogram based tracking method matches the
color histogram of the image and the color histogram of
the model. The object location can be identified through
searching for the peak value of the matching operation
in the image. It needs expensive computation power to
perform the matching operation for the image. A
general purpose computer of Von Neumann architecture
has to access memory at least n m i j times to get
data for each pixel when it processes an n m size
window in the i j size image. It may take a longer time
to access the memory of the whole image than to
calculate the necessary calculation. As a result, it is
almost impossible to process an image in real-time. In
order to reduce the memory access time and the
processing time, we propose a window-based image
processing structure. The Window Based Processing
block stores the necessary pixels for processing each
pixel and its neighboring pixels together. It consists of
the line buffer, window buffer, line buffer controller,
and window buffer controller. The line buffer stores
some parts of the image and its controller generates the
control signals for moving and storing of the pixel value.
The line buffer uses a dual port RAM where the number
is the same as that of the row in a window. Each dual
port pixels can store one line of an image. Thus, the x
coordinate of the pixels can be used as the address for
the dual port RAM. For the incoming pixel where the
coordinate is (x, y), the line buffer controller performs
operations such as in (4), where n is the window row
size, p(x, y) is the incoming pixel value, and L(x, y)
represents each pixel in line buffer.
(4)
L( x, y k ) = L ( x, y (k 1)) where 1 k n 1
L( x, y k ) = p( x, y) where k = 0

the line buffer and its controller generates control

signals for moving and storing of pixel values. Since
pixels of a window are stored in registers, it is possible
to perform a window-based image processing. For the
incoming pixel of which the coordinate is (x, y), the
window buffer controller performs operations as in (5)
where n and m are the row and column size,
respectively in a window. p(i, j) is the incoming pixel
value in a window buffer; p(x, y) is the incoming pixel
value in the line buffer; W(i, j) represents each of the
pixels in the window buffer; and L(x, y) represents each
of the pixels in the line buffer.
W (i k , j ) = W (i (k 1), j ) where 1 k m 1
W (i, j k ) = L( x, y (k 1)) where 1 k n 1 (5)
W (i, j k ) = p (i, j ) = p ( x, y ) where k = 0
Figure 3 shows all actions in the Window Based
Processing block to track the moving object using a
color histogram. For every image from a camera, the
operations of window based processing are performed
about red color, green color, and blue color, respectively.
Their values in a window are stored in registers to be
used for histogram generation.
The Histogram Generation block generates RGB
histograms, Model RGB Histogram, of the model and
the RGB histogram, Window RGB Histogram, of the
window, RGB Window, obtained from Window Based
Processing block. It consists of the model RGB
histogram generator, model RGB histograms, window
RGB histogram generator, and window RGB
histograms. The model RGB histogram generator
defines the model RGB histograms with a model setting
time. The reference model is located in the center of the
image, and then we push a button to initialize the
reference model. The RGB histograms of the model are
stored in the model RGB histograms. The window RGB
histogram generator makes the RGB histograms of the
window containing each pixel and its neighboring pixels
together. The RGB histograms of the window are stored
in the window RGB histograms. The histogram
generator has a parallel and pipeline structure to
improve processing speed. The RGB histogram of the
current window of each red color, green color, and blue
color is calculated concurrently during 5 steps. It needs
a total of 5 clocks to complete the matching operation.
Therefore the matching result from this module has a 5
clock delay. We should compensate for this delay in the
calculation of the estimated position of the object in the
Histogram Matching block.

With these operations, the pixel values in the lines of

an image are stored, where the number of lines is the
same as that of a window row. Because each dual port
RAM stores one line of an image, it is possible to get
one pixel value from every line simultaneously.
The window buffer stores pixel values moving from

1165

Fig. 3.

Schema of all actions in the window-based processing block.

In the Histogram Matching block, the RGB histogram

comparator performs the comparison of the matching
between the model RGB histograms and the window
RGB histogram for every pixel to find the best matching
position ( x , y ) in the image.
3.4 Trajectory presenter
The Trajectory Presenter module presents the position
of the target, Print, on the image using the result,
Trajectory, from the Color Histogram Based Tracker.
The position of the target is presented as the square with
the cross. This module makes and stores the coordinates
of the square with the cross about the trajectory
obtained from the Color Histogram Based Tracker
module. When the position of the current pixel is the
same as the coordinates of the square with the cross, it
transfers a valid signal, Print, to the Digital Video
Output module.
3.5 Digital video output
In the Digital Video Output module, the Digital
Visual Interface specification is applied to this module
to display the image sequence to the LCD monitor. This
module generates the sync signals and image data, DVI,
for DVI using the image sync signals, Sync_HSV,
including the vertical sync signal, horizontal sync signal,
and pixel clock, and digital color image data, RGB_HSV,
including red, green, and blue colors are obtained from
the other modules.

4. EXPERIMENT
The circuit of the color feature tracking using color
histograms is designed using VHDL and is implemented
in a Virtex-4 XC4VLX200 FPGA, with 200,448 Logic
Cells (abotut16M system gates), 6,048 Kbits Block
SelectRAM, and 960 user I/O pins [12]. Table I
indicates a summary of the device utilization
characteristics for the circuit for color feature tracking
using color histograms. Figure 4 shows a color feature
tracking system using this FPGA. This FPGA can
acquire an image from a digital camera, output the
trajectory data of the target, and display the tracking
images.
A high frame processing rate and low latency are
important for many applications that must provide quick
decisions based on events in the scene [13]. When the
color feature tracking system, using the proposed circuit,
is applied to a digital color camera, which produces
images consisting of 640480 pixels and transfers
images through a Camera Link interface, a sample
window consisting of 1515 pixels, it is capable of
processing the images at speeds of up to 81 fps (frames
per second). It can track robustly a moving object and
display these processed images on the LCD monitor.
The performance of this circuit is measured by
evaluating the average maximum delay time. The
performance of the color feature tracking system using
color histograms is shown in Table II. The performance
of the software program which is determined by

Fig. 4. A color feature tracking system employing a color feature

tracking circuit using color histograms.
TABLE I
DEVICE UTILIZATION CHARACTERISTICS FOR THE COLOR FEATURE
TRACKING CIRCUIT USING COLOR HISTOGRAMS
Device Utilization Summary
Logic Utilization
Number of Slice Flip Flops: 10,297 out of 178,176 5%
Number of 4 input LUTs: 57,693 out of 178,176 32%
Logic Distribution
Number of occupied Slices: 31,787 out of 89,088 35%
57,855 out of 178,176 32%
Total Number 4 input LUTs
122 out of
960 12%
Number of bonded IOBs:
1 out of
32 3%
Number of GCLKs:
45 out of
336 13%
Number used as RAMB16s:
48 out of
96 50%
Number of DSP48s:
Total equivalent gate count for design
3,411,214

measuring the average computation time required for

performing repeated operations on the PC, in this case
an Intel Pentium 4 CPU (4.3 GHz), 1GB DDR SDRAM
PC3200 (400 MHz), with Microsoft Windows XP
professional, and Microsoft Visual Studio. All of the
software programs are developed using Microsoft
Visual C++. When the color feature tracking system,
using the software program, is applied to a same
condition with the hardware experiment, it is capable of
processing the images at speeds of up to 9 fps and 0.15
fps as shown in Table II. The performance of the
software program decrease dramatically as the number
of the samples being compared increases.
Figure 5 and 6 show the results obtained from the
tracking of moving objects in an image sequence using
the color feature tracking circuit using color histograms.
We establish the model histogram of two dolls with blue
color and red color, respectively. It is trained by placing
it in front of the camera and saving the corresponding
color pattern as the target reference model when we
push a button. In this experiment, a doll is tracked
which is moved first horizontally, then vertically, and
afterwards, in a circular way. The trajectory, which the
color feature tracking system outputs at time t, is
illustrated in Fig. 5. Figure 6 shows several frames at
TABLE II
PERFORMANCE OF THE COLOR FEATURE TRACKING SYSTEMS USING COLOR
HISTOGRAMS
Performance of the Visual Tracking Systems
Experimental Environment
Performance
Software probabilistic visual tracking
110 ms (9 fps)
system: 1,000 samples
Software visual tracking system:
6,531 ms (0.15 fps)
640480 samples
Hardware visual tracking system:
12.34 ms (81 fps)
640480 samples

1166

the time in more detail. There are some delays between

the images shown in Fig. 6.(a)-(d) and Fig. 6.(e)-(h),
respectively. In Fig. 6.(a)-(d), the doll with blue color is
moving circularly, and the doll with red color is moving
circularly in Fig. 6.(e)-(h). The white square indicates
the trajectory of moving doll in the image sequence. It
shows that the color object can be tracked successfully
in real-time.

interpretation, and reasoning. We have demonstrated

that this real-time color feature tracking circuit,
combined with other technologies, can produce effective
and powerful applications.

REFERENCES
[1]
[2]

5. CONCLUSION
We presented a real-time color feature tracking
system employing a color feature tracking circuit using
color histograms. It offers real-time robust tracking
performance as it can search all areas of an image to
perform a matching operation, can process the images
with VGA resolution at speeds of up to 81 fps. Since the
proposed circuit has been designed using VHDL and
implemented in an FPGA, it can help make a smaller,
lower cost, and lower power color feature tracking
system. It is easy, as required, to modify and update this
circuit. When the proposed circuit is used in a system
which requires the color feature tracking function, only
a small percentage of the system resources are allocated
for color feature tracking. The remainder of the
resources can be assigned to preprocessing stages or to
high level tasks such as recognition, trajectory

[3]

[4]
[5]
[6]

[7]
[8]

TRACK_X
700

TRACK_Y

[9]

600

Coordinates

500

[10]

400
300
200

[11]

100
0
0

40

80

120

160

200

240

280

320

360

400

440

480

Time

[12]
Fig. 5. A trajectory which the color feature tracking system outputs at
time t. The blue line is the trajectory about the x-coordinates. The
pink line is the trajectory about the y-coordinates.

(a) moving blue doll at frame tb.

(b) moving blue doll at frame tb+.

[13]

C. Kwok, D. Fox, M. Meila, Adaptive Real-Time Particle

Filter for Robot Localization, Proceedings of Robotics and
Automation, Vol. 2, pp. 2836-2841, September 2003.
Y. Cui, S. Samarasekera, Q. Huang, M Greiffenhangen,
Indoor Monitoring Via the Collaboration Between a
Peripheral Sensor and a Foveal Sensor, Proceedings of Visual
Surveillance, pp. 2-9, January 1998.
G. R. Bradski, Real Time Face and Object Tracking as a
Component of a Perceptual User Interface, IEEE workshop on
Applications of Computer Vision, Princeton, pp. 214-219,
October 1998.
S. S. Intille, J. W. Davis, A. F. Bobick, Real-Time
Closed-World Tracking, Proceedings of Vision and Pattern
Recognition, pp. 697-703, June 1997.
M. Betke, E. Haritaoglu, S. S. Davis, Multiple Vehicle
Detection and Tracking in Hard Real-Time, Proceedings of
Intelligent Vehicles Symposium, pp. 351-356, September 1996.
A. Eleftheriadis, A. Jacquin, Automatic Face Location
Detection and Tracking for Model-Assisted Coding of Video
Teleconference Sequence at Low Bit Rates, Signal Processing
Image Communication, Vol. 7, pp. 231-248, November 1995.
A. Arsenio, J. Santos-Victor, Robust Visual Tracking by an
Active Obsrver, Proceedings of Intelligent Robots and
Systems, Vol. 3, pp. 1342-1347, September 1997.
T. Xiong, C. Debrunner, Monte Carlo Visual Tracking Using
Color Histograms and a Spatially Weighted Oriented Hausdorff
Measure, Lecture Notes in Computer Science, Computer
Analysis of Images and Patterns, Vol 2756, pp. 190-197,
Springer Berlin Heidelberg, 2003.
K. Nummiaro, E. Koller-Meier, L. Van Gool, A Color-Based
Particle Filter, Proceedings of Generative-Model-Based
Vision, pp. 53-60, June 2002.
Z. Zivkovic, B. Krose, An EM-like algorithm for
color-histogram-based object tracking, Proceedings of IEEE
Computer Vision and Pattern Recognition, Vol. 1, pp.
I-798-I_803, June 2004.
J. Czyz, B. Ristic, B. Macq, A Color-Based Particle Filter for
Joint Detection and Tracking of Multiple Objects,
Proceedings of IEEE Acoustics, Speech, and Signal Processing,
Vol. 2, pp. 217-220, March 2005.
Xilinx Inc., Virtex-4 Data Sheets: Virtex-4 Family Overview,
available from www.xilinx.com, February 2006.
J. I. Woodfill, G. Gordon, R. Buck, Tyzx DeepSea High Speed
Stereo Vision System, Proceedings of Computer Vision and
Pattern Recognition, pp. 41-45, June 2004.

(c) moving blue doll at frame tb+2.

(d) moving blue doll at frame tb+3.

(e) moving red doll at frame tr.

(f) moving red doll at frame tr+.
(g) moving red doll at frame tr+2.
(h) moving red doll at frame tr+3.
Fig. 6. Experimental Results of the color feature tracking system. The circular moving dolls with blue color and red color are tracked by proposed color
feature tracking system employing a color feature tracking circuit using color histograms.

1167