An Eye mouse Control System based on FPGA

Technology
1
1
2

Yih-Ran Sheu
Juing-Shian Chiou
Cheng-Hsin Chuang
1
1
1
Nguyen Van Binh
Wen-Li Shen
Yao-Wei Tseng
1
Department of Electrical Engineering,
2
Department of Mechanical Engineering,
Southern Taiwan University of Science and Technology,
Tainan, TAIWAN
sheu@mail.stust.edu.tw

Abstract
The objective of this paper present a set of techniques
integrated into an eye-mouse system using a field
programmable gate array (FPGA). Our system requires a
camera (TRDB-D5M), FPGA-DE2 board and a personal
computer. A field programmable gate array (FPGA) is used
to control the CCD camera and execute the image
processing operations and control eye-mouse of computer
via RS232 protocol. The coordinate of eye-mouse in our
system is determined by the system based on FPGA
imaging analysis. No computer is necessary in image
processing. Good efficiency can be seen using FPGA
hardware approach to determine the eye-mouse coordinate
per image frame. The color information of eye is used to
estimate the direction of pupil movement and then uses the
direction information to control the mouse of computer.
Keywords: FPGA, eye mouse, direction of pupil
I. INTRODUCTION
Nowadays, eye tracking and eye mouse system more and
more are widely. There are many eye tracking system using
image processing by computer supported by many image
processing tools such as Matlab, Visual C, Visual
Basicetc, but these systems reduce the speed of the
computer and sometimes cause unexpected situations. Thus,
processing image to detect eye movements using FPGA
hardware bring an alternative solution. In this paper, we
practically design and develop the improved eye-mouse
device with TRD-D5M camera, FPGA hardware, and
computer. An eye tracking systems is a system that can
track the movements of the user's eyes. Eye mouse system
is a very good approach for people who do not have the
ability to perform the movements of common control on
computer such as mouse drag and mouse clicketc. In this
paper, we perform image processing to detect pupil
movement in FPGA hardware then sends control
information to the computer to control a cursor on the
screen. There are several different ways to track the
directions of eye movements. Each approach has its own
advantages, disadvantages, and limitations. We use
information about eye color to detect the movement of the
eye. There are many color spaces such as RGB, HSI, YUV,
YIQ ... We choose the YUV color space because it has
many advantages and simple. The image processing in
FPGA is not as good as image processing in personal
computer but this process is totally independent, thus

increasing the processing speed of the system. The eye

mouse system is used to access computers via eye
movements.
II. SYSTEM DESIGN
The eye-mouse system consists of two modules. The first
module including second -stage algorithm is proposed to
implement image processing and pupil movement detection
on FPGA hardware. An efficient pupil detection based on
the pupil color information is employed to find the pupil
and corner of users eye. After this, some algorithms is
used to detect the direction information of pupil/corner of
eye then transmit to computer via RS232 communication
protocol. The second module is software on computer
including cursor control block, user interface block, and
application. The computed direction information of pupil
movements is used to control the mouse of computer.
Through the graphic user interface, users can implement
some application.
The component of eye mouse system is introduced as
follows.

Fig.1 The component of eye mouse system

II-1. HARWARE SPECCIFICATIONS
The system consists of a personal computer, FPGA

hardware with TRD-D5M camera and RS232 cable,

FPGA hardware with a video capture, zoom buttons and
adjust contrast. A real-time eye-tracking system requires an
image capturing device operating at an extremely high
frame rate to be able to effectively track the pupil in the
presence of saccadic motion. Therefore, a high-frame rate
image capture device is necessary in order to build a
general eye tracker system for various human-computer
interaction applications [1]. The TRDB_D5M Kit satisfies
most of the above requirements to develop a 5 Mega
Pixel Digital Camera on the Altera DE2-70 / DE2 boards.

a. The definition of
thresholding
g(i,j) =

255, f(i, j) T

(2)
0, f(i, j) T

In accordance with the above definition, we can find a

pupil value. A set of dark spots where the gray level is
lower than the pupil value is exactly the pupil position we
are searching for. If the binary threshold is not appropriate,
the search result will show a dark spot which is unexpected
[5].

a. Original image

Fig.2 The algorithm diagram for the proposed eye tracking

system.
II-2 Image processing of an eye-mouse system
The image input through the camera is transformed by
FPGA hardware to obtain the image data of the plane
coordinate. Every point (i,j) on the coordinate plane has its
own image data. With this image information, the system is
able to identify the pupil of eye from the image[5].Our
methods based on color information about the pupil of
eye to determine the pixel according to the definition
given. The first problem is to represent images in the
appropriate color space that allows to distinguish two
components: composition luminance, luminous intensity
and color components. RGB color space generally
inappropriate because it mixes the two components.
Although there are other color models such as CMY, YIQ
and HSI, in this paper YUV color model is chose because
of its advantages. In YUV color model, the luminance
information is represented with Y component while the
color information is represented by the U and V
components. The main idea of using color-space
information is to increase the information available about
an image. The pupil feature detection system based on
eigenfeature approach is used for gray-level Y and U, V
color information of image. YUV color components can
be obtained by using the RGB color coefficients as follows
[6].

0.299 0.587 0.114R

- 0.148 -
0.289
0.437 G (1)

0.615 - 0.515 0.100B

The firt thing to do is the thresholding.

b. Intensity image histogram

Fig.3 a. Original image b. Intensity image histogram

a. Y image

b. Gray image of pupil tracking process

Fig.4 a. Y image b. Y image of pupil tracking process
b. Calculation for the coordinate of the pupil
We use equation to calculate the coordinate of the pupil
center:
center searched by system.

1 K
1 K
XC,YC K Y (3)
K 1
1

X,

ATTENTION!
TRIAL LIMITATION - ONLY 3 SELECTED PAGES MAY BE CONVERTED PER CONVERSION.
Where K is the total number of the dark spot at which the
gray level is lower than the threshold after dynamic
binarizing of the eye image. X and Y are coordinates of the
black pixel. [5]

regions of the window is active. For other complicated

applications which require the pupil of the eye moving to
more precise location of the window, only information
detected from the color of the eye of people is not enough.
In addition,
changes in light conditions also reduce the
accuracy of the system. Therefore, in these cases we need
to combine information about color and other methods as
detect edges, Hough Transform to upgrade the system.
Some exact results are shown in the picture.

PURCHASING A LICENSE REMOVES THIS LIMITATION. TO DO SO, PLEASE CLICK ON THE FOLLOWING LINK:
http://www.investintech.com/order_a2e_pro.htm

Fig.5 The result of calculation for the coordinate of the

pupil center.
II-2-2 Eye movement detection
After we have located the pupil center point from the eye
we can proceed to extract the direction information from
the pupil movements. The directions of the pupil
movements are quantized into three directions: right, left,
and center. The third different position of pupil center point We can see that the pupil center point has Xc =156 so it is
center.
corresponding to the third region on the picture with size
320x240 pixel is illustrated as follows.

Fig.6 The 3 different regions of pupil movement

corresponding to the 3 directions of mouse.
In short, we can determine the directions of eye movements
by pupil center position. Example: The pupils center
position is at P(156,99), so pupil is center position. The
pupils center position is at P(192,118), so pupil is right
movement. The pupils center position is at P(75,111), so
pupil is left movement. After the direction of the eye
movement has been computed, we can use the direction
information to move the cursor on the screen.

We can see that the pupil center point has X C=75 so it is

left movement.

II-2-3 Communication between FPGA hardware and

computer
The direction information of pupil is transmitted to
computer by RS232 or USB protocol communication. This
paper we use RS232 protocol communication. Visual Basic
software is used to receive the direction information of the
eye movements from FPGA hardware. Although the baud
rate of RS232 port is lower than USB, this speed is enough to
satisfy our requirement.
III. EXPERIMENTAL RESULTS
III-1 Eye Tracking Experiment
In this paper, we only perform a simple mouse eye system.
The obtained results are quite accurate because only three

We can see that the pupil center point has