A Static Hand Gesture and Face Recognition System For Blind

People
ABSTRACT

This project presents a recognition system, which can be helpful for a blind person. Hand
gesture recognition system and face recognition system has been implemented in this project
using which various tasks can be performed. Dynamic images are being taken from a
dynamic video and is being processed according to certain algorithms. In the Hand gesture
system In this we focus on using pointing behaviour for a natural interface, Hand gesture
recognition based human-machine interface is being developed vigorously in recent years.
Due to the effect of lighting and complex background, most visual hand gesture recognition
systems work only under restricted environment. According to gesture Recognized, various
tasks can be performed like turning on the fan or lights. While in face recognition, Haar
Cascade Classifiers and LBPH recognizer are being used for face detection and recognition
respectively.
CHAPTER 1

INTRODUCTION
This system acts as an Intelligent virtual assistant which helps in bringing the world
closer and helps in meeting different ends together. Through this system use of facial
recognition and hand gesture to assist and work with the environment to make it a better
place to live for the blind. There are different ways in which it is helping to make this happen
and for that hand gesture and facial recognition, system is being used. This system is using
computer vision technology this tech has been used for human computer interaction (HCI)
using a physical medium where hand gesture and facial recognition plays a major role. Hand
gestures have been done since the dawn of civilization and have various meaning depending
on the geographical location. Hand gesture have various application in military gaming etc.
Methods without using computer vision has also been developed like the wearable gloves but
they are too costly as they need sensors and other hardware devices there are numerous
algorithm for hand gesture recognition like KNN (K Nearest Neighbor), artificial neural
network but most of the algorithm require large amount of samples for training and
recognition to overcome this problem convex hull and convexity hull defects are used for
gesture recognition. The recognition of hand gestures are of two types, that is, Dynamic
gesture and static gesture, so in this research we have covered the section of static gesture
recognition. In facial recognition system, two methods are used Haar cascade method and
Linear Binary Pattern (LBP) for better prediction we use LBP method. In 2004, a Haar
cascade classifier technique was proposed by Voila Jones which has been a motivation to
various face recognition systems.

LBP is used as a visual description in which classification using computer vision is

made using open cv each image is converted into a series of code. a pixel is taken and then
the image is given a particular value. This value is predefined based on a series of codes. A
LBP histogram mainly works on four parameters: radius, neighbor, Grid X, Grid Y. A radius
has an initial value 1, which builds a central pixel, which is used to build circular local binary
pattern. Neighbors on the other hand are used to create a sample points to increase the
accuracy of facial detection but it comes with drawback of increasing computational cost.
Grid x and Grid y are horizontal and vertical features to recognize an image.
 Moreover, for real-world applications, e.g. e-passport, driving license, or ID card
identification, only one or very few face images of the subject of interest might be captured
during the data acquisition stage. As a result, one would encounter the challenging task of
undersampled face recognition. Existing solutions to undersampled face recognition can be
typically divided into two categories: patch-based methods and generic learning from
external data. For patch-based methods, one can either extract discriminative information
from patches collected by different images, or utilize/integrate the corresponding
classification results for achieving recognition.

CHAPTER 2

LITERATURE SURVEY
1. Lianzhi Tan ; Zhifeng Li ; Qiao Yu Deep face attributes recognition using spatial
transformer network Information and Automation (ICIA), 2016 IEEE International
Conference on 1-3 Aug. 2016

Face alignment is very crucial to the task of face attributes recognition. The
performance of face attributes recognition would notably degrade if the fiducial points of the
original face images are not precisely detected due to large lighting, pose and occlusion
variations. In order to alleviate this problem, we propose a spatial transform based deep
CNNs to improve the performance of face attributes recognition. In this approach, we first
learn appropriate transformation parameters by a carefully designed spatial transformer
network called LoNet to align the original face images, and then recognize the face attributes
based on the aligned face images using a deep network called ClNet. To the best of our
knowledge, this is the first attempt to use spatial transformer network in face attributes
recognition task. Extensive experiments on two large and challenging databases (CelebA and
LFWA) clearly demonstrate the effectiveness of the proposed approach over the current state-
of-the-art.
2. Kai Liu ; Seungbin Moon Robust dual-stage face recognition method using PCA and
high-dimensional-LBP Information and Automation (ICIA), 2016 IEEE International
Conference on 1-3 Aug. 2016

In this paper, we propose a dual-stage face recognition method which utilized both
holistic and local features-based recognition algorithms. In the first stage Principal
Components Analysis (PCA) is utilized to recognize test image. If the confidence level test is
passed, the recognition process will be terminated. Otherwise, the second stage where High
Dimensional Local Binary Patterns (HDLBP) is employed will be pursued. The performance
of this hybrid method is evaluated on CMU-PIE database, and we obtain improved
recognition rate than PCA alone.

3. M Shujah Islam Sameem ; Tehreem Qasim ; Khush Bakhat Real time recognition of
human faces Open Source Systems & Technologies (ICOSST), 2016 International
Conference on 15-17 Dec. 2016

A simple yet efficient face detection and recognition system is proposed in this paper which
has the capability to recognize human faces in single as well as multiple face images in a
database in real time. Preprocessing of the proposed frame work includes noise removal and
hole filling in color images. After preprocessing, face detection is performed by using viola
jones algorithm. Detected faces are cropped out of the input image to make computation fast.
SURF features are extracted from the cropped image. For face matching, putative feature
matching is carried out and outliers are removed using M-estimator Sample Consensus
(MSAC) algorithm. Single as well as multiple person color images from class persons of
Graz 01 dataset are used to evaluate the system.

4. Bappaditya Mandal Face recognition: Perspectives from the real world Control,
Automation, Robotics and Vision (ICARCV), 2016 14th International Conference on 13-
15 Nov. 2016

In this paper, we analyze some of our real-world deployment of face recognition (FR)
systems for various applications and discuss the gaps between expectations of the user and
what the system can deliver. We evaluate some of the existing algorithms with modifications
for applications including FR on wearable devices (like Google Glass) for improving social
interactions, monitoring of elderly people in senior citizens centers, FR of children in child
care centers and face matching between a scanned IC/passport face image and few live
webcam images for automatic hotel/resort checkout or clearance. Each of these applications
poses unique challenges and demands specific research components so as to adapt in the
actual sites.

5. S. V. Tathe ; A. S. Narote ; S. P. Narote Face detection and recognition in videos India

Conference (INDICON), 2016 IEEE Annual 16-18 Dec. 2016

Advancement in computer technology has made possible to evoke new video

processing applications in field of biometric face detection and recognition. It has wide range
of applications in human recognition, human computer interaction (HCI), behavior analysis,
teleconferencing and video surveillance. Face is vital part of human anatomy that reflects
prominent topographies of a person. Face detection has become popular biometric trait in
recent years due to its importance in security control applications. The first step in practical
face analysis systems is real-time detection of face in frames containing face and complex
objects in background. In this paper a system is proposed for human face detection using
Haar features and recognition using Eigen and Gabor filter in videos. Efforts are made to
minimize processing time for detection and recognition process. The Eigenface method
performs well in terms of computational complexity whereas Gabor filter are robust to pose
changes.

6. Shanshan Guo ; Shiyu Chen ; Yanjie Li Face recognition based on convolutional

neural network and support vector machine Information and Automation (ICIA), 2016
IEEE International Conference on 1-3 Aug. 2016

Face recognition is an important embodiment of human-computer interaction, which has

been widely used in access control system, monitoring system and identity verification.
However, since face images vary with expressions, ages, as well as poses of people and
illumination conditions, the face images of the same sample might be different, which makes
face recognition difficult. There are two main requirements in face recognition, the high
recognition rate and less training time. In this paper, we combine Convolutional Neural
Network (CNN) and Support Vector Machine (SVM) to recognize face images. CNN is used
as a feature extractor to acquire remarkable features automatically. We first pre-train our
CNN by ancillary data to get the updated weights, and then train the CNN by the target
dataset to extract more hidden facial features. Finally we use SVM as our classifier instead of
CNN to recognize all the classes. With the input of facial features extracted from CNN, SVM
will recognize face images more accurately. In our experiments, some face images in the
Casia-Webfaces database are used for pre-training, and FERET database is used for training
and testing. The results in experiments demonstrate the efficiency with high recognition rate
and less training time.

7. Hsiang-Yueh. Lai, Han-Jheng. Lai, "Real-time dynamic hand gesture recognition", 2014
International Symposium on Computer Consumer and Control (IS3C, pp. 658-661, June
2014.
A real time dynamic hand gesture recognition system is performed in this paper. The eleven
kinds of hand gestures have been dynamic recognized, which represent the number from
one to nine. The dynamic images are caught by a dynamic video. We use the YCbCr color
space transformation to detect the skin color and to find the hand contour from the complex
background. Convex defect character points of the hand contour are defined, and the finger
angles and fingertip positions are calculated to recognize the hand gestures. The OpenCV is
used to perform our research. Ten tested users produce 330 cases to recognize eleven hand
gestures, and each hand gestures have three different poses of hand gestures. The accurate
recognition rate is 95.1%.

8. Srinivas Ganapathyraju, “Hand Gesture Recognition Using Convexity Hull Defects to

Control an Industrial Robot” , 2013 3rd International Conference on Instrumentation
Control and Automation (ICA) Bali, Indonesia, pp. 63-67, 2013.

This paper presents a method for hand gesture recognition to control a 6 axis
industrial robot. The image is acquired by means of a web cam system, and
undergoes several processing stages before a meaningful recognition can be
achieved. Some of these stages include skin detection to effectively capture only the
skin region of the hand, noise elimination, application of the convex hull algorithm to
get the outline of the hand, and apply convexity hull defects algorithm to determine
the finger count. Once the finger count has been determined, the information is
transmitted via serial communication to the industrial robot controller. The end
effector of the robot moves in four different directions based on the finger count input
received from the hand gesture recognition module.
CHAPTER 5

EXISTING SYSTEM

This method presents hand gesture recognition to control a 6 axis industrial robot. The image
is acquired by means of a web cam system, and undergoes several processing stages before a
meaningful recognition can be achieved. Some of these stages include skin detection to
effectively capture only the skin region of the hand, noise elimination, application of the
convex hull algorithm to get the outline of the hand, and apply convexity hull defects
algorithm to determine the finger count. Once the finger count has been determined, the
information is transmitted via serial communication to the industrial robot controller. The end
effectors of the robot moves in four different directions based on the finger count input
received from the hand gesture recognition module.

Simultaneous face hallucination and face recognition, here they consider the
algorithms HSV and HSI separately. The major difference between these two algorithms is in
the selection of the LR-HR training pairs when performing face super-resolution for face
verification and identification

In recent decades, due to computer software and hardware technologies of continuous

innovation and breakthrough, the social life and information technology have a very close
relationship in the twenty-first century. In the future, especially the interfaces of consumer
electronics products (e.g. smart phones, games and infota1inment systems) will have more
and more functions and be complex. How to develop a convenient human-machineInterface
(HumanMachine Interaction/Interface, HMI) for each consumer electronics product has
become an important issue. The traditional electronic input devices, such as mouse, keyboard,
and joystick are still the most common interaction way. However, it does not mean that these
devices are the most convenient and natural input devices for most users. Since ancient times,
gestures are a major way for communication and interaction between people. People can
easily express the idea by gestures before the invention of language. Nowadays, gestures still
are naturally used by many people and especially are the most major and nature interaction
way for deaf people [1]. In recent years, the gesture control technique has become a new
developmental trend for many humanbased electronics products, such as computers,
televisions, and games. This technique let people can control these products more naturally,
intuitively andIn case of existing system . The objective of this paper is to develop a real time
hand gesture recognition system based on adaptive color HSV model and motion history
image (MHI). By adaptive skin color model, the effects from lighting, environment, and
camera can be greatly reduced, and the robustness of hand gesture recognition could be
greatly improved.
CHAPTER 6

PROPSOED SYSTEM

The input is taken from front facing Camera. The proposed Recognition System is divided
into two parts:

• Hand Gesture Recognition

• Facial Recognition

Hand Gesture Recognition subsystem is divided into three parts. They are Hand region
detection, pulling out the features, Gesture Recognition. Facial Recognition is subdivided into
Face detection, Face Identification.

Most gesture recognition methods usually contain three major stages. The first stage is the
object detection. The target of this stage is to detect hand objects in the digital images or
videos. Many environment and image problems are needed to solve at this stage to ensure
that the hand contours or regions can be extracted precisely to enhance the recognition
accuracy. Common image problems contain unstable brightness, noise, poor resolution and
contrast. The better environment and camera devices can effectively improve these problems.
However, it is hard to control when the gesture recognition system is working in the real
environment or is become a product. Hence, the image processing method is a better solution
to solve these image problems to construct an adaptive and robust gesture recognition system.
The second stage is object recognition. The detected hand objects are recognized to identify
the gestures. At this stage, differentiated features and effective classifiers selection are a
major issue in most researches. The third stage is to analyze sequential gestures to identify
users’ instructs or behaviors.

CHAPTER 7

DESTIN METHODOLOGY

FACE RECOGNITION
Facial recognition is based on face descriptors. The system calculates the similarity between
the input face descriptor and all face descriptors previously stored in a gallery. The goal is to
find the face(s) from the gallery that are most similar to the input face.

HAND GESTURE RECOGNITION SYSTEM

The proposed architecture consists of THREE stages:

(1) Hand detection

(2) Hand gesture recognition

(3) Finger detection

Now we introduce our part-based hand gesture recognition system. Which illustrates the
framework, which consists of two major modules: hand detection and hand gesture
recognition
 Figure 1: Proposed System Architecture for Hand Gesture Recognition

Hand Detection:

we use camera as the input device, which captures the color image and the depth map at
640*480 resolution. In order to segment the hand shape, we locate the hand position using the
hand tracking function. Then, by thresholding from the hand position with a certain depth
interval, a rough hand region can be obtained. Second, we require the user to wear a black
belt on the gesturing hands wrist, in order to more accurately segment the hand shape. After
detecting the black color pixels, we use RANSAC to a line to locate the black belt. The hand
shape is generally of 100*100 pixel resolution, with possibly severe distortions. After
detecting the hand shape, we represent it as a time-series curve. Such a shape representation
has been successfully used for the classification and clustering of shapes. The time-series
curve records the relative distance between each contour vertex and a center point. We define
the center point as the point with the maximal distance after Distance Transform on the shape
(the cyan point); and the initial point (the red point) is according to the RANSAC line
detected from the black belt (the green line). In our time series representation, the horizontal
axis denotes the angle between each contour vertex and the initial point relative to the center
point, normalized by 360 . The vertical axis denote the Euclidean distance between the
contour vertices and the center point, normalized by the radius of the maximal inscribed
circle. the time-series curve captures nice topological properties of the hand, such as the
finger parts.

PULLING OUT THE FEATURES

Contour Detection has been used to pull out the features. It is one of the famous edge
detection method. A contour is an arc connecting the continuous points having identical
Colour or any feature. Contour approximation is done using Ramer Douglas Puecker
algorithm, which reduces a number of points in a curve by approximating a succession of
points. After the contour is detected, a convex hull is being created around it. Convex hull is a
compact convex bound around the detected shape. Any divergence from the main hull acts as
defect so the gaps between the fingers act as a convexity defect [6]. Using these defects
further recognition of the gesture is done.

GESTURE RECOGNITION

In order to count the fingers and find the angle between the fingers three major points are
being extracted –

• Starting point(St)

• Centre point(Ce)

• End point(En)
FACE DETECTION

It is an object detection method used to discover faces or cars or any object. This system is
provided with positive and negative images and selection of features along with the classifier
training and integral images. Each feature is the difference sum of the pixels within two
rectangular regions. These rectangular regions are nothing but the darker and the lighter
regions. These sectors have same size and shape horizontally or vertically. To find the sum of
pixels, under black and white regions, the concept of integral images is introduced. No matter
how large may be the number of pixels; it operates over only a fixed number of pixels.

AdaBoost does the selection of most appropriate features out of a large set of features. Weak
classifiers are combined to form the final classifier. They are weak because alone they cannot
identify the object. Each stage of cascade must not have a low false negative rate as when
face is classified as non-object then the classification stops. Each stage must have a high false
positive rate which means the erroneous detection of an object as face, thus this error can be
corrected in (n+1)th stage and succeeding stages in classifier.
FACE IDENTIFICATION

LBP, abbreviation for Local Binary Pattern, is the measure of fixed grey scale texture
obtained by thresholding the nearby area of each pixel and the result obtained is considered
as a binary number. The LBP operator is the ratios of the pixel intensities of the center and
the surrounding pixels and can be

computed using the equation

LBP uses the following steps to identify the face

• Firstly, the image is converted to grayscale.
• Section of the image is obtained through 3*3 window.
• Central value of matrix used as threshold, which is used to define new values of other eight
neighbours.
• New value calculated is either 1 or 0 for values higher or lower than threshold respectively
and the concatenation of the Binary values is done.
• Conversion of Binary to Decimal is done in order to get the pixel from the original image.
• Histogram using the GridX and GridY parameters, which is the number of cells in
horizontal and vertical direction, is created which divides the image into multiple grids.
The current Histogram is compared with the histograms already existing in the database using
different distance techniques like Chi Square method, Euclidean Distance,

Absolute Value.

In addition, this distance gives the confidence, which determines the closeness between the
histograms. In the Research, major tasks performed were Detecting, capturing, training,
recognizing the face.

Detection and capturing

The face is detected in real time and when a user wants to save the image, a certain indication
is given to the system, after which the system crops the face detected in the real time system.
This Detection of the face is done through object classifier technique that is Haar cascade
classifier. Then the System asks for the name with a question “who are you” with which a
folder is stored and image with a name at that instant is stored. Pyttsx3 Library has been used
to convert Text to speech, which uses Natural Language Processing methods to achieve such
task. The major advantage to use this library is that the task can be performed offline. This
image is stored in black and white scheme for further processing.
CHAPTER 8

ADVANTAGES AND DISADVANTAGES

ADVANTAGES

• Combines the benefits of existing authentication system to form 3 levels of secure

password.

• Provides security against hackers and robot attacks.

• Unauthorized users cannot access the system.

• Low power consumption.

• Easy to implement.

• Non intrusive

• Cheap technology.

•  Verification time is about five seconds.

• Very high accuracy.

• Is the most economical biometric PC user authentication technique.

• Confidential Data will be saved from thefts

DISADVANTAGES

 Requires special hardware.

 People feel inconvenience.

CONCLUSION

The system in this project can be helpful for a blind person and can act as a virtual assistant
for it. Haar cascade Classifiers and LBPH recognizers has to be used for face detection and
identification in the real time whereas Convex hull and Convex defects algorithm have to be
used to detect the Hand gestures in real time. Skin color recognition is done in YCbCr color
space and different threshold ranges have been used to detect skin color in different lighting
conditions and skin color.

REFERENCES
[1] Srinivas Ganapathyraju, “Hand Gesture Recognition Using Convexity Hull Defects to
Control an Industrial Robot” , 2013 3rd International Conference on Instrumentation Control
and Automation (ICA) Bali, Indonesia, pp. 63-67, 2013.

[2] Voila P, Jones M J. Rapid object detection using a boosted cascade of simple features [C],
Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and
Pattern Recognition (ICCVPR 2001), vol. 1, pp. I-511-I-518, 8-14 Dec. 2001, Kauai, USA.

[3] Kelvin Salton do Prado, “Face recognition: Understanding LBPH Algorithm”, 2017,
Towards Data Science. [Online]. Available : https://towardsdatascience.com/face-
recognition-how-lbph-works- 90ec258c3d6b. . [Accessed : 7th January 2018]

[4] Hsiang-Yueh. Lai, Han-Jheng. Lai, "Real-time dynamic hand gesture recognition", 2014
International Symposium on Computer Consumer and Control (IS3C, pp. 658-661, June
2014.

[5] Muh. Arif Rahman, , I Ketut Edy Purnama, Mauridhi Hery Purnomo,” Simple Method of
Human Skin Detection using HSV and YCbCr Color Spaces”, 2014 International Conference
on Intelligent Autonomous Agents, Networks and Systems Bandung, Indonesia, August 19-
21,
2014.

[6] Hanene Elleuch, Ali Wali, Anis Samet, Adel M. Alimi, “A static hand gesture recognition
system for real time mobile device monitoring”, 2015 15th International Conference on
Intelligent Systems Design and Applications (ISDA), pp. 195 – 200, 2015.
[7] Gaganpreet Singh, , “Training Haar Cascade”, 2012. [Online]. Available
:https://singhgaganpreet.wordpress.com/2012/10/14/traininghaar cascade/. [Accessed : 5th
January 2018].

[8] Sander Soo, “Object detection using Haar-cascade Classifier”, Institute of Computer
Science, University of Tartu.

[9] L. Cuimei, Q. Zhiliang, J. Nan, W. Jianhua, "Human face detection algorithm via Haar
cascade classifier combined with three additional classifiers", 2017 13th IEEE Int. Conf.
Electron. Meas. Instruments, pp. 483-487, 2017.

[1]. X. Tan, S. Chen, Z.-H. Zhou, and F. Zhang, “Face recognition from a single image
per person: A survey,” Pattern Recognit., vol. 39, no. 9, pp. 1725–1745, Sep. 2006.

[2]. T. Ahonen, A. Hadid, and M. Pietikainen, “Face description with local binary
patterns: Application to face recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol.
28, no. 12, pp. 2037–2041, Dec. 2006.

[3]. J. Zou, Q. Ji, and G. Nagy, “A comparative study of local matching approach for face
recognition,” IEEE Trans. Image Process., vol. 16, no. 10, pp. 2617–2628, Oct. 2007.

[4]. J. Lu, Y.-P. Tan, and G. Wang, “Discriminative multimanifold analysis for face
recognition from a single training sample per person,” IEEE Trans. Pattern Anal. Mach.
Intell., vol. 35, no. 1, pp. 39–51, Jan. 2013.

[5]. M. Yang, L. Van Gool, and L. Zhang, “Sparse variation dictionary learning for face
recognition with a single training sample per person,” in Proc. IEEE Int. Conf. Comput.
Vis. (ICCV), Dec. 2013, pp. 689–696.

[6]. M. Yang, L. Zhang, J. Yang, and D. Zhang, “Robust sparse coding for face
recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2011, pp.
625–632.
[7]. M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An algorithm for designing
overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process., vol.
54, no. 11, pp. 4311–4322, Nov. 2006.

[8]. J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, “Robust face recognition
via sparse representation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 2, pp.
210–227, Feb. 2009.

[9]. C.-F. Chen, C.-P. Wei, and Y.-C. F. Wang, “Low-rank matrix recovery with structural
incoherence for robust face recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit. (CVPR), Jun. 2012, pp. 2618–2625.