International Journal of Computer Information Systems and Industrial Management Applications.

ISSN 2150-7988 Volume 10 (2018) pp. XXX-XXX

© MIR Labs, www.mirlabs.net/ijcisim/index.html

Alphabet and Number Recognition of Indonesian

Sign Language Using SIFT Algorithm
Isa Mahfudi1, Moechammad Sarosa2, Rosa Andrie Asmara3 and Muh Bambang Purwanto4
14
Graduate Student of Electrical Engineering Department, State Polytechnic of Malang,
Malang 65141, Indonesia
Isa_mahfudi@polinema.ac.id 1 and muh_bambang@polinema.ac.id 4
2
Department of Electrical Engineering, State Polytechnic of Malang,
Malang 65141, Indonesia
msarosa@polinema.ac.id 2
4
Department of Information Technology, State Polytechnic of Malang,
Malang 65141, Indonesia
rosa_andrie@polinema.ac.id 3

MIR Labs, USA

2
Abstract: The inability to
hear or talk to cause people
to become isolated and
feeling lonely. It really gives
the influence to their social
life. They are the deaf/mute
disabilities. They used sign
language as the primary
language in doing their
activity. Standard sign
language in the country of
Indonesian using Indonesian
Sign Language System
(ISLS) Dictionary. However,
not all people understand
sign language, and normal
people who does not
understand sign language,
will find difficult to
communicate with people
who use sign language to
communicate. To resolve this
problem, sign language
translation allows to help
deaf and mute people to
communicate with normal
people. Sign/Gesture
recognition is a technique to
recognize certain sign/gesture
and the end result can be
translated into form of a text
or sound. Scale Invariant
Feature Transform
Algorithm is one of method
for detecting object. This
method has a good
performance to changes in
scale, rotation, brightness
and noise This method will
have an added advantage by
combining the features of
another matching technique.
In this paper, researchers
proposed the SIFT method to
use on the recognition of
Number and Letter on
Indonesian sign language.
The best performance on the
recognitoin of the character
sign language (letters and
numbers) on the threshold
value of 0, 65 namely: “B, H,
“1”, “2”, “4”, “5” and “10”
with its 100% accuracy.
Change the threshold value
that gives best accuracy is at
the threshold value of 0,45
which produces 79,71%
accuracy.
Keywords: SIFT method,
Indonesian Sign Language,
Threshold.
I. Introduction
The inability to hear or
talk to cause people to
become isolated and feeling
lonely. It really gives the
influence to their social life.
They are the deaf/mute
disabilities. Sign language
generally used them in
communicating. They used
sign language as the
primary language in doing
their activity. Standard sign
language in the country of
Indonesian using
Indonesian Sign Language
System (ISLS)
Dictionary[1]. Sign/Gesture
has content specific
information, sometimes
used by normal people to
communicate as a hand tracker. Strain gauge
communication
nonstandard. this is
generally used to interact
with the computer[2].
Figure 1. Indonesian Sign
Language System (ISLS)
Dictionary[1].
However, not all people
understand sign language,
and normal people who
does not understand sign
language, will find difficult
to communicate with people
who use sign language to
communicate. To resolve
this problem, sign language
translation allows to help
deaf and mute people to
communicate with normal
people. Sign/Gesture
recognition is a technique to
recognize certain
sign/gesture and the end
result can be translated into
form of a text or sound.
Approach to the
recognizing of sign/gesture
Generally be classified into
three, namely: based on data
glove [2][3][4], using
sensors EMG
(electromyography) [3][5]
and based on vision [6][7]
[8][9].
This method of data
glove is a glove consist of
strain gauge sensor and
sensor is used to detect hand
and bend the finger, and
tracker is an electronic
system used to provide
information location,
orientation, speed and
direction of the hand. This
glove data produce good
performance and have a
high reliability in the
recognition of sign
language. This method
eliminated the phases of
pre-processing on vision
technique. The methods has
deficiency, namely: users
need to wear glove, this
system is quite expensive
because uses many sensor.
Many electronic devices
embedded in gloves and a
lot of cables are connected
between the computer and
the gloves, as well as have
the potential to inhibit the
comfort and movement
when using this gloves data
[2][3][4]. For vision based
technique, input device
using a camera that was
used to observe the hand
and its movement. In this
method, the signer is in
front of the camera and the
camera captures sign image,
and then in the process of
using vision method for
recognized and translated.
This method is relatively
inexpensive because
requires only a camera for
pictures and many
developed by the
researchers. For researchers
is a challenge in perfecting
the introduction in any use
of the application.
Researchers have many
presents various methods of
introduction sign/gesture.
Until now, the existing
methods are still not perfect
and will continue to do
development[6] [7].
Scale Invariant Feature
Transform Algorithm is one
of method for detecting
object [1]. This method has
a good performance to
changes in scale, rotation,
brightness and noise This
method will have an added
advantage by combining the
features of another
matching technique. In this
paper, researchers proposed
the SIFT method to use on
the recognition of Number
and Letter on Indonesian
sign language.
II. Related Work
Many researchers have been
done based on sign
language recognition using
vision based. For hand
detection technique
described by [10] and [11],
both use the HSV colour
models and YCbCr colour
model for the process of
segmentation on the skin
color of the hand.
Researcher [11] using
generic fourier descriptor
and nearest neighbour on
feature extraction process,
this research produces the
accuracy 86% for static data
and 69% for dynamic data.
In [12] using labview
recognize of sign language.
Researchers using static
data and dynamic data, the
accuracy obtained 97.79%
for static sign and 80.38%
for dynamic sign.
Researchers [9] using edge
orientation histogram. This
method produces
recognition level of 88.26%
within recognition time of
0.5 second. those results are
tested on a complex
background and mixed
lightning condition.
Researchers [13] combine
between edge detection
algorithm and skin
detection algorithm for
gesture identification, and
using ANN algorithm for
Static hand gesture
recognition. In [14],
researchers develop
recognition hand signals in
depth data using dynamic
How to Format Your Paper for JIAS
time warping. average
recognition rate of 92.4%.
Weighted hidden markov
models (WHMM) is also
used in research [15].
Weighted hidden markov
model is claimed capable of
overcoming the signs from
different signers. It is able
to provide accuracy of
94.74% every experiments.
The SIFT has on
implementations in different
fields. Researchers [16]
have conducted
implementation SIFT for
the automated recognition
of Batik patterns with batik
data collected as many as
120 image that has 20 basic
batik pattern. The use of
SIFT Matching method
produces an equal error rate
of 8.47%. In [17] the use of
SIFT combined with the
technique of bag of features
extracted at the recogntion
of batik pattern produces a
good performance. This
technique produces an
average accuracy of
97.67%, 95.47% and 79%
in normal image, rotated
image and a scaled image.
III. Proposed System
Sign language recognition
system is composed of four
main phases[1]. Image
Acquisition phase, pre-
processing and hand
segmentation phase, hand
posture recognition phase
and finally classification
gesture phase. Figure 2
shows the stages of the
process of the sign language
recognition.
3
(a)
Figure 4. Pre-Processing
and Hand Segmentation
Process
The YCbCr is a family of
colour spaces, Y is the luma
component and Cb and Cr
Figure 2. Block diagram of are the blue-difference and
Sign Language Recognition red difference chroma
(b) components. Calculation on
A. Image Acquicision Figure 3. Sign image the process of
data. (a) Number Sign transformation from RGB
The first step would
definitely be capturing all language (b) Letter Sign colour space to YCbCr
character of sign image by Language. colour space shown in
webcam for further Equation 1[10].
processing. all sign image B. Pre-Processing and
were collected, it aims to Y  0,299  R  0,587  G  0,1
Hand Segmentation
create a database for Cb  0,1687  R  0,3312  G
The images were captured
training and testing data. Cr  0,5  R  0,4183 * G  0
by the webcam and then
Sign language used on this
done the segmentation (1)
research is standard of
process to eliminate the
indonesia sign language Skin detection
background and to get the
(ISLS). Total of sign processing shown in Fig 5.
hand object. This process is
character used is 34 static Transformation result into
usually called skin
data include: the letter YcbCr colour space shown
detection. The sign image
consists of 24 (without 2 in Fig 5(b). Transformation
captured is composed of
letter: J and Z) and number result from YCbCr into
RGB color spaces, This
consist of 10 digits (1-10). RGB and as result of skin
means that sign image
We have been collecting 340 area detection shown in Fig
combined with red (R),
images characters of sign 5(c). Calculation on the
green (G) and blue (B) as of
(letters and numbers), process of transformation
three primary colours.
consist of 170 sample image from YCbCr colour space to
Luminance and
for training data and 170 RGB colour space shown in
chrominance are not
image for testing data. Equation 2[10].
separated in this colour
Each sign image has a
space. The value of R, G
resolution of 96 dbi (dots
and B are linear
per inch) and 24-bit mode
combination against the
with jpg image type. all
luminance of a pixel of
datasets image are
RGB.Researchers propose
conditioned with the same
using YcBcr method on
background and lighting.
research that can take a
Some of sign image
hand and removes
captured are shown in
unwanted background.
Figure 2.
Steps of pre-processing are
shown in Figure 4.
4
features are extracted using
Scale Invariant Feature
Transform SIFT method
[18]. The basic idea is to
extract the invariant key
point which
represents/identifies hand
from the segmented image.
For this purpose of hand
detection, SIFT features are
first extracted from a set of
reference images and stored
in a database. An image
frame is matched by
individually comparing
each feature from the image
frame to this previous
database and finding
candidate matching features
(a) based on Euclidean distance
(b) (c) of their feature vectors.
Figure 5. Skin Detection D. SIFT Algorithm
Process (a) Original Image
(b) The features are invariant to
image scaling, translation,
R  Y  1,4022  Cr and rotation, and partially
invariant to illumination
G  Y  0,3456  Cb  0,7145  Cr
changes and affine or 3D
B  Y  1,7710  Cb projection[18]. These
(2) features share similar
properties with neurons in
The image has noise after inferior temporal cortex that
the skin detection will be are
used for object
done the process of removal recognition
in primate
of noise using vision. SIFT algorithm
morphological filtering includes five steps:(1)
namely: Morphology Scale-Space Local Extrema
opening (erosion followed Detection (2) Keypoint
by dilation). This process is Localization (3) Orientation
just used to the assignment (4) Keypoint
segmentation phases. Descriptor (5) Feature
vektor matching.
1)
1)
1)
1)
1)
1)
1)
(a)
Space-space Local
(b)
Extrema Detection
Figure 6. Result process This stage of the filtering
attempts to identify those
of mophology (a) Original
locations and scales that
Image (b) After morphology
are identifiable from
process.
different views of the same
object. This can be
C. Hand Posture efficiently achieved using a
Recognition
"scale space" function.
The image has been Further it has been shown
processed segmentation under reasonable
which show a picture of a assumptions it must be
hand object and background based on the Gaussian
has been removed. The next function. The scale space is
process is to change this defined by the function[18]:
image to grayscale. The
converted gray scale image
is normalized. Invariant
element feature vector for
L(x, y,  ) = G(x, y,  )  I(x,each
y) keypoint.
(3) 5) Feature Vektor
2) Keypoint Matching
Localization As each feature point
The detected local extrema corresponds to a 128
are good candidates for element feature vector,
keyponts. However, they searching for matching
need to be exactly localized points from two images
by fitting a 3D quadratic becomes searching for
function to the scale-space matching feature vectors.
local sample point. The Euclidean distance is used
quadratic function is for measuring the similarity
computed using a second degree between two feature
order Taylor expansion vectors, the ratio between
having the origin at the the closest Euclidean
sample point. Then, local distance and that of the
extrema with low contrast second closest are computed
and such that correspond to and if the ratio is lower than
edges are discarded a certain threshold [18].
because they are sensitive When the ratio of two
to noise.
distances is less than a
3) Orientation given threshold, accept the
Assignment match points. The Euclidian
distance of two N dimension
Local extrema detected in
functions f1(x) and f2(x) is
DOG scale-space are called
as follows[20]:
keypoints after the
operations of improving d((fx), g(fx)) = (x1 - g(x1 )) 2 +
positioning accuracy and
n
eliminating low-contrast
points. To determine the
=  f (x 1  g ( x1 )) 2
i 1
keypoint orientation, a
gradient orientation
histogram is computed in In this paper use 0.65 the
the neighborhood of the vectors are considered to
keypoint. The contribution
of each neighboring pixel is be   matched   with   each
weighted by the gradient other.   Example   matching
magnitude and a Gaussian
result   with   Threshold   =
window with a value of δ
0.65 is shown figure 7.
that is 1.5 times the scale of
the keypoint. Peaks in the
histogram correspond to
dominant orientations. A
separate keypoint is created
for the direction
corresponding to the
histogram maximum, and
any other direction within
80% of the maximum value.
All the properties of the
keypoint are measured
Figure 7. Matching
relative to the keypoint results with T=0.65.
orientation, which provides
invariance to rotation[19]. IV. Eksperimental
4) Keypoint Descriptor Result
The descriptor is formed This stage it will be shown
from a vector containing the
the result of experiments to
values of all the orientation the recognition of sign
histogram entries. The best
language using SIFT
results are achieved with a algorithm with a variety of
4×4 array of histograms
characters. Figure 8. shows
with 8 orientation bins in a matching result with the
each histogram[20]. this
value Threshold of 0.65.
paper use a 4×4×8 = 128
How to Format Your Paper for JIAS 5

system makes some obtain   best   result   with

mistakes while recognition. T=0.45   that   produces   the
For the characters such as
“B, H, “1”, “2”, “4”, “5” value   recognition   rate   to
and “10” the images of 79,71%.   The   illustrate   of
hand gesture are good results   sign   language
quality after preprocessing
recognition   number   shown
stage. Thus the system
achieves high success rate at Figure 9.
in recognition for these
characters. Table 2 show
test results matching by
changing the value of the
threshold.

(a) Table 1. The Success Rate

(b)
Figure 8. Matching results
with the value of threshold
0.65
The next stage will be
shown the test results with
340 image dataset that
consists 170 sample images
testing data and 170 sample
images trainning data.
Result of the proposed
system is shown on Table 1.
there are 34 characters
Table 2. Recognition results
(Letter and Number)
of sign language number
Indonesian sign language
with total success rate up to
74,12%. The results show
that the success rates of
characters “M”, “N”, “S”,
“T”, “V”, “3”,”6”, “7”, “8”
and “9" are low because
they are shape-like.Thus the
recognition   rate   with “10”
different   threshold.   We accuracy.
of Recognition
Characte Correct Wrong
r (times) (times)
A 9
B 10
C 8
D 9 (a) Exact matching
E 8
F 5
results
G 9
H 10
I 9
K 9
L 9
M 4
N 4
O 7
P 6
Q 6
R 7
S 4
T 4
U 8
V 4
W 8 (b) False matching
X 8
Y 8 results
1 10 Figure 9. Ilustrate
2 10
3 8 results Matching
4 10 performed by SIFT
5 10
Method.
6 8
7 4
8 4
9 5 V.Conclusions
10 10
This paper proposes SIFT
Average method for Indonesian sign
language recognition.
Change the value of the
threshold provide influence
Value on accuracy of recognition.
of Threshold
0.45 0.55 The best performance on the
Recognition 79,71 76,47 recognition of the character
Rate (%) sign language (letters and
numbers) on the threshold
value of 0,65 namely: “B,
Table   2   shows   the H, “1”, “2”, “4”, “5” and
with its 100%
Change the
6

threshold value that gives pattern “Pattern Matching 110, 2004.

best accuracy is at the classification using for Automatic Sign [19] W. Wang, J. Hong,
threshold value of 0, 45 spectral estimation Language and Y. Tang, “Image
which produces 79,71% and neural Translation System matching for
accuracy. network,” SICE, using LabVIEW geomorphic
2007 Annu. Conf., Keywords :,” pp. measurement based
pp. 1108–1111, 660–665, 2007. on SIFT and
Acknowledgment 2007. [13] S. K. Yewale and P. RANSAC
The Authors would like to [6] S. S. Rautaray and K. Bharne, “Hand Methods,” Proc. -
thank to The State A. Agrawal, “Vision gesture recognition Int. Conf. Comput.
based hand gesture using different Sci. Softw. Eng.
Polytechnic of Malang for
recognition for algorithms based on CSSE 2008, vol. 2,
supporting to attend this
human computer artificial neural no. 50505051, pp.
research. interaction : a network,” 2011 Int. 317–320, 2008.
survey,” 2012. Conf. Emerg. [20] J. Zhu, S. Wang,
References [7] N. A. and S. S. Trends Networks and F. Meng, “SIFT
Pragati Garg, Comput. Commun., method for paper
[1] I. Mahfudi, M. “Vision Based Hand no. 1998, pp. 287– detection system,”
Sarosa, R. A. Gesture 292, 2011. 2011 Int. Conf.
Asmara, and M. A. Recognition,” Int. J. [14] G. Plouffe and A.- Multimed. Technol.
Gustalika, Comput. Inf. Eng., M. Cretu, “Static ICMT 2011, pp.
“Indonesian Sign vol. 3, no. 1, pp. and Dynamic Hand 711–714, 2011.
Language Number 186–191, 2009. Gesture Recognition
Recognition using [8] J. Singha, A. Roy, in Depth Data
SIFT Algorithm
Author Biographies
and R. H. Laskar, Using Dynamic
Indonesian Sign “Dynamic hand Time Warping,” Isa Mahfudi,
Language Number gesture recognition IEEE Trans. received the
Recognition using Bachelor of
using vision-based Instrum. Meas., vol. Applied Science,
SIFT Algorithm,” approach for 65, no. 2, pp. 305– in electrical
vol. 336, 2018. human–computer 316, 2016. engineering from
[2] R. M. Prakash, State Polytechnic
interaction,” Neural [15] W. Yang, J. Tao, C. of Malang,
“Gesture Comput. Appl., vol. Xi, and Z. Ye, “Sign Indonesia in
Recognition and 29, no. 4, pp. 1129– Language 2015. He is
finger tip detection 1141, 2018. Recognition System currently a
for Human magister student
[9] J. Pansare and M. Based on Weighted at electrical
Computer Ingle, “Vision- Hidden Markov engineering
Interaction,” in Based Approach for Model,” 2015. departement in
International American Sign [16] I. Nurhaida, A. State Polytechnic
Conference on of Malang,
Language Noviyanto, R. Indonesia. Her
Innovations in Recognition Using Manurung, and A. research interests
Information, Edge Orientation M. Arymurthy, include antenna,
Embedded and computer vision,
Histogram,” 2016 “Automatic image processing,
Communication Int. Conf. Image, Indonesian’s Batik Microcontroller
Systems (ICIIECS), Vis. Comput., pp. Pattern Recognition and signal
2017. 86–90, 2016. Using SIFT processing.
[3] M. E. Al-ahdal, N. [10] A. S. Konwar, B. S. Approach,” Moechammad
Tahir, and U. T. Borah, and C. T. Procedia Comput. Sarosa, received
Mara, “Review in Tuithung, “An Sci., vol. 59, no. the diploma of
Sign Language engineering
American Sign Iccsci, pp. 567–576, technology from
Recognition Language detection 2015. Université de
Systems,” IEEE system using HSV [17] R. Azhar, D. Nancy I, France
Symp. Comput. color model and Tuwohingide, D. in 1989. His
Informatics, pp. 52– master and
edge detection,” Kamudi, doctoral degrees
57, 2012. 2014 Int. Conf. Sarimuddin, and N. in computer
[4] V. A. Farooq Commun. Signal Suciati, “Batik technology is
Husain, Shivani Process., pp. 743– Image Classification received from
Gandhi, Tanisha Bandung Institute
747, 2014. Using SIFT Feature of Technology,
Nijhawan and S. P. [11] A. Izzah and N. Extraction, Bag of Indonesia, in
Sehba Khatun, Suciati, Features and 2002 and 2007,
“Gesture respectively. He
“Translation of Sign Support Vector is currently
Recognition System Language Using Machine,” Procedia lecturer in
Using Matlab: A Generic Fourier Comput. Sci., vol. Department of
Literature Review,” Descriptor and 72, pp. 24–30, Electrical
vol. 2, no. 11, pp. Engineering of
Nearest Neighbour,” 2015. both Brawijaya
425–432, 2012. Int. J. Cybern. [18] D. G. Lowe, University and
[5] K. K. Jung, J. W. Informatics, vol. 3, “Distinctive Image State Polytechnic
Kim, H. K. Lee, S. of Malang. His
no. 1, pp. 31–41, Features from,” Int. current research
B. Chung, and K. 2014. J. Comput. Vis., vol. interests lie in
H. Eom, “EMG [12] A. Domingo, 60, no. 2, pp. 91– information
How to Format Your Paper for JIAS 7

communication
technology,
decision support
system, artificial
intelligence, and
mobile
computing.

Rosa Andrie
Asmara,
received the B.E.
degrees, in
electronics
engineering from
Brawijaya
University,
Indonesia in
2004 and M.S.
degree in
Multimedia
engineering from
Institute of
Technology
Sepuluh
Nopember,
Surabaya,
Indonesia in
2009, and Dr.
Eng degrees from
Saga University,
Japan in 2013.
He is a lecturer in
the Informatics
Management
field, Department
of Information
Technology at
State
Polytechnics of
Malang,
Indonesia. His
research interests
include signal
processing, image
processing,
parallel
processing,
pattern
recognition, and
computer vision.

Muh Bambang
Purwanto,
received her
Bachelor of
Engineering, in
electronics
engineering from
University
Merdeka Malang,
Indonesia in
2015. Currently
he has been
studying in
Electrical
Engineering, at
the third semester
of Master Degree
in State
Polytechnic of
Malang,
Indonesia. Her
research interests
include Control,
computer vision
and image
procesing.