23rd Telecommunications forum TELFOR 2015
Recognizing emotions from videos by studying
facial expressions, body postures and hand gestures
Mihai Gavrilescu
1
Abstract A system for recognizing emotions from videos
by studying facial expressions, hand gestures and body
postures is presented. A stochastic context-free grammar
(SCFG) containing 8 combinations of hand gestures and body
postures for each emotion is used and we show that
increasing the number of combinations in SCFG improves
the systems generalization for new hand gesture and body
posture combinations. We show that hand gestures and body
postures contribute to improving the emotion recognition
rate with up to 5% for Anger, Sadness and Fear compared to
the standard facial emotion recognition system, while for
Happiness, Surprise and Disgust no significant improvement
was noticed.
Keywords multimedia, video signal processing, affective
computing, face recognition, gesture recognition.
I. INTRODUCTION
Recognizing the affective states of individuals is an
important feature for humans in interpersonal interactions.
On one hand, affective states modulate the way we
communicate and react; on the other hand, it offers clues
on the emotional state of others and optimizes our
interaction with them. Hence, for a computer to be able to
understand and communicate with individuals, it has to be
able to recognize and even simulate emotions. Various
researches aimed building such system, but most of them
treat only one of the three components: body postures,
hand gestures or facial expressions.
In terms of body postures, Shibata and Kijima [1] made
use of pressure sensors on chairs and body accelerometers
for verifying the link between body gestures and postures
and facial expression factors such as arousal and
pleasantness. Their research showed that body gestures
introduce another factor called dominance expressed by
arms and legs, while still retaining the other two factors.
Similarly, Radeta and Maiocchi [2] used capture sensors to
analyze low level features of the skeletal postural joints of
human body in relation to the 7 basic emotions,
uncovering patterns between body postures and emotions
that can be exploited via machine learning techniques.
Body postures were combined in [3] with facial
expressions, researchers defining two type of action units:
facial action units and body action units. The results
showed that facial expressions alongside with body
features offer better accuracy than standard facial emotion
recognition.
Similar studies were conducted for hand gestures and
movements. Kipp and Martin [4] showed that gesture
Mihai Gavrilescu is with University Politehnica Bucharest, Department
of Telecommunications, Romania, 1-3 Iuliu Maniu Blvd, 061071,
Bucharest 6, Romania (e-mail: mike.gavrilescu@gmail.com)
978-1-5090-0055-5/15/$31.00 2015 IEEE
Serbia, Belgrade, November 24-26, 2015.
features like motion direction, palm orientation, hand
shape and handedness are associated with pleasure and
arousal factors specific to facial expressions. Moreover,
Glowinski et. Al [5] used visual tracking techniques for
hands from frontal and lateral views to show that the
quality of gestures improves the recognition of emotions.
With these in mind, our paper aims building a system
for emotion recognition by studying facial expressions,
alongside with body postures and hand gestures and show
the improvements such a system brings when compared to
standard facial emotion recognition (FER) systems.
II. THEORETICAL MODEL
Our research is based on the fact that there is a close
link between the emotional dimensions found in facial
expressions and those determined from body postures [1]
and hand gestures [4]. Moreover, the research done in [6]
showed facial expressions, body movements and postures
as well as hand gestures are linked at subconscious level in
the human mind when expressing emotions, hence
studying them together is important if an emotion
recognition system aims human-like performances.
For facial expression recognition, we made use of the
Facial Action Coding System (FACS) developed by
Eckman and Freisen [7] based on the idea that each
emotion determines a facial muscular activity on the face
that can be analyzed in order to determine the true
emotional state of the subject, as these muscular activities
cannot be mimicked. FACS defines a set of 46 action units
(AUs) that have the characteristic of being additive (one
action unit triggers another action unit and form together
an Action Unit Cluster (AUC)) or non-additive (the AU is
independent of other AUs).
In what it concerns hand gestures our paper is based on
the research done in [8], dividing gestures in three large
categories: deictic gestures (hand or finger pointing to an
object), mimetic gestures (accepting / declining actions),
and iconic gestures (defining attributes like shape or size).
We can also view gestures as a set of permutations
generated by the actions of the hand and arm and two
types of systems can be defined: single gesture-based
recognition systems (SHGRS easy to implement, lack
generalization) and multiple hand gesture-based
recognition systems (MHGRS harder to implement, can
generalize gestures). In this paper we build a SHGRS that
will also include the information related to body postures.
In our specific case, body postures will refer to the
subjects body being inclined forward or backwards, to the
left or to the right, or rotated towards left or right. These
body postures along with hand gestures are combined and
a context-free grammar (SCFG) [9] is defined in order to
describe hand gesture and body posture combinations with
the advantage that it can be extended to other gestures and
postures.
720
 III. PROPOSED ARCHITECTURE
A. Facial Expression Recognition (FER) system
As previously mentioned, FACS contains 44 AUs out
of which, in this paper, we use only the first 27 AUs as
they are more specific to the face-related expressions. We
implement a similar module as in our previous work [10],
dividing the face in 5 components (each containing a set of
AUs that can only be additive inside the component)
which are: Brow (Haar-Cascade classifier [11]), Eye
(Haar-Cascade classifier [11]), Cheek (Viola/Jones Face
detector classifier for cheek AU classification [12]), Lips
(lip feature point tracking method [13]) and Wrinkles
(shortest distance classifier [14]). Fig. 1 shows this
module.
Fig. 1. Facial Emotion Recognition (FER) System
For each frame, the head orientation is determined with a
k-nearest neighbors algorithm [15] and based on this the
face is detected and segmented using the Viola-Jones face
detection algorithm [16]. The detected face is then
segmented in the 5 components using the same Viola-
Jones algorithm. For each segment the classifiers
previously mentioned will classify in an inter-subject
methodology the corresponding AUs for each segment and
fetch them to a FER - Neural Network (NN) which takes
the final decision regarding the recognized emotions. The
FERNN receives 27 features (for each 27 AUs classified)
as input nodes, having only one hidden layer with 55
neurons and 6 output nodes representing the 6 basic
emotions.
B. Hand Gesture and Body Posture recognition system
As previously mentioned, a SHGRS system was
implemented that will also include information related to
body postures. The hand gesture recognition system has
three basic components: hand segmentation, hand tracking
and gesture recognition. Also, the hand has two main
components: static hand pose and hand posture (static
hand independent of the movement of the body), and hand
gesture (dynamic movement of the hand). Moreover, hand
gesture has also two components: global hand motion and
the local finger motion. Global hand motion refers to
changes in the orientation or position of the hand and it is
analyzed in relation with the global body motion as they
are mutually related. The global body motion analyses the
changes in the upper body postures, referring to the
721
inclination of the body to left or right, forward or
backwards, as well as the rotation of the body towards left
or right. Local finger motion refers to the movement of
fingers considering the position of the hand unchanged.
Fig. 2 shows the overall architecture of the Gesture
Emotion Recognition (GER) system. The first layer
contains an AdaBoost block trained to determine if a
frame contains body or hand features and flag it as
negative if not. Frames that pass the filter are sent in
parallel to Haar-cascade classifiers [11] that return body,
hand, and finger features. These are fetched to three neural
networks (NN): Local Finger Motion Neural Network
(LFM-NN), Global Body Motion Neural Network
(GBM-NN), Global Hand Motion Neural Network
(GHM-NN). The output of each NN is sent to the Gesture
Detection Neural Network (GD-NN) providing the
recognized emotion.
Fig. 2. Gesture Emotion Recognition (GER) System
As this is a pattern recognition task and a bottom-up
architecture the neural networks considered are feed-
forward multi-perceptrons. All three neural networks have
16 input nodes corresponding to 16 body-hand-finger
features pre-determined for our training set, as well as one
hidden layer (for GBM-NN 34 hidden nodes, for GHM-
NN - 32 hidden nodes, for LFM-NN - 20 hidden nodes).
As previously mentioned, we make use of a stochastic
context-free grammar (SCGF) containing a set of hand
gesture and body posture combinations that were pre-
determined from the training database as Haar-like feature
combinations related to specific emotions. The GBM-NN,
GHM-NN and LFM-NN are trained using
backpropagation and only when all are trained and the
error rate is low enough, the last GD-NN is considered for
training using also backpropagation. The GD-NN has 18
input nodes (the outputs of three neural networks), one
hidden layer with 32 neurons and 6 output nodes
corresponding to the 6 basic emotions.
The module functions in two stages:
- training stage: the Haar-like features are processed
and saved in the SCFG grammar as well as used to
train the LMF-NN, GBM-NN and GHM-NN and
GD-NN through backpropagation;
- testing stage: Haar-like features determined from video
testing sample are processed using the SCFG
 grammar by GBM-NN, GHM-NN and LMF-NN and
the hand gesture and body posture combination and
its corresponding emotion are determined.
C. Overall Architecture
Fig. 3 presents the overall architecture of the system.
We can see that the video input signal with the recorded
subject is fetched in parallel to the two modules on
different threads. C++ and OpenCV library were used for
developing the classifier algorithms and neural networks.
One last Haar-cascade classifier is implemented (Emotion
Recognition Classifier - ERC) that analyzes the output of
the two modules providing the final recognized emotion.
Fig. 3. Proposed model Overview
IV. EXPERIMENTAL RESULTS
In order to test the system we built our own database
containing recording of the facial expressions, hand
gestures and body postures of 64 subjects when watching
6 emotion inducing videos for each of the 6 basic emotions
(fear, anger, surprise, happiness, sadness, disgust). The 64
subjects participating in the experiment are 32 males and
32 females, with ages between 18 and 35. The protocol
used was sitting, with hands in view, only the upper body
being recorded. An example is shown in Fig 4.
Fig. 4. Emotional reactions upper body recording
(a) Anger; (b) Sadness; (c) Happiness; (d) Disgust
Videos used for inducing the 6 basic emotions are from
the discrete LIRIS-ACCEDE database [17]. Because our
system needs longer videos with enough frames for
training, we combined similar 8-12 seconds videos
pertaining to the same emotion from LIRIS-ACCEDE
database in 1 minute inducing videos for each emotion.
All the neural networks and classifiers were trained with
frames from these recordings in an inter-subject
methodology. At first the FER classifiers were trained on
Cohn-Kanade [18] and tested on MMI database [19] in
order to achieve over 90% accuracy in cross-database test
for each of the 27 AUs. With the classifiers trained, we
proceeded with training the FER-NN on recordings from
our own database so that it outputs the emotion that was
supposed to be induced by the video watched by the
subject. FER-NN was trained on video samples from 63
out of the 64 subjects in an inter-subject methodology with
722
a leave-one-out approach. The next step was to determine
the Haar-like features for hand gestures and body postures
from the recorded frames for the same 63 subjects and use
them to construct the SCFG. Then the GHN-NN, GBN-
NN, LFM-NN and GD-NN were trained in a similar
manner as the FER-NN was trained, so that for the
recorded reaction of each subject the NNs will output the
emotion induced by that video. After all these modules
were trained on 63 subjects, the entire system was tested
on the remaining subject in an inter-subject methodology
with leave-one-out approach. 64 tests were conducted for
studying the inter-subject variability and recognition rates.
First we tested the GER system alone to verify its
accuracy. We first included only 2 combinations of hand
gestures and body postures for each emotion in the SCFG,
and then increased this to 4 and 6 to verify its degree of
generalization. Results are detailed in Table 1. We can see
that for controlled scenario (trained and tested on the same
combinations) as the number of combinations in SCFG
increase, the accuracy increases from 65.1% (SCFG with 2
combinations) to 75% (SCFG with 8 combinations). Also,
in uncontrolled scenario (trained on a set of combinations,
tested on the other set) we observed a similar increase
from 54.5% (SCFG with 2 combinations / tested on
remaining 6) to 60.4% (trained on 6 combinations / tested
on remaining 2). This indicates that as the number of hand
gesture and body posture combinations in SCFG increases,
the systems degree of generalization increases.
TABLE 1: EMOTION RECOGNITION RATES ON DIFFERENT NUMBER OF
COMBINATIONS IN SCFG GRAMMAR.
Number of Emotion Recognition Emotion Recognition
combinations in Rate (%) controlled Rate (%) random
SCFG / scenario scenario
emotion
2 65.1% 54.5%
4 67.5% 56.3%
6 70% 60.4%
8 75% -
With the SCFG containing all the 8 combinations, we
performed a new inter-subject leave-one-out validation test
analyzing which of the 6 emotions is recognized by GER
with highest accuracy. Table 2 shows that the highest
sensitivity is recorded for Fear (86%), followed by Anger
and Happiness (77%), although the specificity in these
cases is not extremely good.
TABLE 2: TRUE POSITIVE AND TRUE NEGATIVE RATES FOR THE GER
SYSTEM.
Emotion True positive rate (%) True negative rate (%)
Anger 77% 64%
Happiness 77% 65%
Sadness 73% 68%
Surprise 73% 70%
Disgust 64% 72%
Fear 86% 62%
The lowest sensitivity is obtained for Disgust showing
that hand gesture and body posture combinations do not
emulate successfully this type of emotion. The inter-
subject coefficient of variance (calculated as the ratio
between the standard deviation and the mean value) is also
high, reaching a value of 14.5%.
Next we tested FER-GER system and compared it with
the standard FER system using the same inter-subject
methodology and leave-one-out approach for cross
validation. Results are detailed in Table 3 and show that
when using facial expressions together with body posture
and hand gestures it resulted in an increase in recognition
rates for Anger (up to 6%), Sadness (4%), and Fear
(4.5%). This shows the GER system introduces new
emotional features compared to the facial expressions
ones. For Happiness and Surprise differences were not
impressive, mainly because the combinations considered
in the SCFG do not bring anything new compared to facial
expressions. For Disgust, the recognition rate was not
improved at all, expected considering the GER low
recognition rates for Disgust. Moreover the inter-subject
coefficient of variance decreased to 6.2%.
TABLE 3: RECOGNITION RATES FOR STANDARD FER AND FOR FER-GER
SYSTEMS.
Emotion Recognition rate (%) Recognition rate (%)
standard FER FER + GER
Anger 81.6% 87.8%
Happiness 88.5% 89.2%
Sadness 80.5% 84.4%
Surprise 88.4% 89%
Disgust 76.5% 76.7%
Fear 85.3% 89.7%
Compared with the state-of-the-art, the system showed
higher recognition accuracies than those in current
researches. Comparable results are detailed in Table 4.
TABLE 4: COMPARISON WITH THE STATE-OF-THE-ART.
Research Recognition rates [%]
Koelstra and Pantic - 2010 [20] 70.25%
Valstar and Pantic - 2012 [12] 72%
Gavrilescu - 2014 [10] 83.5%
This Work 86.4%
The system was implemented on a 3.4 GHz i7 processor
testbed with 8 GB of RAM memory. The average time to
compute an emotion from 1 minute video was 20 seconds.
V. CONCLUSION
An emotion recognition system based on facial
expressions, hand gestures and body postures was
presented. We showed that the GER system can determine
Anger, Happiness and Fear with accuracies of 75% and
that there increasing the number of hand gesture and body
posture combinations in the SCFG increases the systems
degree of generalization for new combinations. Tested
together with FER system, we showed that for Anger,
Sadness and Fear, the GER system improves the standard
FER with up to 5% suggesting hand gestures and body
postures contain emotional information that cannot be
acquired from facial expressions. For Happiness and
Surprise the recognition rate increase is not high, hence
hand gestures and body postures for these emotions do not
add new features compared to facial ones. As a drawback,
for Disgust the recognition rates were low, hence we need
to consider other approach for this emotion. Moreover, no
other way of measuring emotion was included in our
system and this might have decreased the classification
accuracy, hence considering an additional measure of
emotion alongside with enriching the SCFG could increase
the accuracy and this is the subject of our future research.
REFERENCES
[1] Shibata, T., Kijima, Y., Emotion recognition modeling of siting
postures by using pressure sensors and accelerometers, 21st
International Conference on Pattern Recognition, pp. 1124-1127,
Tsukuba, 2012.
723
[2] Radeta, M., Maiocchi, M., Towards automatic and unobtrusive
recognition of primary-process emotions in body postures, 2013
Humaine Association Conference on Affective Computing and
Intelligent Interaction (ACII), pp. 695-700, Geneva, 2013.
[3] Gunes, H., Piccardi, M., Fusing face and body gesture for machine
recognition of emotions, IEEE International Workshop in Robot
and Human Interactive Communication, pp. 306-311, Nashville,
August 2005.
[4] Kipp, M., Martin, J.-C., Gesture and emotion: can basic gestural
form features discriminate emotions?, 3rd International Conference
on Affective Computing and Intelligent Interaction, pp. 1-8,
Amsterdam, September 2009.
[5] Glowinski, D., Dael, N., Camurri, A., Volpe, G., Mortillaro, M.,
Scherer, K., Toward a Minimal Representation of Affective
Gestures, IEEE Transactions on Affective Computing, 2 (2), pp.
106-118, April 2011.
[6] Meeren, H. K. M., Heijnsbergen, C. C. R. J., Gelder, B., Rapid
perceptual integration of facial expression and emotional body
language, Procedings of the National Academy of Sciences of
USA, 102(45), pp.1651816523, November 2005.
[7] Ekman, P., Friesen, W. V., Facial Action Coding System:
Investigators Guide, Consulting Psychologists Press, CA, 1978.
[8] Shanis, J. M., Hedge, A., Comparison of mouse, touchpad and
multitouch input technologies, Proceedings of the Human Factors
and Ergonomics Society, pp. 746-750, Denver, October 2003.
[9] Chen, Q., Georganas, N. D., Petriu, E. M., "Hand gesture
recognition using Haar-like features and stochastic context-free
grammar", IEEE Transactions on Instrumentation and
Measurement, 55(8), pp. 1562-1571, April 2008.
[10] Gavrilescu, M., Proposed architecture of a fully Integrated
Modular Neural Network-based Facial Emotion Recognition system
based on Facial Action Coding system (FACS), 10th International
Conference on Communication, pp. 1-6, Bucharest, May 2014.
[11] Wilson, P. I. ,Fernandez, J., Facial feature detection using Haar
classifiers, Journal of Computing Sciences in Colleges, 21(4), pp.
127-133, April 2006.
[12] Valstar, M., Pantic, M., Fully Automatic Facial Action Unit
Detection and Temporal Phases of Factial Actions, IEEE
Transactions on Systems, Man, and Cybernetics, 42(1), pp. 28-43,
February 2012.
[13] Lien, J. J.-J., Kanade, T., Cohn, J. F., Li, C. C., Detection,
Tracking, and Classification of Action Units in Facial Expression,
Journal of Robotics and Autonomous Systems, 31(3), pp. 131-146,
1999.
[14] Pantic, M., Tomc, M., Rothkrantz, L. J. M., A hybrid approach to
mouth features detection, Proceedings of IEEE International
Conference on Systems, Man, Cybernetics, pp. 1188 - 1193,
Tucson, Arizona, October 2001.
[15] Feijun Jiang, Ekenel, H. K., Shi, B. E., Efficient and robust
integration of face detection and head pose estimation, 21st
International Conference on Pattern Recognition (ICPR), pp.1578-
1581, Tsukuba, 11-15 Nov. 2012.
[16] Viola, P., Jones, M., Robust real-time face detection, IEEE
International Conference on Computer Vision, vol. 2, pp. 747,
Vancouver, 2001.
[17] Baveye, Y., Dellandrea, E., Chamaret, C., Chen, L., LIRIS-
ACCEDE: A Video Database for Affective Content Analysis,
IEEE Transactions on Affective Computing, 6(1), pp. 43-55, 2015.
[18] Kanade, T., Cohn, J. F., Tian, Y., Comprehensive database for
facial expression analysis, 4th IEEE International Conference on
Automatic Face and Gesture Recognition, pp. 46-53, Grenoble,
2000.
[19] Pantic, M., Valstar, M. F., Rademaker, R., Maat, L., Web-based
database for facial expression analysis, IEEE International
Conference on Multimedia and Expo, pp. 317 - 321, Amsterdam,
July 2005.
[20] Koelstra, S., Pantic, M., Patras, I., A dynamic texture based
approach to recognition of facial actions and their temporal
models, IEEE Transactions on Pattern Analysis and Machine
Intelligence, 32(11), pp. 1940-1954, November 2010.