Proposal and Implementation of a novel scheme for
Image and Emotion Recognition using Hadoop
Parag Saini1, Tanupriya Choudhury 2, Praveen Kumar3 , Seema Rawat4
Amity University Uttar Pradesh, Noida1,2,3,4
paragsaini22@gmail.com1, tchoudhury@amity.edu2, pkumar3@amity.edu3, Srawat1@amity.edu4
Abstract—The digital media especially social media is very
popular in recent time. Due to which we have more number of
videos than before. These videos are generally not tagged and
classified. This paper aims for parallel video processing in
Hadoop for fast processing and automatic tag emotions. This
system recognizes face and tag emotions automatically on
hadoop clusters for fast and efficient performance. This will
make human work easier in classification of videos and
processing on parallel system make it quick.
Keywords : Automatic emotion tagging, parallel video
processing, face recognition
I. INTRODUCTION
A growth of videos on internet has been increased in recent
few years because of social media and sites like youtube,
dailymotion. According to the recent studies [1] more than
60% people watch videos online. These videos are generally
not classified and not tagged properly. Due to which video
searching become such a difficult task.
Tagging is a process by which we will assign metadata to the
videos which helps to organize and manage the resources.
Video searching depend upon these tags which contains meta
data of the videos like most viewed video. Some more
important information about the video is not provided.
In 2004, [1] data is processed by the MapReduce clusters.
From then MapReduce is used to large amount of data. Video
processing can be accelerated by MapReduce. This technique
is used for cheap, scalable and capable for video processing. It
uses apache Hadoop and some open source projects.
A large amount of video can be handled and time taken to
process the data can reduced by utilizing the clusters. The
system automatic tagging process can be done parallel with
other system.
[7]. Video is basically a structured media by nature and our
first step is to make it into temporal unit by segmented it.
Histogram is a technique which is used by separating each
frame into blocks smaller in size and “Histogram difference”
of frames is taken in succession.
There are many approaches existed for key frame extraction
which are depends upon detected shots. One of the technique
is ‘Shot Boundary Detection’.
According to “Shot Boundary Detection”, First read the video
as an input and shot boundary method is used to process it.
978-1-5386-0569-1$31.00 2017
c IEEE
Then frames are divided into sub frames by using the same
method. Then difference between each sub frame gives the
block difference
Calculate the block difference from required formula and
calculate the mean deviation and standard deviation. This will
give you the threshold. If threshold which is calculated is
smaller than that of block difference of frame, then frame will
become key frame.
Face detection and recognition is done together. First video
stream is taken as an input. It is converted into image sequence
of frames. Then important key frames are extracted from the
image sequence by techniques like PCA or EGM. Then with
the help of some techniques we detect and track the face. The
next step is Image processing and face alignment which
consists of three parts. First one is Histogram equalization
after that resize image and last one is image raster scan. The
extracted features are used to create face print. Then feature is
matching with the done from the trained set.
Video tagging is defined as collection of contents in video like
a scene, or a shot or information about it and description is
much more relevant than the whole annotated video.
Through this paper an efficient method is proposed to detect
the faces and recognize faces in the video and automatically
tag the emotions which is done on parallel systems with the
help of Hadoop clusters. The key objective of this paper is to
recognize faces from video, extract key features, and use
technique for image recognition so that character can be
identified using a trained data and important information is
labelled and classified under different classes. A template
consists of important information such as name about
character and pre-configured nodal points. The nodal points
are calculated in mathematics so that we can have a face print
which refers to the database. The face is recognized and
automatic emotions are tagged with them. This process is very
slow so to make is more efficient Hadoop clusters are used so
that parallel processing is done.
Fig.1 Face detection
1358
The techniques which are generally used for feature extraction
are described below: -
EGM: -[3] In this method we gather the data store it and
process the features of the face in image graph. The jets are
the local image descriptors is transformed by wavelet
transform. These are results of the Gabor wavelet which have
a 2-D wave field. The jets form graph which will represent the
face of the character. If the graph is matches with the
geometry, then it considered has same faces.
PCA: - [3] It is one of the widely-used algorithm in facial
recognition. Eigen faces ae formed by extracting the principal
components in the face images. Whenever there is a new
image we have to determine is it a face image or not. If it is a
face image, then the person is recognized by the weight
pattern. But it is very expensive to operate and couldn’t work
well with complex images. Still this is used most widely
because it is robust and unsupervised.
[A] Video Annotation
To improve searching, classifications and indexing of the
videos we have to assign some labels, tags to the content of
video. Automated video tagging can be done by two ways.
The first one is “open-set tagging” in which[9] extraction is
required and tags for that extracted video content can be
chosen from information which is associated with content like
phrases, group of words or sentences. The second one is
“closed set tagging”. It provides classification of tags in
predefined classes like cricket, news report. Hierarchical
method is used to store video annotation in the form of data.
According to semantic levels we group the data of video
annotation. the video shot corresponding to another feature
Fig.2 Working of MPEG-7
2017 International Conference On Smart Technology for Smart Nation
vector , and in-turn, it maps to other shot label. A number of
labels maps to video event, which means one layer of video
annotation are calculated by previous layers of video
annotation data.
Multimedia content descriptor like MPEG-7 is used to store
video annotation in hierarchical manner. For fast retrieval of
videos the description is associate with
the content. XML is used so that meta data can be stored and
it can associate with time-code and events can be
synchronized.
According to the standards of MPEG [] a Description
containing of a descriptive scheme (structure) and the set of
Descriptive values (instantiations) that describe the Annotated
Data. The Descriptor value can be defined as an instantiation
of a Descriptor for a given data set. DDL[10] defines the
structural relation among descriptors which is based on XML
language.
[B] Key Feature Extraction Algorithm
The algorithm mainly focuses on techniques which are
different and video sequence fundamental dynamics. An
algorithm is explained for key frame extraction.
Step 1: Calculation of all the dissimilarity between general
frames and reference.
Step 2: Locate the maximum difference in the shots.
Step 3: Depending upon the relationships we can determine
the shots which is existed between maximum and mean
deviation.
Step 4: Calculate the position of the key frame.
Fig.3 Working of Key frame extraction
1359
[c] Apache Hadoop System Architecture
It is basically divided into two sub-systems: one of them is
task scheduling and other one is distributed system storage.
The tasking scheduling system consists of Job Tracker which
is master of schedule MapReduce task and Task tracers
between the clusters. Name node is master node of storage
system and Data nodes. With Datanode and Task tracers an
special node called slave node is deployed. To improve the
performance MapReduce task are sent to slave. We can
manage storage, do failure recovery and schedule task by this
framework.
The key feature of MapReduce is encapsulating data in key
values pair which helps the Mapper and Reducer can process
in concurrent manner via parallel processing. .
HDFS is mounted to local file system by Fuse-DFS and the
video data present on HDFS made available to JavaCV. Video
analysis ability of JavaCV is inherited from OpenCV and
FFMPEG, which helps to make the libraries available from
video IO to MapReduce.

Fig.5 Internal working of Apache Hadoop

Fig.4 Apache Hadoop Architecture
The Procedure of the process is given below:-
[5] Fuse-DFS is sub part of Apache Hadoop system. It acts as
Video data is read by RecordReader through interface given
an interface so that the gap between local file system and
by javaCV, data is encapsulated into key-value and then it is
HFDS can be filled and files designed for local file system can
submitted to inputted format.
be benefitted.
Multiple key value pair can be accepted by one Input format
There are two most commonly used libraries in computer
which is provided by RecordReader. All key value are paired
vision i.e. OpenCV and FFMPEG. Video processing will not
by InputFormat and it is then submitted to Mappers.
be completed without the help of these libraries. With the help
These key value pairs is then grouped according to algorithm
of Fuse-DFS we can use them in Hadoop. Initially they are
requirements and then these are dispatched to the Reducers.
made to run in C/C++ only and Hadoop is made to run in java.
Key value is processed by the Reducers and final results are
Then one more project is made which is hosted by Google
submitted to the OutputFormat.
Code named as JavaCV, helps and give a better solution to
These results are then written to the HDFS by the
port all the video processing libraries including OpenCV and
RecordWriter which is employed by outputFormat.
FFMPEG to java on multiple operating system like Linux,
Windows, Android, with the support from hardware
Hadoop processes videos concurrently with JavaCV. There are
acceleration. The system is explained below.
multiple Datanode in which processing in done. Each process
HDFS help us to store distributed services for video data.
is implemented in different Datanode due to which parallel
Fuse-DFS helps to mount distributed file into local file.
processing is done and it will help our system to work
JavaCV helps in port two video processing libraries OpenCV
concurrent work. This will improve the efficiency and make
and FFMPEG to java.
the system faster.
For video data to be concurrent MapReduce programming
model is used.

1360 2017 International Conference On Smart Technology for Smart Nation

II. PROPOSED WORK

After the analysis of the existing system and new techniques
which can be used to make a system which can recognize the
face as well as emotion which gives one extra feature for
tagging which is helpful in classification in videos.
The System is worked on Hadoop due to which the efficiency
of the system is increased. This paper proposed an
automatically tagged image and emotion recognition system.
Step 3: In this step, all the background is suppressed using
image thresholding. Frames passed from previous steps and
cross-checked with technique used in step 4. So after this
process we are only left with rectangular windows with only
the facial pixels (faces present in that frame).

Step 1: The video frame rate, first of all is trimmed to 10 FPS

to 15 FPS from the original frame rate of 25-29. This is done,
to achieve efficiency and less processing of video per second
by catching the key frame easily.

Fig. 8. Detected faces & Suppressing Non facial pixels

Step 4: To recognize the face, first of all a “GFK(General Face

Knowledge)”, which is basically a massive database of
metadata and the information of the images in the database

Fig. 6. Video Stream to Image Sequence

Step 2: In this step, the key frame is extracted and detected.

This facilitated by subtracting of frames that are consecutive.
This technique detects potential frames where a movement
due to living/non-living entity is detected. Since this
technique may not be able to detect very miniscule
movements let’s say: flinching of eyes. It has to filter down
from the next process.
Fig. 9. Face Recognition using Elastic Graph Matching
Step 5: Using Bezier Curve Algorithm, we will detect the
important features of the face which recognize emotion like
facial expressions, lip flinching, rolling of eyes etc. The
obtained data from these steps is compared with existing data
linked with an emotion/expression. The data for the emotion is
converted into XML format according to the emotion shown in
that particular frame

Fig. 7. Metadata with probe image Fig. 10. Emotion Detection through Bezier curve

2017 International Conference On Smart Technology for Smart Nation 1361

Step 6: The processing of image recognition and emotion
recognition will run parallel and the facial emotion as well as the
image are automatically tagged in this step.
Fig. 11.F low diagram of the working system
This diagram shows how these above steps will work. Firstly
we reduce frame per second and extract the key frame by
differentiate the images. Then we detect faces by supressing all
non-face pixels. Parallel we extract facial expressions for image
recognition. We compare the image and from our database and in
parallel compare emotions from our database. Then we
automatically tag both face and emotion.
III. RESULT
We find that when we have done both image and emotion
recognition on one system it takes a lot of time while if we do on
Hadoop clusters the process speed up. It takes less time while we
do on Hadoop clusters.
Fig. 12. Graph showing speed variation
1362 2017 International Conference On Smart Technology for Smart Nation
As ween in the above graph as the processes done on single
system the speed is very less but when we do the same
processes on Hadoop clusters the speed increase.
.IV. CONCLUSION
The project work is used to classify video and gives an extra
dimension which is emotion which is helpful to classify videos
in more appropriate manner. This image and emotion
recognition will be effective in terms of performance because
we used Hadoop for parallel processing. Both image and
emotion recognition is processed parallel which make the
application fast and most efficient. After both the process is
done on the parallel nodes, and both the results are then
combined which will help in automatic tagging. This
automatic tagging is helpful in search the videos which will
help to save both time and energy. Also the cost is not much
high due to which it is easily affordable. It will reduce the
human effort to do task because it will automatically tag the
videos.
V. Future Work
This work can be used in robotics with the help of IOT. With
this we can make a robot we can automatically recognise the
face of the person which is used to identify the owner of the
robot and also identify the mood of the owner by using
emotion recognition.
As our work is on running on Hadoop nodes the robot will be
fast to identify his owner as well as his mood of his owner.
REFERENCES
[1]. Gill, Phillipa, Martin Arlitt, Zongpeng Li, and Anirban Mahanti.
"YouTube traffic characterization: a view from the edge." In
Proceedings of the 7th ACM SIGCOMM conference on Internet
measurement, pp. 15-28. ACM, 2007.
[2] Bloehdorn, Stephan, Kosmas Petridis, Carsten Saathoff, Nikos
Simou, Vassilis Tzouvaras, Yannis Avrithis, Siegfried
Handschuh, Yiannis Kompatsiaris,SteffenStaab, and Michael G.
Strintzis. "Semantic Annotation of images and videos for
multimedia analysis." In the Semantic web: research and
applications, pp. 592-607. Springer Berlin
Heidelberg, 2005.
[3] M. Turk and A. Pentland, "Eigenfaces for recognition," Journal
of Cognitive Neuroscience, vol. 3, no. 1, pp. 71-86, January
1991.
[4] Zhu, Xiangxin, and Deva Ramanan. "Face detection, pose
estimation, And landmark localization in the wild."In Computer
Vision and Pattern Recognition (CVPR), 2012 IEEE Conference
on, pp. 2879 2886. IEEE, 2012.
[5] C.-H. Chen. Mohohan: An on-line video transcoding service
via apache Hadoop. [Online]. Available:
http://www.gwms.com.tw/TRENDHadoopinTaiwan2012/1002d
ownload/C3.pdf
[6] F. Yang and Q.-W. Shen, “Distributed video transcoding on
Hadoop,” Computer Systems & Applications, vol. 11, p. 020,
2011.
[7] Anastasios D. Doulamis, Nikolaos D. Doulamis and Stefanos D.
Kollias, National Technical University of Athens, Department
of Electrical and Computer Engineering,
[8] Sanchita , kalpana Jaiswal , Praveen Kumar , Seema Rawat
“Prefetching web pages for improving user access latency using
integrated Web Usage Mining “in the proceeding OF International
Conference on Communication Control and Intelligent System
(CCIS-2015) organised by GLA University Uttar Pradesh , INDIA
(Published in IEEE Explorer) , November 07-08, 2015. PP 401 –
405.
[9] Praveen Kumar , Dr. Vijay S. Rathore “Improvising and Optimizing
resource utiliza tion in Big Data Processing “in the proceeding of
2017 International Conference On Smart Technology for Smart Nation
5th International Con-ference on Soft Computing for Problem
Solving (SocProS 2015) organised by IIT Roorkee, INDIA
(Published in Springer) , Dec 18-20, 2015. PP 586 – 589.
[10] Seema Rawat ,Praveen Kumar, Geetika, “Implementation of the
principle of jamming forHulk Gripper remotely controlled by
Raspberry Pi “in the pro-ceeding of 5th International
Conference on Soft Computing for Problem Solving (SocProS
2015) organised by IIT Roorkee, INDIA, Dec 18-20, 2015. PP
199-208.
1363