i

Smart Ad-an Emotion Recognition Based Feedback System

on Public Ad.

A PROJECT REPORT

Submitted by

MONIL SHAH (15CP051)

ADITYA UPADHYAY (15CP065)

JAY VALA (15CP064)

In fulfillment for the award of the degree

Of

BACHELOR OF TECHNOLOGY

in

COMPUTER ENGINEERING

Birla Vishvakarma Mahavidyalaya,

Engineering College, Vallabh Vidyanagar, Anand
[An Autonomous Institution]
Gujarat Technological University, Ahmedabad
 ii

Birla Vishvakarma Mahavidyalaya

Computer Engineering Department
2018

CERTIFICATE

Date :

This is to certify that the project entitled “Smart-AD Emotion Recognition

Based Feedback System on Public Ad.” has been carried out by MONIL

SHAH (15CP051), ADITYA UPADHYAY (15CP065) and JAY VALA

(15CP064) under my guidance in fulfillment of the degree of Bachelor of

Technology in Computer Engineering (7th Semester) of Gujarat Technological

University, Ahmedabad during the academic year 2017-18.

Guides : Prof. Pranay Patel

Mr. Bhadresh Patel

Head Of Department
 iii

ACKNOWLEDGEMENT

I have taken efforts in this project. However, it would not have been possible without the kind

support and help of many individuals and organizations.I would like to extend my sincere thanks

to all of them.

I would like to thank the Department of Computer Engineering,BVM Engineering

College, vallabh vidhyanagar.And i would like to express my gratitude towards our internal

guide Prof. Pranay Patel and Mr. Bhadresh Patel for their constant guidance and supervision as

well as for providing necessary information regarding project.

I would like to thank my industrial guide Mr. Jigar Mori sir of Techno Samarthyam company for

their kind cooperation and giving me such attention and time.

It is noteworthy that this project has given us a fantastic experience which will prove beneficial

in future when we will actually work for IT firms. The overall exposure to different technologies

was really worth experiencing and the project was indeed a good tunneling path which had

darkness of difficulties in its middle but also a glowing light of success at its end.

Aditya Upadhyay(15CP065)

Jay Vala(15CP064)

Monil Shah(15CP051)
 iv

ABSTRACT

Due to trend of the technology of smart phones, online services, online application,
emotion recognition is becoming a necessity part for providing emotional care
about to people. Primarily emotional care can enhance users’ experience to
improve the quality of life. The conventional method of emotion recognition may
not feasible to the need of mobile users for their value-added emergent services.
Moreover, because of the dynamicity and heterogeneity of mobile applications and
services, emotion recognition that is one of the challenge for collecting
information, analyze, and process emotional communications in real time and
highly accurate manner with a minimal computation time.

Due to the smart feature of mobile and easy availability mobile application is
aware of emotion and user acceptability become easy. We use real time and highly
accurate system that is detect facial emotion. Due to limited processing power of
mobile device, the algorithm in the emotion recognition system should be
implemented using less computation. High performance for mobile applications we
proposed emotion detection. In the given system, using camera of a smart phone or
webcam facial video is captured. Video is separated into frames, and a face
detection module is applied to extract the face regions in the frames.. Local
histograms and binary patterns are calculated for every block, and finally it is
concatenated over all the blocks. Results of the experiment show that the given
system achieves high recognition accuracy in a reasonable time.

KEYWORDS : Image processing, Google cloud Platform, Convolution

Neural Network(CNN),Emotion Recognition.
 v

LIST OF FIGURES

Figure No Description Page No

2.1 Basic Block diagram 15

3.1 The Prototyping Model 19

3.2 Activity Diagram 20
3.3.1 Data Flow Diagram - Level 0 21
3.3.2 Data Flow Diagram - Level 1 21
3.3.3 Data Flow Diagram - Level 1.1 22
3.3.4 Data Flow Diagram - Level 2 22
3.4 Use Case Diagram 23
3.5.1 Sequence Diagram for Admin 24
3.5.2 Sequence Diagram for Company 24

4.1 Input the video using WebCam 25

4.2 Separating frames from Video 26

5.1 Types of Expression 28

5.2 Kaggle Dataset 28

6.1 Training the CNN model 30

7.1 Viewer's Image 31

7.2 Output-1 32
7.3 Output-2 32
 vi

LIST OF SYMBOLS, ABBREVATIONS &

NOMENCLATURE

CNN :- Convolutional Neural Network

ANN:- Artificial Neural Network

GCP:- Google Cloud Platform

DIP:- Digital Image Processing

 vii

TABLE OF CONTENTS

Acknowledgement i

Abstract ii

List of Figures iii

List of Tables v

List of Abbreviations vi

Table of Contents vii

Chapter : 1 Introduction to Project

1.1 History
1.2 Motivation
1.3 Problem Domain
1.4 Project's Objectives
1.5 Report Structure

Chapter : 2 Literature Review

2.1 Overview
2.2 Surveys and Sytems
2.3 Obtained insights about Image Pre-processing

Chapter : 3 System Analysis and Design

3.1 Implementation Requirements
3.2 Utilized Software Development Model
3.3 System Design and Modelling Diagram
3.3.1 Use case Diagram
 viii

3.3.2 Activity Diagram

3.3.3 Sequence Diagram
3.3.4 Data Flow Diagram
3.3.5 Class Diagram

Chapter : 4 Image Pre-processing

4.1 Overview
4.2 Required Characteristics of Image

Chapter : 5 Preparing Dataset

5.1 Different Dataset
5.2 Cohn-Kanade Dataset
5.3 Kaggle Dataset

Chapter : 6 Implementing neural network on GCP

6.1 Why GCP used?
6.2 Why CNN?
6.3 Training Process

Chapter : 7 Result Analysis and Conclusion

7.1 Test Model
7.2 Conclusion

References
 1

CHAPTER 1
INTRODUCTION
__________________________________________________________

1.1. History

Once upon a time, people believed that the mind of human was given by gods or God.
Emotions, in contrast, were said to live within body of living species, like an inner beast that
needed to be controlled by rational thought. Charles Darwin replaced God with natural
selection, and shortly after that, psychology was born. golden age of emotion research began,
as physiologists and neurologists searched for the physical basis of emotions. They
discovered that, emotions live in parts of the brain that control the body.the mythical "inner
beast" made real. Psychology fell prey to a scourge known as behaviorism, the study of pure
behavior, in which intangibles like thoughts and feelings were deemed immeasurable and
therefore irrelevant to science. Nothing worthwhile was published on emotions for half a
century till 1960. After that cognitive revolution arrived, in the preceding time, rescuing
psychology from the darkness, and the science of emotion experienced a renaissance. once
Emotions were discovered and for all to have distinct and universal facial expressions,
bodily patterns, and brain circuitry, and we all lived happily ever after.
 2

View of emotion is simple and intuitive, and it’s going to become big business now. Many
Companies including Google, Facebook, Apple, and Microsoft are collectively
investing billions of dollars to develop softwares to read the signs of people’s emotions. The
U.S. Transportation Security Administration training airport security agents to detect signs of
deception in airline passengers. The Dalai Lama has weighed in as well, commissioning a
$750,0000 web-oriented "Atlas Of Emotion" to illustrate the nature of the "five universal
emotions": anger, disgust, fear, enjoyment, and sadness. Which can be used in varios areas.

1.2. Motivation

Because of the ongoing growth along with the extensive use of smart phones, services and
applications, recognition of emotion is becoming an essential part of providing emotional
care to people. emotional care can be greatly enhances users experience to improve the
quality of life. The conventional method of emotion recognition may not be cater to the
need of mobile application users for their value-added emergent services. Moreover, because
of the dynamicity of mobile applications and services, it is a challenge to provide an emotion
recognition system that can collect, analyze, and process emotional communications in real
time and highly accurate manner with a minimal computation time.Facial emotions are
important factors in human communication that help us understand the intentions of others.
In general, people infer the emotional states of other people such as joy, sadness, and anger,
using facial expressions and spoken tone. According to different surveys, verbal components
convey one-third of human communication, and nonverbal components convey two-thirds.
Among several nonverbal components, by carrying emotional meaning, facial expressions
are one of the most used information channels in interpersonal communication. that is why it
is natural that research of facial emotion has been gaining lot of attention over the past
decades with applications not only in the perceptual and cognitive sciences, but also in
affective computing and computer animations and emotions of the people can be classified as
below table.
 3

1.3. Problem Domain

We have seen many advertises on social media platform and we reacted on them based upon
whether it liked or not. Till now the sellers can not get direct feedback from users. So they
can't get actual idea that their products are either effective or not among user. So currently we
haven't that kind of system from which sellers or compaigners can directly get the analysis of
their particular advertisement.

So here we have tried to provide a way that takes a emotional expression of the viewer of the
ad, which can be in web application, mobile application and can be in social media platform
like youtube. By using this project campaigners or marketers can get the emotional
expression as a feedback through web cam and then analyze them which leads them towards
the efficient marketing of the product.

1.4. Objectives

Objective of our project is to analyse the data which is taken from the facial emotions of the
people who viewing the advertise. So that advertiser can determine how people are reacting
to their marketing campaigns on advertise on web or mobile application and social media
platforms like Youtube, Facebook, Twitter, and Instagram etc.

This gives them the feedback about kind of content is either working and enables them to
fine tune their marketing strategy accordingly. Companies can leverage such AI tools and
classify all the customers into categories related to the emotion they express when they
interact with the brand. This can be used to drive targeted engagement.
 4

1.5 Report Structure

The concept of this project has been divided into 8 different chapters where each chapter
explains about a single module. The succeeding chapter reviews a brief history of previous
works done over the same project and explanation of implemented approach in brief. Chapter 3
focuses on literature reviews and obtained insights from the referred literatures for each module.
Chapter 4 provides an overview of project’s design including implementation methodology,
project’s requirements and utilized software development model. Moreover, the same chapter
also incorporates project visualization through number of software development diagrams.

Chapter 5 discusses image processing module which is the first one for this implemented project.
The chapter encompasses all the required steps and actions performed over the input image in
order to make it compatible for feeding it to recognition algorithm.

Chapter 6 describes the CNN model which is the central module for this project. It includes
training process, feature extraction and appropriate reasons to prefer CNN over other machine
learning algorithms for text recognition. Chapter 7 describes the conclusion of project with the
result analysis.
 5

CHAPTER 2
Literature Survey
__________________________________________________________

2.1. Overview

The very first step in order to avoid previously encountered issues was literature survey. Many
literatures about emotion recognition were referred and several insights about them were
obtained. Various papers have been written regarding emotion recognition,in which various
approaches are used to recognize emotions. With the use of some research papers we have been
able to do our project.

Fig 2.1 Block Diagram

 6

2.2. Surveys and Systems

Early reviews covering feedback system through emotion recognition can be found in certain
literatures. More recent reviews can be also found by song yang and Lin chen,P.W.C prasad,
A.K.singh.in their research papers.

Facial Recognition is done based on Emotion Recognition Model in Virtual Learning

Environment

This study advise a learning emotion recognition model, which consists of three
stages: 1.Feature extraction, 2.subset feature and 3.emotion classifier. A Haar Cascades method
used for detect the input image, a face, as the extraction of eyes and mouth, and then through
the Sobel edge detection to obtain the characteristic value. Through Neural Network
classifier training, six typs of different emotional categories are obtained. Experiments using
JAFF database show that the this advise method has high classification performance. The goal of
emotion recognition is to gather data and analyse expression of subjects, to make appropriate
responses possible. Such data may obtained from different features such as face voice, body
movements, and other biological physical signals. However, learners’ emotions are expressed
through facial expressions which can be divided into six kinds of categories: sadness, happiness,
surprise, fear, anger and disgust. Given Fig lists the emotion recognition process. The facial
recognition process consists of three main stages: acquisition, feature extraction, and
emotion classification. This paper is organised along the three stages.

TARGETING ADVERTISEMENTS BASED ON EMOTION:

A computer system, an applied method in computer, and computer readable media configured to
target advertisements based on emotional states. Advertisers specify desired emotional states of
users they will target with advertisements. Advertisers also give emotional tags having the
specified emotional State of users that should see the advertisements joined to the emotional
tags. Online activities for users ar obtained and processed to assign emotional states to the users.
An advertisement engine selects advertisements that are showing emotion compatible based on
the assigned emotional States and therefore the emotional states provided by the advertisers.
 7

2.3 Obtaining Insights about Image processing :

DIP focuses on developing a computer system that is ready to perform processing on a picture.
The input of that system could be a digital image and also the system process that image using
efficient algorithms, and provides an image as an output.

The entire literature survey made one fact clear that a raw image cannot be used as an input to
any recognition because first of all video will be captured by camera and then after it will be
divided into some frames of images and some image processing techniques should be required
for further processing. Then face of the viewer will be detected. The goal of image processing is
to boost or compress image/video information.
 8

CHAPTER 3
System Analysis and Design
__________________________________________________________

3.1. Implementation Requirements

All the modules of this project require certain software support and preparation of dataset
required some basic hardware too. Following are the required software tools which were used for
project’s implementation:-

1] Python 3.6.4:- This is an open source programming language with certain powerful libraries
for designing neural networks and performing image processing.

2] Image processing libraries: - OpenCV for performing required actions for image pre-
processing.

3] Neural Network’s libraries: - Keras and Tensorflow for implementing and training the CNN
model.

4]Web camara access:-To capture the video of the viewer.

5]Google cloud Platform:- We have Used GCP for implementing neural network on cloud
platform..
 9

3.2. Utilized Software Development Model

A variety of software development models were available to be followed of which the

Prototyping model was selected for implementation. The basic idea here is that instead of
freezing the requirements before a design or coding can proceed, a throw away prototype is built
to understand the requirements. This prototype is based on current requirements. By using this
prototype, the client can get the actual feel of the system, since the interactions with the
prototype can enable the client to better understand the requirements of the desired system. The
prototype are not usually complete systems and many of the details are not built in the
prototype. The goal is to provide a system with overall functionality.

Figure 3.1 : The Prototyping Model

 10

3.3 System Design and Modeling Diagram

For better visualization of implemented methodology, software design diagrams were

constructed which provided a glimpse of flow of control as well as data in our project. Following
diagrams proved useful for better understanding of this project

start

Capturing Video

Separating into frame

Train the model

Test captured Image

Analyze the expression of viewer

Generate statistic of viewer

end

Fig.3.2 Activity Diagram

 11

Fig. 3.3.1 DFD Level 0

Fig.3.3.2 DFD level 1

 12

Fig.3.3.3 DFD LEVEL 1.1

Fig.
3.3.4 DFD Level 2
 13

Fig.3.4 Use case Diagram

 14

Fig.3.5.1 Sequence Diagram For Admin

Fig.3.5.2 Sequence Diagram For Company

 15

CHAPTER 4
Image Pre-processing
__________________________________________________________

4.1. Overview

Mathematically, an image is a matrix of pixel values which is stored in memory of a computer. A

raw image when directly passed for recognition can lead to inefficient results which was
concluded from the literature survey. We are capturing video for some constant time and
seperating it into some frames and those frames are stored in the folder of machine.This process
is shown by followed image.

Video Capturing

Fig.4.1 capturing the video using webcam

 16

Separating Into Frames

Fig.4.2 separating frames from the video

4.2 Required Characteristics Of image

When we are Storing the RGB images in the folders,the requirernents of the memory will be
high.Because RGB image require more space than the simple gray-scale image.so that if we try
to convert the RGB image into the Gray-scale image then the use of memory can be reduced.and
Face detection algorithm can be easily applied.also there are many advantages of grayscale
image over RGB image.So before storing the frames in the folder we must convert each frame to
the Gray-scale.
 17

CHAPTER 5
Dataset Preparations
__________________________________________________________

5.1 Different Datasets

Following are the various datasets that can be used in Emotion Recognition.

1] Cohn-Kanade Facial Expression Database.

The Cohn-Kanade Facial Expression Database can be used in automatic facial image analysis
and synthesis, as well as for developing project. There are two sets of photos. The first set
contains 486 images from 97 posers, following Ekman's FACS model (given an emotional
label). The second set contains both posed and non-posed (spontaneous) expressions and
additional types of metadata.

2] Kaggle Dataset

This dataset is exactly what we expected. the images will be converted into csv file which
contained the 48 X 48 pixels. There are approximately 28900 sample images. The final test
images are where . The main task is to categorize each face based on the emotion shown in the
facial expression in to one of seven categories (0 : Angry, 1 : Disgust, 2 :Fear, 3 : Happy, 4 : Sad,
5 : Surprise, 6 : Neutral). train.csv contains two columns, "emotions" and "pixels". The
"emotions" column contains a numeric code ranging from 0 to 6. inclusive, for the emotion that
is present in the image. The "pixels" column contains a string surrounded in quotes for each
image. The contents of this string a space separated pixel values in row major order. test.csv
contains only the "pixels" column and your task is to predict the emotion column.
 18

Fig.5.1Seven types of Expression

Fig.5.2 Kaggle Dataset

 19

Chapter-6
Implementation of CNN on GCP
__________________________________________________________________

6.1. Why GCP is used for project?

There are other options to choose for machine learning project. For example, local machines,
google cloud platform, microsoft azure. When the machine is not able to train the sample data
which refers as the training data, we have to choose the different perspectives. We have chosen
the GCP which is very simple to setup the environment for neural network. There are several
advantages of Google cloud platform over local machines. Major uses of GCP are machine
learning, Storage, big data, hosting and computing. It provides the number of virtual cpu to
implement the AI, machine learning projects. Also we have to pay some negligible amount of
cost. Here we have to create the instances for predefined CPUs and memory. It costs 0.1 $ per hr.
Also the cost depends on the number of CPUs you are using. The cost will increase as the
number of virtual CPUs are increased.

6.2 Why CNN?

We have implemented the convolution neural network using Keras library. For avoiding the
complexity of feature extraction and low-level data operation problem during most of facial
expression recognition, we used a method of CNN (Convolution Neural Network) for facial
expression recognition. Firstly, the image is normalized and the extracted implicit feature using
the trained convolution model. Then used maximum pooling for reducing dimension of extracted
implicit features.

It is good to use deep learning for image classification to build a convolution neural network
(CNN). In python Keras library is good to use it becomes pretty simple to build a CNN. We can
classify image pixel renderized and mobalized into 48*48. Seven layers have been used. A
neural network is required to process nodes arranging in a layer-by-layer discipline generally
trained end-to-end
 20

6.3. Training Process:

The CNN model using keras has been trained according to the prepared dataset. The expressions
were classified into 7 different categories.

1. Sad

2. Anger

3. surprise

4. Disgust

5. Neutral

6. Happy

7. Fear.

Figure 6.1: Training the CNN model

Finally, we have made the model which has been trained. Afterwards once the model is trained I
saved it as cnn_model.h5 . So when we want to analysis the expression just load the saved cnn
model for predicting the emotion.
 21

Chapter-7
Results and Conclusions
______________________________________________

7.1. Testing the model

On the testing part, system has to analysis of expression of the person who has watched the ad on
the platform.

Fig.7.1 Viewers Image

 22

Fig.7.2 Output -1

Here in the previous figure(7.1) A boy seems like very tensed. And below the figure, the analysis of the
his emotions are classified. The prediction of the emotion has been analyzed.

Fig.7.3 Output-2

So the output we will get about the expressions included during the video. This pie graph
says that the ad which viewed by the person affected in negative direction. It means sad
and fear expressions came as the analysis of that particular advertise.
 23

7.2. Conclusions

Generally speaking, In earlier time companies who advertised for their product on websites, they
did not get feedback from the customers. This system helps to companies for getting the
feedback from the user. When one watches the ad, this system will capture the video of the
viewer containing the expressions for ad. This video is differentiated into the frames and
convolution neural network -CNN using keras will be implemented onto the frames for analysis
the expression of viewer. The statistics will be displayed as a result. This result will be seen by
the company so that they can get the feedback of their ad whether it affects negative or positive
to the public. Here one disadvantage is that it tracks the privacy of user. If someone does not
agree with the policy, then we will not get his/her expression while watching the ad.

References:

1]. Automatic Emotion Recognition Using Facial Expression: A Researched by Monika Dubey,
Prof. Lokesh Singh from International Research Journal of Engineering and Technology
(IRJET).

2]. Sentdex Youtube: https://www.youtube.com/channel/UCfzlCWGWYyIQ0aLC5w48gBQ

3].Generate statistics: https://medium.com/python-pandemonium/data-visualization-in-python-

pie-charts-in-matplotlib-c71fb0fe5c56

4]. https://medium.com/data-science-group-iitr/artificial-neural-network-for-text-
classification-b7aa5994d985

5]. https://blog.algorithmia.com/introduction-to-emotion-recognition/