University of Mumbai

Multi-Modal Emotion Detection

Submitted in fulfillment of requirements
For the degree of

Bachelors in Technology
by
Anushka P Chhadva
Roll No: 1511069
Mahir R Dave
Roll No: 1511071
Priyal K Thakkar
Roll No: 1511115
Guide
Dr. J.H Nirmal
Dr. Prasanna Shete

Department of Computer Engineering

K. J. Somaiya College of Engineering, Mumbai-77
(Autonomous College Affiliated to University of Mumbai)
Batch 2015 -2019

Department of Computer Engineering Semester VIII 2015-19 Batch

 K. J. Somaiya College of Engineering, Mumbai-77
(Autonomous College Affiliated to University of Mumbai)

Certificate

This is to certify that the dissertation report entitled Multi-Modal Emotion Detection
submitted by Priyal Thakkar, Mahir Dave and Anushka Chhadva in the year 2018-19
under the guidance of Dr. Jagannath Nirmal of Department of Computer Engineering in
partial fulfillment of requirement for the Bachelors in Technology degree in Computer
Engineering of University of Mumbai

_________________ _____________________
Guide Head of the Department

_________________
Principal

Date: 05/16/19
Place: Mumbai-77

Department of Computer Engineering Semester VIII 2015-19 Batch

 K. J. Somaiya College of Engineering, Mumbai-77
(Autonomous College Affiliated to University of Mumbai)

Certificate of Approval of Examiners

We certify that this dissertation report entitled Multi-Modal Emotion Detection is bona
fide record of project work done by Priyal Thakkar, Mahir Dave and Anushka Chhadva.
This project is approved for the award of Bachelors in Technology Degree in Computer
Engineering of University of Mumbai.

___________________________
Internal Examiners

____________________________
External/Internal Examiners

Date: 05/16/19

Place: Mumbai-77

Department of Computer Engineering Semester VIII 2015-19 Batch

 K. J. Somaiya College of Engineering, Mumbai-77
(Autonomous College Affiliated to University of Mumbai)
DECLARATION

We declare that this written report submission represents the work done based on our and
/ or others’ ideas with adequately cited and referenced the original source. We also
declare that we have adhered to all principles of intellectual property, academic honesty
and integrity as we have not misinterpreted or fabricated or falsified any
idea/data/fact/source/original work/ matter in my submission.

We understand that any violation of the above will be cause for disciplinary action by the
college and may evoke the penal action from the sources which have not been properly
cited or from whom proper permission is not sought.

______________________________ ______________________________
Signature of the Student Signature of the Student
_______________________________ _______________________________
Roll No. 1511115 Roll No. 1511071

______________________________
Signature of the Student
_______________________________
Roll No. 1511069

Date: 5/16/19

Place: Mumbai-77

Department of Computer Engineering Semester VIII 2015-19 Batch

 Acknowledgement

First of all we would like to express our deep gratitude towards our advisor and guide
Dr. Jagannath Nirmal who has always been a guiding force behind this project work.
His highly influential personality has provided us constant encouragement to tackle any
difficult task assigned. We are indebted to him for his invaluable advice and for
propelling us further in every aspect of our academic life. His depths of knowledge,
crystal clear concepts have made our academic journey a cake walk. We consider it our
good fortune to have got an opportunity to work with such a wonderful personality. Next,
we want to express our respect to Dr. Prasanna Shete for providing us necessary
information about project work and helping us learn various tough concepts. He has
always been great sources of inspiration to us and we would like to convey our deep
regards to him. We are grateful to all faculty members and staff of the Department of
Electronics as well as Department of Computer Engineering, K. J. Somaiya College of
Engineering for their generous help in various ways for the completion of this thesis. We
would like to thank all our friends and especially our classmates for all the thought
provoking discussions we had, which inspired us to think beyond the obvious. We are
especially indebted to our parents for their love, sacrifice, and support. They are our first
teachers after we came to this world and have always been mile stones to lead us a
disciplined life.

Department of Computer Engineering Semester VIII 2015-19 Batch

Abstract:-

Emotion plays important roles in expressing feeling as it tends to make people acts
differently from their normal behavior. We have already seen systems which recognize
emotion using speech or facial. We are going to design hybrid multimodal emotion
detecting system using two parameter i.e speech signals and facial features. This study
will enable us to get more information regarding nature and function of emotion.
Furthermore, this work will help in developing a generalized voice emotion recognition
system, which will increase the efficiency of human-machine interaction systems. We
have implemented Speech recognition using CNN and feature extraction using MFCC.

Our model is trending today as every company or large scale industries are using robots
which requires such multimodal emotion detecting implementing system to gain
knowledge of environment(which can be user) and react as per situation so our model can
be used in such scenario.

In our model we are going to take user input data i.e speech sample and face images
through webcam and detect emotions like happy, angry, sad, surprise, neutral, fearful
through this sample input which will help us to identify the emotion for many application
like employee recruiting and for psychologist to detect emotion of patient. We have
implemented this system using Python.

Department of Computer Engineering Semester VIII 2015-19 Batch

List of Figures…………………………………………………………………….. i

List of Tables…………...………………………………………………………… iii

Contents 1
Introduction…………….………………………………………..……………………..1

1.1 Problem Definition…………………………….……………………………………2

1.2 Scope………………………………………………………...………………………3

1.3 Software Requirements…………………...………………...……………………….4

1.4 Hardware Requirements…………………………...………...………………………4

2 Literature Survey…..………………………………………………………………...6

2.1 Overview on Emotion Recognition System………………………….……………...7

2.2 Emotion recognition from Assamese speeches using MFCC features and GMM
classifier…………………………………………………………………………….…..10

2.3 Facial Emotion Recognition using Deep Convolutional Networks…………………12

3 Project Design……..……………………………………………………………….…14

3.1 Proposed System Model……………………………………………………………..15

3.2 Software Project Management Plan………………………………………………….16

3.3 Software Requirement Specification Document………………………………….....20

3.3.1. Introduction……...…………………………………………………………..........20

Department of Computer Engineering Semester VIII 2015-19 Batch

3.3.1.1 Purpose……………………………………………………….………..….…...20
3.3.1.2 Product Scope………………………………………………………………...20

3.3.2 Overall Description…………………………………………………………....21

3.3.2.1 Product Perspective………………………………………………………….22

3.3.2.2 Product Functions…………………………………………………………...22

3.3.2.3 User Classes and Characteristics………………………………………........23

3.3.2.4 Operating Environment……………………………………………………..23

3.3.2.5 Design and Implementation Constraints……………………………………23

3.3.2.6 User Documentation………………………………………………………...23

3.3.3. External Interface Requirements

3.3.3.1 User Interfaces………………………………………………………………24

3.3.3.2 Hardware Interfaces…………………………………………………………24

3.3.3.3 Software Interfaces………………………………………………………….24

3.3.3.4 Communications Interfaces……………………………………………...….25

3.3.4. System Features

3.3.4.1 Customer specification………..…………………………………………….26

3.3.4.2 management specification……………………….………………………….26

3.3.5. Other Nonfunctional Requirements

3.3.5.1 Performance Requirements…………………………………………………..27

3.3.5.2 Safety Requirements………………………………………………………....28

Department of Computer Engineering Semester VIII 2015-19 Batch

3.3.5.3 Security Requirements………………………………………………………..28

3.3.5.4 Business Rules………………………………………………………….…….29

Software Design Document……………………………………………………………30

4 Implementation and experimentation of Modules

4.1 Speech Emotion Recognition………………………………………………….…..36

4.1.1 Dataset Collection………………………………………………………….……36

4.1.2. Methodology…………………………………………………….………..…….37

4.1.3 Implementation………………………………………………………………….45

4.2 Facial Emotion Recognition………………………………………………….…...47

4.2.1 Prototype model implementation……………………….…………...…..…….....47

4.2.2 Dataset Collection……………………………..…………...………..…………..48

4.2.3 Data preprocessing.………………………………………………………..…….49

4.3 Integration of UI with our model………………………………..…….………...55

4.4 Testing………………………………………………………………….….……...56

5 Conclusions and scope for further work………………..……………..…..….......59

5.1 Conclusion……………………………………………………………..………….60

Department of Computer Engineering Semester VIII 2015-19 Batch