1

Emotion Sensing Smartphone

12.6.2015
To: Dr. Regina Dugan
By: Michael Rosenfield

Abstract
Application developers and content distributors, such as Youtube, rely on historical data
and like buttons to predict what a user wants to watch next. Unfortunately, because
recommendation algorithms rely on this proxy data, results are not satisfactory. The
paper proposes the integration of emotion sensing, such as a GSR sensors, into a
smartphone and then securely sending the data to the foreground application. The apps
recommendation algorithm will respond with much more relevant recommendations. To
limit the scope of the project, the plan is to develop a smartphone case which houses the
sensors and communicates with a smartphone through bluetooth. Once prototypes are
developed, experiments will be conducted to determine the benefit of the emotional
data being used within recommendation algorithms.

Table of Contents
Section
1.
2.
3.
4.
5.
6.
7.
8.

Introduction
Project Description......
Literature Review.........
Plan of Work........
Work Schedule........
Researcher Qualifications.........
Budget..
References...............

Page #
2
2
3
5
6
6
7
8

Introduction:
There are currently two billion smartphone users worldwide and the average user
spends 9 hours on the device every week. A significant portion of smartphone usage is
media consumption, including watching videos, browsing the web, and exploring social
media. One of the largest challenges for application developers is content
recommendations and discovery. The reason this is difficult is because the applications
must rely on two main types of data. The first type of data is historical data, such as what
videos a specific user has chosen to watch in the past. The second type of data is surveys
that require participation from the user, such as clickings thumbs-up or thumbs-down on
a video. Consequently, the weak proxy information that is used in current
recommendation algorithms, renders the results futile. Because of this, the addition of
emotional state data, a better proxy, would improve content recommendations and
discovery.

Project Description
I propose to research an implementation of emotion
sensing technologies, specifically Galvanic Skin Response
(GSR), into a smartphone to increase the accuracy of
content recommendations. A GSR sensor is capable of
reading a persons skin conductance, which is directly
correlative to his or her emotional activity. Through
bluetooth, the raw data will be sent to the smartphone,
converted into usable emotional data which is then
passed to the application on the phone. An application
developer can securely use these measurement, in
real-time, to recommend content. To limit the scope of
the project, I plan to develop a smartphone case which
houses the sensors and communicates with a
smartphone through bluetooth. (Figure 1)

Literature Review
Current Methods of Recommendations:
Whether it is for a news app or a video sharing app, most modern recommendation
systems are very similar. Due to its complex nature, a prime example of such a system is
YouTube. To recommend new content YouTube relies on user activity: the videos the
user watched, favorited, and liked. The videos that meet this criterion are inputted into a
recommendation algorithm that creates a node for each video and builds branches of
other videos that relate to the seed videos. [1] Once the hundreds of recommended
videos are created, the system tries to rank the videos using properties that are specific
to the user such as how many times the user watched the video and how much of the
video they watched. [2] These factors are effective, but it is common for a user to have
watched a video completely, yet completely dislike it.

Galvanic Skin Response Sensor:

Because view count and time watched are
extremely indirect proxies, I propose the addition
of a better proxy of user enjoyment: skin
conductance. A galvanic skin response sensor is
made up of two capacitive touch points that are
connected by a voltage divider. When a person
places two of their fingers on the touch points,
their body acts like a large resistor as the
electricity is passing from finger to finger. This
resistance is a factor of how sweaty the users
fingers are, as the more sweat on ones finger, the
more conductive the finger is. Using the voltage
divider, we can calculate the exact resistance of the body and therefore measure how
much sweat the user is producing. [3] Studies have shown that that skin conductance is
directly correlated to emotional activity. [4] Due to the small circuitry required to build a
GSR sensor, it will not be difficult to fit it in either a smartphone body or a smartphone
case. [3]
There have been studies conducted in the past that use GSR in consumer products to
measure emotional activity. (Figure 2) For example, researchers at the Blekinge Institute
of Technology performed a study where they implemented emotion sensors near the
triggers on a video game controller. The study used galvanic skin response sensors as

well as electroencephalography, electrocardiography, electromyography and eye tracking

equipment to measure how involved a player is when playing in a videogame. The
researchers noted that GSR sensors are capable of providing enough information to
detect a users level of arousal, which can be correlated to a users level of immersion. [5]

Using the Data:

A key aspect of the project is using the emotional data to benefit both the user and
application developer. A study run by researchers at the University of Maryland showed
emotional data, collected using GSR, could be used as a means to recommend videos to
unique users. The study implemented a system that converted the emotional data into
an emotion status ranging from disturbed to amused. The study concluded that the
optimal system uses a decision tree machine-learning algorithm, which is an algorithm
that uses past emotional data to create a mood classification. [6] By implementing this
same algorithm, I could convert raw GSR data into simple mood classifications that is
sent to the foreground application. This will ameliorate the security connotations that
arise with reading emotional data, as the application is limited to reading emotional
state. Another study showed that, by correlating emotional state to specific music scores,
one could recommend music suitable to a users current emotional state. This was done
by grouping music chords with identified emotions. [7] The study shows yet another
example of improving a users experience using emotional data.

Significance of the Project:

Though this research involves a single emotion sensor (GSR), research has shown that by
combining sensor data from eye trackers, EKG, EEG, accelerometers, and others, one can
create a phenomenal emotional model. [8] This emotional model will be tremendously
useful if read from modern smartphones. When using a phone, one is often consuming
content. The majority of that content is driven by recommendations derived from bad
proxy data such as view count, ones friends activity, or how long one spend watching a
video. By integrating emotional data into the recommendation algorithm, suggestions
will be significantly more tailored to individuals. Users will be presented with content that
they want to consume. With emotional data, application developers will also benefit: click
through rate will surge and users will spend more time on ones application. Both of
these attributes will grow revenue and increase customer satisfaction. [9]

Plan of Work
In order to complete my research, I have split the work into three phases. Phase I is
building the prototype, Phase II is developing the algorithm, and Phase III is conducting
the study.

Phase I: Build The Prototype

During Phase I, I will purchase the necessary circuitry to build three galvanic skin sensors
that communicate with a smartphone and are all housed within a 3d printed case. This
will require basic circuit components, as well as low cost Arduino boards that have
integrated bluetooth. Following the circuit design of a GSR sensor, I will build three
simple GSR kits that I can place in different enclosures. Next, I will use a 3d printer to
design three different smartphone cases with unique conductive sensor placement. I will
lastly integrate the GSR kits into the distinct cases.

Phase II: Develop the Algorithm

During Phase II, I will build on current recommendation algorithm approaches, but with
the addition of emotional data. I will follow the work of YouTube and Amazon, using a
deep neural network. In addition, I will develop a channel to allow the smartphone case
to communicate with the foreground application on Android. The best way to enumerate
the emotional data will be discovered during this research.

Phase III: Conduct Experiment

By Phase III, I will have three working prototypes, each with different GSR sensor
placements. During Phase III, I will conduct user studies to determine whether access to
emotional data increases the quality of recommendations and which sensor placement is
most effective. The studies will be conducted by finding 100 participants and letting them
use my demo video application. After a some time, I will show recommendation results
returned by conventional algorithms and recommendations returned using emotional
data. Once I have collected feedback from all participants, I will compile a final report
containing the results.

Work Schedule
Overall, the whole project is estimated to take about ten weeks with the deliverables
being three working prototypes, a working emotion integrated recommendation
algorithm, and a final report. At the end of each week, I plan to devote a few hours of my
time to write status reports that will be send to all mentors of the project, including
yourself. This timeline takes into account the roadblocks that are likely going to arise
throughout the process.

Task

10

Source and Purchase Circuitry

Build Prototype

Develop Recommendation Algorithm

Recruit Study Participants

Collect, Analyze, and Report Experimental

Data

Researcher Qualifications
As a third year computer science student at Case Western Reserve University, I have
gained the necessary academic knowledge to accomplish this research. Over the past
two years, my relevant coursework has included Introduction to Programming in Java,
Introduction to Material Science, Introduction to Data Structures, Software
Craftsmanship, Computer Networks, Programming Language Concepts, Logic Design and
Computer Organization and Introduction to Database Systems. Each of these courses
have given me insight and breadth in the world of computer science and engineering.
Combining knowledge from each of these classes will allow me to personally create the
prototype hardware, as well as develop an exemplar recommendation algorithm.
In addition to the academic background, I have also gained tremendous experience in
the technology sphere with my past internships at Google and Motorola. Most relevant to
this research, I interned on the Android team at Google where I worked with prototype

smartphones. I also internet within the ATAP team at Motorola, where I gained
experience developing biometric technologies in the mobile space.
If any problems arise or gaps of knowledge appear, I have the tools and drive to tackle
anything necessary.

Budget
Due to the simplicity of the electronics and the software focus, the project will cost no
more than $5000. The following table describes the few items that will need to be
purchased to complete the research.

Android Smartphone

$500

3D Printer

$1000 - $3000

Galvanic Skin Response Electronics [x3]

$50

Arduino [x3]

$100

TOTAL

$1650 - $4650

Conclusion
Dr. Dugan, well aware of your busy schedule, I will be sure to keep your responsibility on
the project to a minimum. As stated above, I will send you weekly status reports and any
feedback or ideas will be welcome. Though, if I run into any difficulty, I might need your
expertise in product development and research to overcome them.
If all goes well, your involvement will most likely grow as we think past the prototype and
towards integrating the research into a consumer product.

References
[1] S. Baluja et al., "Video suggestion and discovery for youtube," Proceeding of the 17th
international conference on World Wide Web - WWW 08, 2008.
[2] J. Davidson et al., "The YouTube video recommendation system," Proceedings of the
fourth ACM conference on Recommender systems - RecSys 10, 2010.
[3] M. V. Villarejo, B. G. Zapirain, and A. M. Zorrilla, "A stress sensor based on galvanic
skin response (GSR) controlled by ZigBee," Sensors, vol. 12, no. 12, pp. 60756101, May
2012..
[4] S. Khalfa, P. Isabelle, B. Jean-Pierre, and R. Manon, "Event-related skin conductance
responses to musical emotions in humans," Neuroscience Letters, vol. 328, no. 2, pp.
145149, Aug. 2002.
[5] L. Nacke and C. A. Lindley, "Flow and immersion in first-person shooters," Proceedings
of the 2008 Conference on Future Play Research, Play, Share - Future Play 08, 2008.
[6] X. Y. Chen and Z. Segall, "XV-Pod: An emotion aware, Affective mobile video player,"
2009 WRI World Congress on Computer Science and Information Engineering, 2009.
[7] M.-K. Shan, F.-F. Kuo, M.-F. Chiang, and S.-Y. Lee, "Emotion-based music
recommendation by affinity discovery from film music," Expert Systems with
Applications, vol. 36, no. 4, pp. 76667674, May 2009.
[8] J. Gonzalez-Sanchez, M. E. Chavez-Echeagaray, R. Atkinson, and W. Burleson, "ABE: An
agent-based software architecture for a Multimodal emotion recognition framework,"
2011 Ninth Working IEEE/IFIP Conference on Software Architecture, Jun. 2011.
[9] G. Linden, B. Smith, and J. York, "Amazon.com recommendations: Item-to-item
collaborative filtering," IEEE Internet Computing, vol. 7, no. 1, pp. 7680, Jan. 2003.

Figure 2:
http://www.element14.com/community/servlet/JiveServlet/showImage/38-4762-46098/M
akeArticleFig01_02.png