See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/318404775

Can we Improve the User Experience of Chatbots with Personalisation?

Thesis · July 2017

DOI: 10.13140/RG.2.2.36112.92165

CITATIONS READS

4 3,638

1 author:

Daniëlle Duijst
University of Amsterdam
2 PUBLICATIONS   4 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

User Experience of Chatbots View project

RealME: A Digital Tool for Prisoners to Develop Employability Skills View project

All content following this page was uploaded by Daniëlle Duijst on 13 July 2017.

The user has requested enhancement of the downloaded file.

Can we Improve the User Experience of
Chatbots with Personalisation?

University of Amsterdam

A thesis submitted for the degree of

Master of Science

Daniëlle Duijst
Studentnumber: 11266988

First reader:
Dr. Jacobijn Sandberg

Second reader:
Dr. Dan Buzzo

Supervisor Capgemini:
Drs. Wouter Slager

July 13, 2017

1
 Can we improve the User Experience of Chatbots with
Personalisation?
Daniëlle Duijst
11266988
University of Amsterdam
Science Park 904
danielle.duijst@student.uva.nl
ABSTRACT
A conversational interface is an interface that the user can
interact with by means of a conversation. The conversation
can occur by speech but also by text input. When a conver-
sational interface uses text, it is also described as a chatbot
or a conversational agent. In this study, the user experience
factors of these so called chatbots were investigated. The
main research question is “What is the added value of per-
sonalisation for the user experience of chatbots?”. A variety
of user experience frameworks and evaluation methods are
described in the literature review. Two kinds of chatbots
have been designed, one with and one without personali-
sation factors. The design of this research is a two-by-two
factorial design. The independent variables are the two chat-
bots (unpersonalised versus personalised) and the specific
task or goal the user can achieve with the chatbot in the
financial field (a simple versus a complex task). The results
are that there was no significant interaction effect between
personalisation and task on the user experience of chatbots.
A significant difference was found between the two tasks
with respect to the user experience of chatbots, however
this difference was not due to personalisation. Also in our
study, we created the satisfaction construct which appeared
to be reliable.
Keywords
User Experience, Chatbots, Conversational Interfaces, Per-
sonalisation, Finance, Fintech
1. INTRODUCTION
Chatbots are computer programs that are written to sim-
ulate a conversation with the user by means of auditory or
textual inputs. Recently, the popularity of using chatbots
within companies increased (Knight, 2016). Companies are
advancing technically and companies research and explore
more possibilities in the area of artificial intelligence and ma-
chine learning (Press, 2016). When using a new technology,
such as chatbots, it is important to know how to implement
this technology in a good way. One way to measure the
quality of technology, is by looking at the user experience.
Also with conversational interfaces, in particular chatbots,
user experiences plays a big part (O’Brien, 2017). The user
experience of machine learning applications becomes more
popular to research because these technologies did not expe-
rience as much design innovations as other technologies have
(Dove, Halskov, Forlizzi, & Zimmerman, 2017). Therefore,
2
this study focuses on researching the kind of factors which
influence the user experience of chatbots. Previous research
has shown that personalisation factors influences the user ex-
perience (Fan & Poole, 2006). Personalisation is “a process
that changes the functionality, interface, information access
and content, or distinctiveness of a system to increase its
personal relevance to an individual or a category of individ-
uals” (Fan & Poole, 2006).
Because of the gap in research about user experience of
machine learning applications, and the potential of person-
alisation, we chose the following research question “What is
the added value of Personalisation for the User Experience
of Chatbots?”. The research is conducted at the Financial
Services business unit of Capgemini (Capgemini, 2017).
The structure of this document is as follows: Firstly, we
discuss the relevant literature. Secondly, we present the re-
search question and hypotheses. Thirdly, we explain our
methodology and our approach. Fourthly, the results are
presented. Thereafter, the results are presented and dis-
cussed.
2. RELATED WORK
2.1 Chatbots
A conversational interface is an interface that the user
can interact with by means of a conversation. This can
be via speech, but also via typed natural language. The
terms Natural User Interface (NUI) and Conversational In-
terfaces are sometimes used interchangeably. A NUI is an
interface where you interact by using natural inputs like:
speech, touch and hand gestures (Wigdor & Wixon, 2011).
Sometimes, NUIs can recognise faces, their environment or
emotions (Kaushik, Jain, et al., 2014). A chatbot is an ex-
ample of a conversational interface. When we speak about
chatbots, or conversational agents, we mean a computer pro-
gram that is written to simulate a conversation with the user
by means of auditory or textual inputs. In this study, we
focus solely on chatbots that use textual inputs. Chatbots
are used online and are sometimes driven by artificial intel-
ligence.
2.1.1 Implementations of chatbots
Chatbots can be used for a wide range of fields, such
as education (Letzter, 2016), information retrieval (Shawar,
Atwell, & Roberts, 2005), business and e-commerce (Chai et
al., 2001), and for customer service (MarutiTechlabs, 2017).
In their study, Tatai et al. (2003) compared implementa-
tions of chatbots and identified three main roles of chatbots,
namely:
• Role of a Digital Assistant;
• Role of an Information Provider;
• Role of a General Chatbot.
(Tatai, Csordás, Kiss, Szaló, & Laufer, 2003)
Examples of digital assistant chatbots are from IKEA, which
was launched in 2005 1 and Niki.ai which helps you shop
through chat (Niki, 2017).
Chatbots can be used to improve the communication be-
tween doctor-patient (DP) and clinic-patient (CP). The re-
searchers claim that using chatbots for this purpose (DP and
CP communication) could reduce costs and time on routine
operations (Abashev, Grigoryev, Grigorian, & Boyko, 2016).
More implementations of chatbots are: credit score coach,
lawyer, personal stylist, food orderer, personal concierge,
doctor, pension or finance advisor, teacher, newsreader, toy,
accountant, and lastly, as a partner (Jee, 2017).
2.1.2 Chatbots in The Netherlands
In The Netherlands, companies are also starting to use
chatbots for example on their Facebook pages or company
websites (Adformatie, 2017; Emerce, 2016; Goodfellow, 2017;
ASR, 2017). Another example of an implementation of chat-
bots is Dawnbot. Dawnbot is a Dutch chatbot that helps
you create a video advertisement (Dawnbot, 2017). The user
answers the questions dawnbot asks by selecting the corre-
sponding emoji. A possible advantage of communicating
with emoji’s could be that the interaction can be quicker,
and more efficient. On the other hand, miscommunications
can occur between the user and the maker of the system,
since emoji’s can be interpreted differently (Miller et al.,
2016).
2.1.3 Customer Service: Finance & Insurance Chat-
bots
Chatbots can be found in different domains such as: cus-
tomer service, e-commerce, insurance, healthcare, retail and
more. 95% of the respondents of a chatbot survey believed
that the customer service domain is going to be ’the major
beneficiary of chatbots’ (Mindbowser, 2017). We choose this
field to illustrate the financial and insurance field specifically.
In the field of insurance, we also see a rise of chatbots.
Lemonade is an insurance company from the US. Lemonade
claims that they transform the business model of insurance
by: ”injecting technology and transparency into and indus-
try that often lacks both”. They say that they provide a fast,
affordable and hassle free insurance experience (Lemonade,
2017).
The expert virtual insurance agent Evia was released last
year (2016). Evia is created by Insurify, a spinoff company
from MIT (Buhr, 2016). Evia tries to find you a car insur-
ance by using a photo of your license plate. The makers of
Evia claim that they will match you with the right coverage
package for you situation and needs, will monitor rates for
you, gather multiple quotes from companies in one place and
will skip the process of filling in long, confusing forms.
An example of a chatbot implementation in the financial
field is K2 Bank. This chatbot checks your balance, your
1
https://www.youtube.com/watch?v=rRY5XtLpNuU
3
recent history of transactions and can make simple money
transfers (Lipiec, 2017).
2.1.4 Conversational aspects
Human-system dialogue consists of the inquirer (the user),
looking for information, and the expert (system), providing
information. There are two ways that chatbots can converse
with users. System-initiated chatbots, where the system
leads the conversation, and user-initiated chatbots, where
the user leads the conversation. Systems that contain both
methods of initiation, are called mixed initiative systems
(Hung, Elvir, Gonzalez, & DeMara, 2009).
To understand how a conversational interface should be
represented, it is important to investigate how human dia-
logues work. Quarteroni, et al. (2006) researched the as-
pects and issues related to human dialogues and they pro-
posed a list of essential item for an interactive question and
answering system:
• Context Maintenance: utilising the context of the con-
versation to correctly interpret the user’s input. This is
important for follow-up questions, or for clarification.
• Utterance Understanding: the detection of follow-up
and clarification within the context of the previous
conversation.
• Mixed Initiative: the user should be able to take ini-
tiative within the conversation (by quitting, or asking
questions).
• Follow-up Proposal: meaning that the system moti-
vates the user to give feedback on the answers that
the system gives (if the user is satisfied or not). Until
the user has achieved his or her goal.
• Natural Interaction: covering and generating a variety
of utterances to create a smooth conversation and to
keep the dialogue active.
(Quarteroni & Manandhar, 2009)
McTear (2016) reviewed key features of conversation in
chapter 3 of his book on conversational interfaces. He de-
scribes that the following aspects are important when de-
signing conversational interfaces:
• Conversation as action: meaning that utterances of
users could be seen as actions that speakers carry out
to achieve a goal.
• The structure of conversation: regarding how utter-
ances from a conversation relate to each other. Exam-
ples of ways to recognise structure in dialog acts can
include adjacency pairs, exchanges, discourse segments
and conversational games.
• Conversation as a joint activity: describes how two
parties take turns and reduce the risk of a miscommu-
nication by using the grounding process in their con-
versation.
• Conversational repair: a repairing process that can be
initiated by one of the two parties in the conversation.
Sometimes the speaker repairs his own utterances be-
fore the receiver has time to repair it.
 • The language of conversation: the tone of voice in a
spoken text can be a way to convey additional informa-
tion, such as emotions and affect. For example, when
persons raise their voice when they are angry. For writ-
ten text, emotions and affect can also be conveyed, for
example by using emoticons or capital letters.
(McTear, Callejas, & Griol, 2016)
Incremental processing is an important process in human
to human conversation (McTear et al., 2016). Incremental
processing means that overlap occurs within a conversation.
In human-machine interaction, a latency between turns is
present. This is one reason why human-computer interac-
tion can sometimes feel less natural than human to human
interaction. Another benefit of incremental processing is
that the dialog becomes more fluent and efficient (McTear
et al., 2016). Google search applies incremental processing
by completing the user’s query during typing and in Voice
Search by showing the recognised words while the user is
still speaking.
2.1.5 Dialogue Management Strategy
McTear (2016) describes that one of the core aspects of
conversational interfaces is the design of the dialogue man-
agement strategy, in which the system’s conversational be-
haviour is defined. The design of dialogue management
strategy was done manually in the past, but the research
community has found ways to automate this process by
training the model with real conversations (McTear et al.,
2016).
Two arising design strategies of dialogues in chatbots are
the interaction strategy and the choice of a confirmation
strategy (McTear et al., 2016). Automatic speech recogni-
tion is not always accurate, but by asking the user for con-
firmation or reprompt them, some errors could be avoided.
Too many confirmation asking can also be annoying.
There are three types of interaction strategies in chatbots,
namely: user-initiated, system-initiated or mixed initiative.
Limitations of user-initiated dialogues are errors in speech
recognition and understanding, since users can say anything
they want. The limitation of system-initiated dialogues, is
that the user’s input is limited, but the advantage is that
the interaction is more efficient. The advantage of a mixed-
initiated dialogue is that the system can guide the user, but
the user is also free to say anything he wants and take ini-
tiative, ask questions and introduce new topics. The limi-
tations are that the system has to be technically advanced
to keep track of his own structure/agenda, understand and
answer the user’s utterances correctly and remember the rel-
evant information said.
Confirmation strategies are strategies to prevent errors in
recognition and understanding of the user’s utterances. The
disadvantage of using these confirmations is that it can make
the interaction inefficient, repetitive and lengthy which can
eventually lead to a frustrating user experience. A solution
for these problems, is to create implicit confirmations. In
this way, the user’s input is used in the next system’s out-
put and extra information is added. To clarify the difference
between implicit and explicit confirmation, we give the fol-
lowing example. When the user says the price of the house
he wants to buy is e180.000, the system could say: ”So the
house you want to buy is e180.000?”, which is an explicit
confirmation. An implicit confirmation of the system would
be: ”So the house you want to buy costs e180.000. How
4
much will an optional refurbishment add to the costs?”. A
limitation of implicit confirmation strategy is that the user is
responsible for correcting the system when it did not recog-
nise the user’s utterances correctly (McTear et al., 2016).
2.1.6 Maturity Model Chatbots
Smiers (2017) describes some levels of maturity of chat-
bots based on three categories, namely: interaction, intel-
ligence and integration (see Fig 1). The interaction area
describes that the user experience of chatbots is different
than the user experience of websites, since the interaction
with chatbots is done through textual input. The intelli-
gence area describes the capability of the chatbot to under-
stand and provide a relevant utterance which is in line with
the intent of the user. The integration area is about the
back-end of the chatbot and how well it is integrated with
other websites, servers or services from other websites and
applications (Smiers, 2017).
2.2 User Experience
User experience is a modern concept, which includes “all
the users’ emotions, beliefs, preferences, perceptions, physi-
cal and psychological responses, behaviours and accomplish-
ments that occur before, during and after use.” (ISO 9241-
210, 2010). This broadness implies that many aspects are
involved in the user experience. To scope our study, we
divide user experience in three basic needs: usefulness, us-
ability and user satisfaction.
2.2.1 Usefulness
The Technology Acceptance Model (TAM), is developed
to test the user acceptance of information systems. Ques-
tions of the standardized TAM questionnaire measure the
perceived usefulness and the perceived ease of use of infor-
mation systems (Davis, 1985). The 10 items that measure
usefulness are:
1. Using this product improves the quality of the work I
do.
2. Using this product gives me greater control over my
work.
3. This product enables me to accomplish tasks more
quickly.
4. This product supports critical aspects.
5. This product increases my productivity.
6. This product improves my job performance.
7. This product allows me to accomplish more work than
would otherwise be possible.
8. This product enhances my effectiveness on the job.
9. This product makes it easier to do my job.
10. Overall, I find this product useful in my job.
The other half of the TAM questionnaire, measures the ease
of use.
 Figure 1: Maturity model for chatbots by Smiers (2017)
2.2.2 Usability
Usability is focused on the performance of the task and is
an integrated aspect within user experience (Kaye, 2007).
To measure Usability, a popular post-test that could be
used, is the System Usability Scale (Brooke, 1985). This
SUS questionnaire is described as a ’quick and dirty’ method,
because the execution is relatively fast since the question-
naire only consists of 10 questions which the users have to
answer without thinking too much about it. Another advan-
tage of using the SUS method, is that no license is needed
and a score can be calculated which can be easily compared.
The SUS provides an easy scoring system. These scores can
also be compared to the scores that the researchers identi-
fied. The mean of the SUS scores that the researchers found
is 68. With this scoring system, we can compare the score
of chatbots to other information systems.
Finstad (2010) investigated Usability Metrics for User Ex-
perience (UMUX) which consists of four questions which are
highly correlated (0.8) with the SUS scale. Therefore Fin-
stad claims that the UMUX can be used as a standalone
usability metric (Finstad, 2010). The ISO definition of us-
ability is about effectivity, efficiency and satisfaction of a
system. The four likert scales presented in UMUX question-
naire represent the ISO definition, they are:
1. The system’s capabilities meet my requirements;
2. Using this system is a frustrating experience;
3. This system is easy to use;
4. I have too spend too much time correcting things with
this system (Finstad, 2010).
Lewis (2013) confirms in a critical review on the UMUX
questionnaire that researchers who need a shorter question-
naire than SUS should consider using the UMUX question-
naire (Lewis, 2013).
5
SUPR-Q (Standardized Universal Percentile Rank Ques-
tionnaire) is a rating scale which measures the perception
of the user with respect to usability, credibility, trust, ap-
pearance, and loyalty for websites (Sauro & Lewis, 2016).
The reliability of the SUPR-Q scores is all above 0.7, which
means that this scale is reliable. The limitation of this ques-
tionnaire is that it mainly focuses on websites and the goals
which can be achieved on websites. Therefore we think it is
not an easy questionnaire to apply to chatbots. For exam-
ple the question ”It is easy to navigate within the website” is
hard to apply to chatbots, since navigation plays a different
role in chatbots than on websites.
2.2.3 Satisfaction
McTear describes that the most popular methodology for
performing overall system evaluation, is the PARAdigm for
DIalogue Evaluation System (PARADISE) (McTear et al.,
2016). This method evaluates the user satisfaction on a
scale, after the users have interacted with the system. Hung
et al. (2009) propose evaluation metrics derived from the
PARADigm for DIalogue System Evaluation (PARADISE)
in their paper. These evaluation metrics reflect the dialog
performance and task success as described above, see Table
6 in the Appendix.
Quarteroni, et al., evaluated their Interactive Question
& Answering system, YourQA, by collecting user feedback
(Quarteroni & Manandhar, 2009). One way they collected
user feedback, was by gathering objective information from
chat logs. Also, they collected information about the user
experience, by using a questionnaire about a Wizard of Oz
experiment. The Questions were inspired by a WOz exper-
iment from (Munteanu & Boldea, 2000). See Table 1 for
their questionnaire. In this questionnaire, Q1 and Q2 assess
the performance. Q3 and Q4 focus on interaction difficul-
ties and Q5 and Q6 focus on the overall satisfaction of the
user. Q7 and Q8 were taken from the PARADISE evalua-
tion metrics (PARAdigm for DIalogue System Evaluation) use.
(Walker, Litman, Kamm, & Abella, 1997). Q9 was chosen
by the researchers to measure the perceived difference be- • User engagement relates to behavioural proxies such
tween their two prototypes. The result of their evaluation as the frequency, intensity, or depth of interaction over
was that users tend to be more satisfied with the interactive some time period. For example, if a user uses your app
version of YourQA than with the baseline version. The ques- one hour everyday.
tionnaire they used appeared to be sufficient for achieving • Adoption metrics track how many new users start us-
their evaluation goals. ing a product during a given time period.
The Questionnaire for User Interaction Satisfaction (QUIS),
assesses user’s subjective satisfaction with respect to specific • Retention metrics track how many of the users from a
aspects of the human-computer interface (Chin, Diehl, & given time period are still present in some later time
Norman, 1988). QUIS measures attitude towards eleven in- period.
terface factors: screen factors, terminology and system feed-
back, learning factors, system capabilities, technical man- • Task success encompasses the traditional behavioural
uals, on-line tutorials, multimedia, voice recognition, vir- metrics of user experience, such as efficiency (e.g. time
tual environments, internet access, and software installation. to complete a task), effectiveness (e.g. percent of tasks
Each area measures the users’ overall satisfaction with that completed), and error rate.
facet of the interface, as well as the factors that make up
that facet, on a 10-point scale. The advantage of the QUIS 2.2.5 User Experience Questionnaire
is that the researchers successfully created a measure that
After analysing the methods and questionnaires, we choose
is highly reliable across many types of interfaces. Therefore
to use the following for our research.
we can say that this questionnaire could also be reliable if
For usability, we use the UMUX questionnaire. The ad-
we apply it on a conversational interface, such as a chatbot.
vantage of the UMUX questionnaire is, that it consists of
User satisfaction with search dialogues has been defined
only four questions which are reliable & valid (Finstad, 2010;
by analysing real dialogues of a commercial intelligent assis-
Lewis, 2013). Since we want to measure three things, we
tant (Kiseleva et al., 2016). The researchers also found that
want to use short and reliable questionnaires.
dialogues consist of a single task search dialogue or a multi-
To measure usefulness, we use the usefulness questions
task search dialogue. The user satisfaction was measured by
from the TAM questionnaire. The advantages of the TAM
the aggregation of all dialogue’s tasks, rather than dialogue’s
questionnaire is that it is a standardised and commonly used
queries separately, because of the dependency between the
questionnaire to measure usefulness and ease of use (Davis,
tasks and that the context of the dialogue is also important
1985). We only use the questions that measure usefulness.
for the user satisfaction (Kiseleva et al., 2016). This means
We want to measure satisfaction by using one question
that when researching the user experience of chatbots, the
from the chatbot evaluation questionnaire from Quarteroni
whole task should be used as a measurement for the user
et al. (2007). Based on previous experience with measuring
experience instead of seperate tasks (due to the dependency
the happiness aspect of the HEART framework, we created
between tasks).
three other questions (e.g. if the user enjoyed using the chat-
2.2.4 Industry defined metrics for Conversational In- bot, if they would recommend it to a friend and if there were
terfaces any functionalities missing). In total we combined four ques-
tions to measure satisfaction. We did not choose a specific
Besides the previously discussed three needs of user expe-
satisfaction questionnaire, because these had overlap with
rience, there is also user experience metrics from the indus-
the usability questionnaires.
try. An objective in the industry is, among others, to reduce
costs and ensure user satisfaction. Objective metrics from 2.3 Personalisation
the industry could be: time-to-task (time to start engag-
ing), correct transfer rate (if customers are redirected ap- 2.3.1 Personality
propriately), containment rate (percentage of calls handled
McTear defines personality as a set of characteristics of
by the system) and abandonment rate (percentage of calls
people that uniquely influences their cognitions, motivations,
hang up). Subjective measures can be done by e.g. ques-
and behaviours in different situations (McTear et al., 2016).
tionnaires, such as Subjective Assessment of Speech System
Five traits of personality of humans are defined by Mc-
Interfaces (SASSI) which covers six factors, namely: system
Crae and John (1992), namely: openness, conscientious-
response accuracy, likeability, cognitive demand, annoyance,
ness, extroversion, agreeableness and neuroticism (OCEAN)
habitability and speed. These factors could also be applied
(McCrae & John, 1992).
to conversational interfaces without speech.
With respect to personality in (information) systems, Leite
Another way to measure the User Experience, is by using
et al. (2013) did an evaluation of the friendliness of robots
user-centered metrics such as described in Google’s HEART
my measuring the: stimulation of companionship, intimacy,
framework (Rodden, Hutchinson, & Fu, 2010). The re-
reliable alliance, self-validation, and emotional security (Leite
searchers (at Google) describe that the HEART framework
et al., 2013). For chatbots, we need to find a middle way in
can measure user experience quality. HEART is an abbre-
characteristics, since robots and humans both posses a vi-
viation for: happiness, engagement, adoption, retention and
sual appearance, whereas with chatbots this would manifest
task success.
through a user interface.
• Happiness consists of factors which relate to subjective Personality traits of users could be assessed before they
aspects of user experience, like satisfaction, visual ap- use the chatbot to fine-tune the user model and create a
peal, likelihood to recommend, and perceived ease of better fitting user experience. Due to technical and time

6
 Table 1: Chatbot Evaluation Questionnaire (Quarteroni et al., 2007)

N0 Questions Focus on
Q1 Did you get all the information you wanted using the system? Performance
Do you think the system
Q2 Performance
understood what you asked?
Q3 How easy was it,to obtain the information you wanted? Interaction Difficulties
Q4 Was it easy to reformulate your questions when you were invited to? Interaction Difficulties
Q5 Overall, are you satisfied with the system? Overall Satisfaction
Q6 Do you think you would use this system again? Overall Satisfaction
Q7 Was the pace of interaction with the system appropriate? * PARADISE evaluation metrics
Q8 How often was the system slow in replying? * PARADISE evaluation metrics
Q9 Which version of the system did you prefer and why? * Measure Perceived Difference

* Q7, Q8 and Q9 are additional questions when the user used both prototypes

Table 2: Four types of personalisation and their focus (Fan & Poole, 2006)

Type of personalisation: Focuses on: Motives to use:

Creating a pleasant user space and unique experience. Aesthetic value and
Architectural
Relates to the interface aspect of the system. expressing himself/herself
Utilising the information systems to enhance efficiency
Instrumental Productivity/efficiency
and personal productivity.
Mediating the interpersonal relationships and utilising Welfare/psychological
Relational
the relational resources. well-being
Differentiating the product service and information to Material and psychic
Commercial
increase sales and customer satisfaction. well-being

limitations, we did not apply this in our chatbot. Also,
filling in a questionnaire before using the chatbot could be a
tedious experience for the user and there is a risk of wrongly
labelling the user.
2.3.2 Personalisation
In a survey about personalisation from Fan & Poole (2006),
the researchers found that the definitions of personalisa-
tion differ per field that the concept has been researched
in. Personalisation has been researched in a variety of do-
mains, such as: knowledge retrieval and recommender sys-
tems (Shaikh, Phalke, Patil, Bhosale, & Raghatwan, 2016).
In a marketing study (Postma & Brokke, 2002), the re-
searchers found that personalisation, and self-reported inter-
ests, appeared to have a positive effect on the click-through
rate of their newsletters send by e-mail. They also found
that the personalisation effect grows over time. Most re-
search on personalisation is done in the field of computer
science, marketing and e-commerce (Fan & Poole, 2006). In
marketing and e-commerce the focus of personalisation lays
on delivering unique value and benefits to each individual
customer. Whereas in the human interaction field, person-
alisation is seen as a way to close the gap between user and
computer. Fan & Poole (2006) analysed the definitions and
found that most of the definitions of personalisation consist
of these elements:
1. The purpose or goal of personalisation
2. What is personalised (e.g. interface or content)
3. The target of personalisation (e.g. user or consumer)
The definition of personalisation we use in this study is that
personalisation is “a process that changes the functionality,
interface, information access and content, or distinctiveness
of a system to increase its personal relevance to an individual
or a category of individuals” (Fan & Poole, 2006).
The personalisation effect describes that students learn
more deeply from a multimedia explanations when this is
presented to them in a conversational style (Mayer, 2003).
Also, when presenting a conversational style (instead of ”the
financial situation” the system says ”your financial situa-
tion”), the students are more likely to engage in cognitive
processes.
Three reasons to personalise are to access information,
to accomplish work goals and to accommodate individual
differences (Fan & Poole, 2006). McTear (2016) states that
when you want agents to become recognisable individuals,
they should have life-like interaction capabilities, such as
convincing and intuitive behaviour. When an agent has a
similar personality as the user, the perceived intelligence and
competence of this agent increases (McTear et al., 2016).
The implementation of personalisation is constructed in
three dimensions (McTear et al., 2016):
1. What to personalise: four aspects that can be person-
alised are: content, user interface, channel/information
access and functionality.
2. To whom to personalise: the target of personalisation
can either be individuals or a group of individuals.
When the user identifies with a certain category, the
user is likely to perceive the personalisation as if it is
personalised for him or her.
3. Who executes the personalisation: personalisation can
happen based on the input and information given by
the user (explicit personalisation) or personalisation

7
 can be done automatically by the system (implicit per-
sonalisation).
Different technologies to enable personalisation in infor-
mation systems can be divided in the following categories
(Amoroso & Reinig, 2004):
• User-behavior tracking: for example by clickstream
tracking, hover technologies, pattern recognition;
• Personalisation database: for example by collaborative
filtering, webhousing, intelligent agents, data mining,
profiling, statistical analysis;
• Personalised User Interface Technologies: for example
by content management, streaming audio/video, user,
information filtering, user-preference interface design,
personalised searching;
• Customer Support Technologies: just-in-time customer
support, wireless customer service.
When designing personalised systems, a distinction can
be made between affective design, which is process-oriented,
and utilitarian design, which is result-oriented (Fan & Poole,
2006). Also, research supports that the majority of per-
sonalisation systems are designed to enhance productivity,
because people us these systems to achieve a goal. Person-
alised systems may not only fulfil the functional aspects of
human needs, but also their entertainment needs (Fan &
Poole, 2006).
Fan & Poole (2006) distilled four ideal types of personali-
sation in their survey, namely: the architectural, relational,
instrumental and commercial perspective (see Table 2). Ar-
chitectural personalisation focuses on the construction of the
digital environment to create a pleasant user space and a
unique experience for the user through arrangement and de-
sign of digital artefacts in a way that meet the user’s needs
and reflect his or her style and taste. This perspective re-
lates to the interface aspect of the system. Instrumental
personalisation refers to utilisation of information systems
to enhance efficiency and personal productivity by provid-
ing, enabling, delivering useful and user-friendly tools in a
way that meet the user’s needs. Three aspects of instrument
personalisation are:
1. Providing tools;
2. Designing tools;
3. Utilising tools.
Also, instrumental personalisation highlights the importance
of the user’s situated needs. Relational personalisation re-
lates mainly to the differentiation of product service and
information to increase sales and customers satisfaction by
addressing their needs and goals in a given context. Com-
mercial personalisation fulfils the users material needs and
therefore contribute to their psychic welfare (Fan & Poole,
2006). Fan & Poole (2006) discuss that a combination of
paradigms could best meet the different needs of the users.
2.3.3 Social Behaviour
Examples of social behaviour which could be implemented
within conversational interfaces are mirroring and showing
empathy.
8
Mirroring is the alignment of an emotional response be-
tween two parties, for example an agent and an user. Mir-
roring can make the agent perceive as socially competent
(Iacoboni, 2009). McTear (2016) describes that mirroring
can occur in embodied conversational agents by: choice of
vocabulary, head posture, gestures and facial expressions.
For chatbots the above mentioned aspects applicable are
choice of vocabulary and facial expressions in the sense of
emoji’s (also knowns as ideograms, smileys or emoticons
which are used in electronic messages to convey an emo-
tion or facial expression). McTear claims that agents that
use effective mirroring can be perceived as being more em-
pathic.
Conversational agents which are made for coaching, health
care or as social companions should be empathic and trust-
worthy to be perceived as social (McTear et al., 2016).
McTear (2016) states that when an agent always behaves
in the same way, the user satisfaction could decrease over
time. He also describes that boredom is more persistent
and associated with poorer learning and less user satisfaction
than frustration. Therefore we can say that user engagement
plays a big part in maintaining a long-term relationship with
your users. To improve user engagement, long-term user
models can be build to predict the future user’s affect and
actions (McTear et al., 2016).
Applying humour can provide a more fun and believable
robot (Tatai et al., 2003). Aspects that users may not ap-
preciate from robots are social control and criticism (Baron,
2015). This behaviour could also be not appreciated in chat-
bots.
2.4 Applying personalisation in chatbots to im-
prove User Experience
Kim et al. (2014) investigated chatbots that provides re-
sponses based to the user input, the user itself and its own
personality and common sense. The systems stores user-
related facts automatically from user input. The system also
keeps track of changes in user interest (Kim et al., 2014).
Jeesoo Bang et al. (2015) introduced a chatbot based
on EBDM system with a personalisation framework using
long-term memory (Bang, Noh, Kim, & Lee, 2015). The
researchers found that implementations of chatting systems
were: using EBDM dialogue system, using dialogue acts and
part of speech (POS) tagged tokens and using long-term
memory and knowledge extractor.
The study of Shaikh et al. (2016) paper gives a survey
on previous work about chatbots and describes an example
based chat-oriented dialogue system using personalised long
term memory. In their study, the researchers describe a chat-
bot framework based on Example-based Dialogue Manage-
ment (EBDM). An EDBM is a system that uses dialogue ex-
ample to index information, rather than probabilistic mod-
els or rules. The advantage of using an EBDM framework
is that it frees the designer from manual labelling and anno-
tation. The results from the researched experiments show
that using this framework could improve the performance of
the system (Shaikh et al., 2016).
2.5 Tasks
A task is an action that can be done on a website or
information system. There are different tasks possible in the
financial domain. To investigate which tasks were relevant to
use for our study, we investigated three banking websites in
The Netherlands (ABN AMRO, Rabobank and ING). The
analysis can be found in Figure 5 in the Appendix.
The overlapping categories of the tasks found on the web-
sites are: paying, saving, mortgage, loan, investing, insur-
ance and retirement. We divided the tasks into three cate-
gories, namely:
• Simple tasks, for example: opening a bank account,
blocking a bank card, finding basic information and
activate mobile banking.
• Complex tasks, for example calculations of mortgage
or loan which is rule-based information. Also, advisory
tasks based on rules.
• Very complex tasks, for example giving advice for a
specific package of a specific situation where emotions
and human interaction is essential.
Other dimensions we found after analysing the main user
tasks on the website, were problem-based versus opportunity-
based user tasks.
Table 3: Types of Tasks
Problem-based Opportunity-based
Simple Task 1 - The structure consists of the sections participants, design,
Complex - Task 2
For this research, we want to compare two tasks that dif-
fer fundamentally from each other. Therefore we chose two
tasks with a different goal, however both tasks are placed
in a financial context. Task 1 is mainly focused on a simple
and problem-based scenario. Task 2 is more focused on a
complex and opportunity-based scenario, see Table 3.
The tasks we use for our chatbots are:
• Task 1: Blocking bankcard
The chatbot solves a problem for the user. In this
case, the user lost his bankcard and has to block his
bankcard. In this task, urgency also plays a role.
• Task 2: Financial advice
The chatbot fulfils the role of an financial advisor. The
user can do a financial ’health’ check. In the second
part, if the user has spare money, the chatbot advices
on what kind of option the user has (e.g. investing,
fixed savings account, etc.). Basically, the chatbot
functions as a (simple) recommender system.
2.6 Hypotheses
As mentioned in the introduction, our research question
is:
“What is the added value of Personalisation for
the User Experience of Chatbots?”
It is not clear from the literature it is not clear if users
want a personalised chatbot, that shows empathy, when
there is a sense of urgency present. On one hand we could
assume that users prefer a chatbot that shows empathy to
calm them down after their bank card is stolen. On the
other hand, we could assume that users do not care about
personalisation and empathy in a chatbot when they are in
a hurry.
9
The same counts for the second task (financial advice).
We could assume that users prefer personal advice, or the
users could prefer no personality, because they just want to
obtain the advice from the chatbot.
We do not know exactly what the effect of personalisation
could be, therefore we investigate the interaction effect of
personalisation and tasks with respect to the user experi-
ence. We wanted to find out if personalisation matters to
users.
We used the following hypotheses:
H1 0: Personalisation has no effect on the User
Experience of chatbots.
H1 1: Personalisation has a positive effect on the
User Experience of chatbots.
H2 0: The User Experience is the same for simple
and complex tasks.
H2 1: The User Experience is different for simple
and complex tasks.
Hinteraction: Personalisation and Tasks have
an interaction effect.
3. METHODS
In this section, we describe the methods of this study.
materials and procedure.
3.1 Participants
We have 121 respondents for our online survey. In group
1 we had 31 respondents, group 2 had 30 respondents, group
3 had 28 respondents and group 4 had 32 respondents. 9%
was below 20 years old, 61% of our data set was between
20 - 25 year old, 18 % was between 25 - 30 years old. The
other 12 % was older than 30 years old. 57% of the people
was female, 41% was male and 2% did not want to say. The
majority of our group was highly educated, namely 41% aca-
demics (WO), 17% applied university (HBO), 21% academic
preparation schooling (VWO) and 12% applied university
preparation schooling.
The qualitative data was obtained by convenient sampling
of our sample group. This means we selected five people at
Capgemini to test one chatbot individually. Which makes a
total of 20 participants for the qualitative user testing. The
participants differed in gender and age, but were mainly
around 25 years old and technically literate. They took part
in the research, because of personal reasons and they got a
small reward (a Dutch cookie).
3.2 Design
For this research, we used a mixed methodology. First we
sent out a survey. Subsequently, we conducted qualitative
user tests and observations.
The research design is a 2x2 between-subject design. We
compared the two independent variables with a dependent
variable. The two major independent variables (factors) are:
The Chatbots (unpersonalised & personalised) and the type
of tasks of the Chatbot (blocking the bankcard & asking for
financial advice). The dependent variable is the UX score
which is calculated with the results of the survey.
3.3 Materials
3.3.1 Chatbot
The chatbot can be placed in level 1 of the maturity model
for chatbots (Smiers, 2017). This means that the interaction
will be done through one language, one channel and human-
to-bot interaction. The intelligence will consist of a simple
Q&A, menu-based and word-based rules. Integration with
other websites is done through simple links.
We used chatfuel due to time and skill restrictions. Chat-
fuel is a platform that enables the creation of chatbots with-
out needing programming skills (Chatfuel, 2017). This sig-
nificantly shortens the time and effort needed to create the
four chatbots needed for this research. Also, the platform is
free to use and unlimited chatbots can be created. Another
advantage is that Chatfuel supports multiple languages, such
as Dutch which is used in our case.
The structure of the chatbot’s conversation needed to be
designed. To do this, we created conversational user flow
diagrams which we used as our blueprint for the chatbots.
These conversational flow diagrams can be found in the ap-
pendix. In Figure 6 we illustrate the conversational user
flow of the first task of the chatbot, namely: blocking a bank
card. Different icons are placed to see which actor is doing
what, for example the user icon shows that the user has to
either give input by choosing from quick replies or by a free
text field. In Figure 7 we illustrate the conversational user
flow of the second task, namely: receiving financial advice. Figure 2: Startscreen of the personalised chatbot for block-
We exported and tested the chatbot in Facebook Messen- ing a bank card. The options are: “My card is stolen”, “I
ger. Facebook messenger is a instant messaging service de- lost my card” and “Nothing”.
veloped by Facebook and has 1.2 billion users (Kerr, 2017).
Facebook Messenger was chosen as the environment in which
the chatbot was exported and tested, due to practical rea-
sons and limitations of Chatfuel. The chatbot is shown in
Figure 2 and 3. The rest of the screenshots, and the links of
the chatbots, can be found in the Appendix.

3.3.2 Quantitative Test

The questions of our questionnaire were made on Google
Form and were subdivided in categories. The first four ques-
tions are based on the UMUX questionnaire and measure
usability. The second group of four questions measures the
usefulness of the chatbot and is based on the usefulness
questions of the TAM questionnaire. The third group of
four questions measures the user satisfaction of the chatbot.
These four questions are based on the chatbot evaluation
questionnaire from Quarteroni et al (2007). Together, these
questions form the base of our user experience questionnaire.

3.3.3 Qualitative Test

For the qualitative research, we used the think aloud method
(Someren, Barnard, Sandberg, et al., 1994). This means
that the user was asked to think aloud while executing the
task so that the user experience during the task could be
measured.
Besides making notes of the user’s utterances, we also
recorded the screen and the audio during the observations.
The test was executed on a Macbook Pro. The researcher
coded certain emotions and utterances into a coding scheme
simultaneously (see Figure 12 in the Appendix). The base Figure 3: The second screen of the personalised chatbot for
emotions chosen were based on Plutchick’s wheel of emo- blocking a bank card. The chatbot says: “What a shame
tions (Plutchik, 2003). Once the task was completed, the your pass is stolen! Don’t forget to report it to the police!”
tester was asked about his or her experience and interaction

10
with the chatbot. The tester was asked to elaborate on any 4.1 Quantitative results
difficulties experienced. Then the structured questions were In this section we will discuss the results of our data anal-
the following: ysis. The descriptive statistics of our groups can be found
in Table 4.
• What did you think of this experience? Why (not)?
Table 4: Means and Standard Deviations of the User Expe-
• Do you think you would use this chatbot in real life? rience Score
Why (not)?
Chatbot Group n M SD
• Do you think this chatbot is useful? Why (not)? Task 1, Unpersonal 31 5.44 0.932
Task 1, Personal 30 5.35 1.087
• Did you think there were functionalities missing in the Task 2, Unpersonal 28 4.53 1.548
chatbot? Which ones? Task 2, Personal 32 4.51 1.448

3.4 Procedure
After implementing and testing the chatbots ourselves,
the chatbots were ready for being tested by users. We used
a combination of quantitative and qualitative research meth-
ods to obtain the results from user testing.
The quantitative data were collected through surveys. Peo-
ple were only allowed to take part in one survey. In this
questionnaire, the user was first asked to answer questions
on demographics. Then the user was asked to open the link
of the chatbot in Facebook Messenger to execute the ex-
plained task. After completing the task the user was asked
to return to the questionnaire and finish to answer the last
12 questions about the chatbot. In total, the whole survey
took around 10 minutes.
The participants were semi-randomised, because the sur-
vey link was sent out to random groups of people and some
were distributed with a snowball effect. Also, the researchers
distributed the link to friends and family through a personal
Even though all our groups were around n = 30, we still
investigated if our groups were normally distributed. We
executed the Shapiro-Wilk test. It appeared that the data
in our four groups were normally distributed. The scores
are as follows: Chatbot 1 (.333), Chatbot 2 (.087), Chatbot
3 (.008), Chatbot 4 (.007). The three constructs appeared
to be reliable. The construct Usability consisted of 4 items
(α = .822), The construct Usefulness consisted of 4 items
(α = .883) and The construct Satisfaction consisted of 3
items, after removing 1 item (α = .818). The three con-
structs together formed the total user experience score. The
total user experience score consisted of 11 items and was
highly reliable (α = .934).
6
Blocking bankcard
Financial advice
User Experience Score

message on Facebook Messenger. This was the most practi- 5.5

cal manner, because to use the chatbot, the users needed to
have Facebook Messenger. The participants took part be-
cause they knew the researchers and wanted to help them.
Another reason is because the researchers filled in a survey
for the participants so that the participants took part the
survey from the researchers.
The qualitative data were collected through observations
and interviews. The user first listened tot the instructions
of the researcher. The tester was asked to think aloud while
doing the task. Then the user had to independently com-
plete the task (same chatbot task as in the survey). The
participant was observed by the researcher. Also, the re-
searcher made notes of the facial expressions, experiences
and utterances of the user. After executing the task, the
tester did a semi-structured interview with the researcher.
3.5 Data Analysis
We used SPSS to analyse the data. We used a two-way
ANOVA test to compare the four independent groups to
find an interaction effect of the two independent variables
on the dependent variable (UX score). Subsequently, we did
a two-way MANOVA test, to investigate the efffect of task
and personality on the three constructs (usability, usefulness
and satisfaction).
4. RESULTS
In this section, we describe the results of our research. We
start with presenting the quantitative results and thereafter
we present the qualitative results.
5
4.5
Unpersonalised and Personalised
Figure 4: Plot of effect of Task & Personalisation with re-
spect to the user experience of chatbots
A two-way ANOVA was conducted that examined the ef-
fect of tasks and personalisation on the user experience of a
chatbot.
There was no significant main effect of personalisation
with respect to the user experience, F (1, 117) = .052, p
= .820, ηp2 = .000445, observed power = .056. Therefore, we
have to accept our H1 0 and reject our H1 1.
However, there was a significant main effect of tasks with
respect to the user experience, F (1, 117 ) = 14.341, p <
.005, ηp2 = .109, observed power = .964. Therefore, we ac-
cept the H2 1 and we reject the H2 0.
A two-way MANOVA was conducted to examine the effect
of the tasks with respect to the three constructs of user ex-
perience (Usability, Usefulness and Satisfaction). See Table
5 for the descriptive statistics.

11
 Table 5: Means and Standard Deviations of the Usability, Usefulness and Satisfaction of Chatbots

Dependent Variable Personality Task n M SD

Usability Total No personalisation Blocking Bankcard 31 5.61 1.008
Financial Advice 28 5.05 1.573
Personalisation Blocking Bankcard 30 5.50 1.303
Financial Advice 32 4.70 1.559
Usefulness Total No personalisation Blocking Bankcard 31 5.63 0.970
Financial Advice 28 4.41 1.685
Personalisation Blocking Bankcard 30 5.72 1.183
Financial Advice 32 4.66 1.512
Satisfaction Total No personalisation Blocking Bankcard 31 4.97 1.367
Financial Advice 28 3.98 1.640
Personalisation Blocking Bankcard 30 4.67 1.141
Financial Advice 32 4.06 1.482

The dependent variable Task has a significant effect with
respect to usability, usefulness and satisfaction F(3, 115) =
8.532, p < .005, W ilks0 λ = .818, ηp2 = .182, observed power
= .993. In the test results of the between-subjects test, we
see that Task has the following effect scores on the three
constructs. We found that Usefulness has the biggest effect
F (1, 117) = 21.187, p <.0001, ηp2 = .153, observed power
= .995. The second biggest effect was by Satisfaction F (1,
117) = 9.580, p = .002, ηp2 = .076, observed power = .866.
The third biggest effect was by Usability F (1, 117) = 7.318,
p = .008, ηp2 = .059, observed power = .765.
There was no significant interaction effect (Task * Per-
sonalisation) with respect to the user experience, F (1, 117)
= .027, p = .869, ηp2 = .000233, observed power = .053.
Therefore, we can not accept our Hi nteraction.
4.2 Qualitative results
We obtained our qualitative data through observations
and semi-structured interviews. We divided the obtained
feedback from the users into the following categories: Us-
ability, Usefulness and Satisfaction. These categories are
being described below. In the discussion, the qualitative
results will be further discussed.
4.2.1 Usability, Usefulness & Satisfaction
In our prototype we let the users choose if they wanted to
use the quick reply or the text field. Half of the users started
immediately with typing. We saw that when personalisation
effects were added, users expected even more that they were
able to type everything they wanted. Some users mentioned
that they expected this possibility, since the chatbot spoke
to them in a personal manner.
The emotions that were observed by the researchers, were
mainly neutral. Some users showed frustration when the
chatbot did not understand their utterances. Some users
mentioned that the chatbot’s credibility plays an important
part in how the user perceives the chatbot and what kind
of emotions are associated with the chatbot. According to
these users, trust plays a big part. Some users said that it is
better to not say in the introductonairy text that the user is
interacting with a chatbot, while other users prefer to know
that they are talking to a chatbot beforehand.
A couple of users noticed the use of emojis in the chatbot
and reacted on it. Most people who noticed, were positive
about the emojis, tone of voice and showing empathy. How-
ever some users were more negative. One user even felt like
it was ’fake empathy’ and that he had no need for that, es-
pecially not in a situation of urgency. Some users found that
the personalisation aspects did not appear to play any role,
as may be derived from the fact that they did not note or
mentioned it. Most of the users found it strange that emoji’s
were combined with a certain formal way of addressing the
users (in Dutch we say ’U’ for formal events and ’Je’ for in-
formal events). This can be solved by changing ’U’ to ’Je’,
so that the tone of voice is not a mixture of informal and
formal contact. However, some users prefer that the chatbot
is very formal and addressed them with ’U’ and no emoji’s.
Mainly younger users preferred that the chatbot addressed
them with ’Je’ and emoji’s, however most users also men-
tioned not to overdo the emoji’s (maximally 2 emojis).
A beneficial factor for the satisfaction of this chatbot was
that this chatbot can be used remotely. Users that don’t
like to call were satisfied that they were able to perform this
task through a chatbot.
Most users found these chatbots useful. Contributing fac-
tors are that the goal of the bots were clear and they ’did
what they had to do’. For the chatbot to be relevant, the
chatbot has to tell or teach new things, be efficient and ef-
fective according to some users. Some user’s also mentioned
that the whole customer journey should be taken into ac-
count in the chatbot, for example when the chatbot is fin-
ished with blocking the bank card, it could ask if the user
immediately wants to apply for a new bank card.
5. DISCUSSION
In this study we found that personalisation does not have
a significant effect on the user experience of chatbots for sim-
ple and for complex tasks. There was a significant difference
between the tasks in the chatbot, where the simple task was
perceived more useful, with a better usability and a higher
user satisfaction. The following aspects were identified that
could have influenced the user’s opinions, such as: artificial
intelligence, the goal of the chatbot, visual and UI elements,
the way the text is presented, security issues and usefulness.
These aspects are discussed in the exploratory findings.
5.1 Exploratory Findings
These results are not directly linked to our research ques-
tion, however they are still important for chatbots in general
and its user experience.
5.1.1 Artificial Intelligence

12
 Artificial intelligence (AI) is an important aspect of chat-
bots. One of the problems when having an unintelligent
chatbot is that the chatbot does not react on user’s input.
Artificial Intelligence could also improve the satisfaction of
the users. In general, most users were unsatisfied about the
inability to use the free text input and they preferred to use
both quick replies and free text input. Even though some
users also admitted that quick replies prevent errors and are
quicker to use. Maybe by improving the quick replies, the
users do not feel the need to use the free text input, but by
showing the possibility doesn’t make them feel restricted.
5.1.2 Goal chatbot
Explaining beforehand to the user how he or she can inter-
act with the chatbot, and the goal of the chatbot, is impor-
tant. Some users had the expectation that the chatbot would
validate their user details. The chatbot should mention in
what kind of form or syntax certain details (e.g. birth date)
have to be filled in or natural language processing could be
implemented which extracts the relevant information from
the user’s input. Another way the chatbot should commu-
nicate with his users is for example in the bank card block
chatbot, the chatbot could say here how long it will take (or
visually show) to block the bank card.
5.1.3 Quick replies
Quick replies are text bubbles that the user can choose
to interact with the chatbot. In this way, the user does not
have to think what the possibilities of the chatbot are and
therefore the interaction and the user flow can be more fast
and error proof than having a text input field. Most users
understood and used the quick replies and thought it was
convenient for when they don’t want to read through the
whole website where they would get an information over-
load, or when they are looking for a certain person/expert.
However, some users mentioned that they didn’t like the re-
strictions and limitations of the quick replies and would love
to see that both options are available. It appeared that the
’reactance effect’ can also occur when limiting the choices of
interaction in chatbots (Brehm, 1966). Someone said that
it felt like he just had to go through a form or a fixed menu,
not like a conversation.
5.1.4 User Interface
A more visual problem that occured by using quick replies,
was that the buttons were sometimes overseen by the users.
This could be improved by making them more clear, or
putting them on a more prominent place in the screen (e.g.
below the introductionairy text bubble of the chatbot). Cur-
rently typing feels the most natural way of chat interaction
for users. Interacting with quick replies is a relatively new
way of interacting through chat. By time, or when the users
tried it once, using quick replies could feel more natural for
users. Also, when the user starts typing, the quick reply but-
tons disappear. Some users said they want to still be able
to use and see the quick reply buttons. Other users said
that a back button should be implemented when an error is
made. Either way, correcting errors should be made more
easy in this chatbot, otherwise the user has to start over the
whole scenario and that could lead to frustration. Speed and
clarity are also factors which influenced the satisfaction of
the users. Also, a return or redo button would be beneficial
for the user satisfaction. Some users were annoyed by text
13
balloons having a certain overlap, that should be prevented.
In general, the text balloons should not be too long. Most
users did not read all the text and a more visual way of com-
municating is being preferred. For example when blocking
the bank card, a green check could appear to show visually
that the bank card is blocked.
5.1.5 Information overload
Other observations were that sometimes the text blocks
are shown really fast after each other. Some users said they
would not read all the text at once. A possible solution is
to send a larger text in smaller bites, and preferably with a
bit of time in between, so that user can easily digest all the
information. Also after each text block, it is important to
show the user what the next logical step is, or why something
is happening. A couple of users got stuck in the prototype
because they felt like they arrived at the end.
5.1.6 Accessibility
Almost all users were able to complete the chatbot sce-
narios successfully. However, a couple of users said that
older people might have more difficulties with completing
these scenarios. Another user also had difficulty with read-
ing the small letters. Accessibility has to be taken into ac-
count when implementing chatbots.
5.1.7 Security
Security is important for most users especially when the
chatbot handles financial data. It is very simple for users to
block their bankcard, but it is also easy for others to block
their bankcard. User detail validation would also be highly
appreciated by users. Also, some users indicated they would
like to receive a clear confirmation of that the bankcard is
blocked e.g. notification or letter.
5.1.8 Usefulness
The usefulness of the chatbot also depends on the chat ap-
plication used. Some users did not have a Facebook account
and were therefore not able to use the chatbot. For those
users, this Facebook chatbot is not so useful. Some users
said that they would not use this kind of chatbot if it was
from Facebook, but they would use it if it was integrated
in the website or application from the bank, or in another
chat service (e.g. Whatsapp, Slack). Another possibility is
to login with the corresponding bank account when using
the chatbot.
Some users also mentioned that it would be useful for
the company that has such a chatbot, so the chatbots could
support for example the customer service department. Also,
when users are not able to call, it is convenient that there is
a chatbot as an alternative. One user mentioned that chat-
bots are more convenient than calling, because sometimes
people speak with with an indistinguishable accent and the
customer service is not able to help them. This problem
would not occur with a chatbot.
The chatbot would be even more useful if it would remem-
ber the user’s input and would use it in the future or when
referring the user to another expert, that the information
would be send too.
5.2 Difference of tasks
The difference of tasks could be explained by the following
aspects. From the interviews, it appeared that the first task
(blocking bank card) had a much clearer goal for the user
than the second task (financial advice). Also for the first
task, more users said that they would use it in real life.
This could be a reason why the first task was rated more
useful. For the second task, the users mentioned that the
goal and the flow were not always clear to them.
The first task was also shorter than the second task. We
see that when the task is shorter, less mistakes are being
made and when a mistake is made, it is easier to correct it.
This could have had impact on the usability score.
The first task also meets the expectations of the user more
and this resolves into a higher satisfaction rate. The users in
the second task were sometimes confused, made more errors
and were therefore more frustrated than users from the first
task. This could have had influence on the satisfaction.
5.3 Contribution
In this study, theoretical and methodological contribu-
tions were made. The theoretical contributions of this study
are two-fold. Firstly, an overview of the state-of-the-art of
chatbots is given. Secondly, the study gives insight into the
effect of personalisation.
The methodological contribution consists of a way to mea-
sure satisfaction by means of 3 questions with a Chronbach’s
α = .818. Also, a UX questionnaire for chatbots has been
made which consists of the chatbot’s usability, usefulness
and satisfaction scores combined (see Table 7 in the Ap-
pendix).
5.4 Limitations
Throughout this research, multiple limitations emerged.
We will discuss them below.
The platform that was used to create the chatbots, was
limiting in the functionalities (e.g. artificial intelligence) of
the chatbot and the ways to export this chatbot. Now, we
were only able to export it to Facebook Messenger, but a lot
of people do not use this application and we don’t want to
force people to use it.
Since user experience has not been strongly defined in
the literature, we focused on the most mentioned aspects of
user experience (usability, usefulness and satisfaction). It
is debatable if these three constructs are valid to measure
the total user experience. This is a decision that was made
based on time restrictions. To give a broader image of user
experience, other measurements should have been used in
addition (e.g. EEG). We tried to tackle this by using a
combination of quantitative and qualitative user testing.
The surveys were distributed through Facebook, for ex-
ample on facebook groups where it was possible to exchange
surveys. It is possible that some participants from these
groups not motivated to fill in the questionnaire correctly.
We had to take this into account when analysing the data.
However, the general reliability of our constructs was high
enough.
The limitation of the qualitative user testing was that the
users were all people working at Capgemini (an Informa-
tion Technology company). Everyone was technically liter-
ate and some even worked with chatbots before. Nonethe-
less, they gave important feedback on our user experience
aspects.
5.5 Future Work
Due to time and technical limitations, not all aspects of
14
chatbots were possible to research. In this section we discuss
some of the aspects that are interesting to be researched
further.
Artificial intelligence would enhance the user experience
of this chatbot. It would be interesting to replicate this
research when artificial intelligence is implemented.
Embodiment in chatbots is also an interesting field to be
explored. Especially how personalisation relates to embodi-
ment when applying that in chatbots. Besides embodiment,
other personalisation factors can be researched (e.g. hu-
mour). Another interesting aspect is personalisation of con-
versational interfaces which use speech as input. Another
interesting aspect to research is to improve the personalisa-
tion within chatbots by creating a user model of the user,
before the user uses the chatbot (e.g. the user could fill in a
personality questionnaire so that the chatbot adapts to the
user).
In our research we chose the three constructs of usability,
usefulness and satisfaction to measure user experience. How-
ever, there could be other constructs relevant to measure
user experience. Also, it would be interesting to research
chatbots with a different kind of UX framework or evalua-
tion method. Also, looking into measuring user engagement
of chatbots could be interesting.
Besides the financial domain, there are multiple other do-
mains that would be interesting to further investigate, such
as: the field of insurance, education, e-commerce, etc. Also,
within the financial domain, there are multiple other tasks
which are interesting to investigate more, such as: a mort-
gage chatbot or a chatbot that helps you to finance your
start-up.
In this study, we mainly user-tested technical literate peo-
ple. For future work, testing on people outside of the con-
venient sample size would be interesting, e.g. the elderly or
non-technical people.
6. CONCLUSION
The popularity of using chatbot’s is increasing, however
when applying a new technology like chatbots, it is impor-
tant to do this effectively. A way to measure effectivity, is
to investigate the user experience. Therefore we researched
the effect of personalisation on the user experience of chat-
bots. The research question of this study is “What is the
added value of personalisation for the User Experience of
chatbots?”. The corresponding hypotheses are:
H1 0: Personalisation has no effect on the User
Experience of chatbots.
H1 1: Personalisation has a positive effect on the
User Experience of chatbots.
H2 0: The User Experience is the same for simple
and complex tasks.
H2 1: The User Experience is different for simple
and complex tasks.
Hinteraction: Personalisation and Tasks have
an interaction effect.
After conducting a two-way ANOVA test, no significant
main effect was found between the two chatbots with re-
spect to personalisation. Therefore, we accepted our H1 0
and rejected our H1 1. This means that personalisation has
no significant effect on the user experience of chatbots.
 There was a significant main effect between the differ-
ent tasks with respect to the user experience. This means
that the user experience score for the first task (blocking
bankcard) was rated significantly higher than the second
task (financial advice). Therefore, we accepted the H2 1 and
we rejected the H2 0. When we investigated the effect of
task on the constructs of user experience (usefulness, usabil-
ity, satisfaction), we found that the biggest effect of task was
by Usefulness, then Satisfaction and then Usability.
No significant interaction effect was found (Task * Person-
alisation) with respect to the user experience of chatbots.
Therefore, we have to reject our Hi nteraction.
Additionally, we can conclude that our satisfaction con-
struct was highly reliable.
We can conclude that a simple task with a clear goal
(problem-based) receives a significantly higher user experi-
ence score than a more complex task with a more uncertain
outcome (opportunity-based) in a chatbot. Lastly, we can
conclude that we created a brief, but reliable, questionnaire
to measure the user experience of chatbots.
Acknowledgements
First of all, I want to thank Jacobijn Sandberg from the
University of Amsterdam and Wouter Slager from Capgem-
ini for this opportunity and the patience, advice and help
they have given me.
Secondly, I want to thank my family and friends for sup-
porting me throughout the process of conducting this study.
References
Abashev, A., Grigoryev, R., Grigorian, K., & Boyko, V.
(2016). Programming tools for messenger-based chat-
bot system organization: Implication for outpatient
and translational medicines. BioNanoScience, 1–5.
Adformatie. (2017). Slimme chatbot en web-
site zetten eva jinek als journalistiek merk
op de kaart. Retrieved June 24, 2017, from
http://www.adformatie.nl/nieuws/slimme-
chatbot-en-website-zetten-eva-jinek-als-
journalistiek-merk-op-de-kaart
Amoroso, D. L., & Reinig, B. A. (2004). Personalization
management systems. In System sciences, 2004. pro-
ceedings of the 37th annual hawaii international con-
ference on (pp. 1–pp).
ASR. (2017). Retrieved May 5, 2017, from www.asr.nl
Bang, J., Noh, H., Kim, Y., & Lee, G. G. (2015). Example-
based chat-oriented dialogue system with personalized
long-term memory. In Big data and smart computing
(bigcomp), 2015 international conference on (pp. 238–
243).
Baron, N. S. (2015). Shall we talk? conversing with humans
and robots. The Information Society, 31 (3), 257–264.
Brehm, J. W. (1966). A theory of psychological reactance.
Buhr, S. (2016). Retrieved April 4, 2017, from
https://techcrunch.com/2016/01/28/mit-
spinout-insurify-raises-2-million-to-replace-
human-insurance-agents-with-a-robot/
Capgemini. (2017). Retrieved June 26,
2017, from https://www.capgemini-
consulting.com/industries/financial-services9
Chai, J. Y., Budzikowska, M., Horvath, V., Nicolov, N.,
Kambhatla, N., & Zadrozny, W. (2001). Natural lan-
15
guage sales assistant-a web-based dialog system for on-
line sales. In Iaai (pp. 19–26).
Chatfuel. (2017). Retrieved March 14, 2017, from
https://dashboard.chatfuel.com (Platform to cre-
ate Chatbots)
Chin, J. P., Diehl, V. A., & Norman, K. L. (1988). Devel-
opment of an instrument measuring user satisfaction
of the human-computer interface. In Proceedings of
the sigchi conference on human factors in computing
systems (pp. 213–218).
Davis, F. D. (1985). A technology acceptance model for
empirically testing new end-user information systems:
Theory and results (Unpublished doctoral disserta-
tion). Massachusetts Institute of Technology.
Dawnbot. (2017). Retrieved May 15, 2017, from
http://www.dawnbot.nl/
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017).
Ux design innovation: Challenges for working with
machine learning as a design material. In Proceed-
ings of the 2017 chi conference on human factors in
computing systems (pp. 278–288).
Emerce. (2016). Eneco lanceert chatbot in face-
book messenger. Retrieved June 24, 2017, from
https://www.emerce.nl/nieuws/eneco-lanceert-
chatbot-facebook-messenger
Ergonomics of human-system interaction – Part 210:
Human-centred design for interactive systems (Stan-
dard). (2010, March). International Organization for
Standardization.
Fan, H., & Poole, M. S. (2006). What is personaliza-
tion? perspectives on the design and implementation
of personalization in information systems. Journal of
Organizational Computing and Electronic Commerce,
16 (3-4), 179–202.
Finstad, K. (2010). The usability metric for user experience.
Interacting with Computers, 22 (5), 323–327.
Goodfellow, J. (2017). Klm introduces emoji-led directions to
its messenger chatbot. Retrieved June 25, 2017, from
http://www.thedrum.com/news/2017/03/02/klm-
introduces-emoji-led-directions-its-
messenger-chatbot
Hung, V., Elvir, M., Gonzalez, A., & DeMara, R. (2009).
Towards a method for evaluating naturalness in con-
versational dialog systems. In Systems, man and cyber-
netics, 2009. smc 2009. ieee international conference
on (pp. 1236–1241).
Iacoboni, M. (2009). Mirroring people: The new science
of how we connect with others. Farrar, Straus and
Giroux.
Jee, C. (2017). Retrieved June 25, 2017,
from http://www.techworld.com/picture-
gallery/personal-tech/9-best-uses-of-
Chatbots-in-business-in-uk-3641500/
Kaushik, D., Jain, R., et al. (2014). Natural user inter-
faces: trend in virtual interaction. arXiv preprint
arXiv:1405.0101 .
Kaye, J. (2007). Evaluating experience-focused hci. In
Chi’07 extended abstracts on human factors in com-
puting systems (pp. 1661–1664).
Kerr, D. (2017). Retrieved April 12, 2017, from
https://www.cnet.com/news/facebook-messenger-
hits-1-2b-users-thats-a-lot-of-people/
Kim, Y., Bang, J., Choi, J., Ryu, S., Koo, S., & Lee, G. G.
 (2014). Acquisition and use of long-term memory for
personalized dialog systems. In International work-
shop on multimodal analyses enabling artificial agents
in human-machine interaction (pp. 78–87).
Kiseleva, J., Williams, K., Hassan Awadallah, A., Crook,
A. C., Zitouni, I., & Anastasakos, T. (2016). Predict-
ing user satisfaction with intelligent assistants. In Pro-
ceedings of the 39th international acm sigir conference
on research and development in information retrieval
(pp. 45–54).
Knight, W. (2016). 10 breakthrough technologies: Conver-
sational interfaces. Retrieved June 24, 2017, from
https://www.technologyreview.com/s/600766/10-
breakthrough-technologies-2016-
conversational-interfaces/
Leite, I., Pereira, A., Mascarenhas, S., Martinho, C., Prada,
R., & Paiva, A. (2013). The influence of empathy
in human–robot relations. International journal of
human-computer studies, 71 (3), 250–260.
Lemonade. (2017). Retrieved April 4, 2017, from
https://www.lemonade.com/
Letzter, R. (2016). Ibm’s brilliant ai just helped teach a
grad-level college course. Retrieved June 24, 2017,
from http://uk.businessinsider.com/watson-
ai-became-a-teaching-assistant-2016-
5?international=truer=UK&IR=T
Lewis, J. R. (2013). Critical review of ‘the usability metric
for user experience’. Interacting with computers.
Lipiec, M. (2017). Retrieved April 14, 2017, from
https://chatbotsmagazine.com/what-we-learned-
designing-a-chatbot-for-banking-2dd2c51d7c2c
MarutiTechlabs. (2017). Retrieved June 24, 2017, from
https://www.facebook.com/Customer-Support-
Bot-1857341381220252/
Mayer, R. E. (2003). The promise of multimedia learning:
using the same instructional design methods across dif-
ferent media. Learning and instruction, 13 (2), 125–
139.
McCrae, R. R., & John, O. P. (1992). An introduction to
the five-factor model and its applications. Journal of
personality, 60 (2), 175–215.
McTear, M., Callejas, Z., & Griol, D. (2016). The conver-
sational interface. Springer.
Miller, H., Thebault-Spieker, J., Chang, S., Johnson, I.,
Terveen, L., & Hecht, B. (2016). “blissfully happy”
or “ready to fight”: Varying interpretations of emoji.
Proceedings of ICWSM , 2016 .
Mindbowser. (2017). Retrieved April 12, 2017, from
http://mindbowser.com/chatbot-market-survey-
2017/ (300+ individuals from a wide range of
industries participated in this survey. This study fo-
cuses on understanding the current state of Chatbots
and their future outlook.)
Munteanu, C., & Boldea, M. (2000). Mdwoz: A wizard of oz
environment for dialog systems development. In Lrec.
Niki. (2017). Retrieved June 5, 2017, from
https://niki.ai/ (digital chat-based artificial in-
telligence powered shopping assistant)
O’Brien, G. (2017). Retrieved June 26, 2017,
from https://tutorials.botsfloor.com/the-user-
experience-of-creating-a-chatbot-1f9055496349
Plutchik, R. (2003). Emotions and life: Perspectives from
16
psychology, biology, and evolution. American Psycho-
logical Association.
Postma, O. J., & Brokke, M. (2002). Personalisation in
practice: The proven effects of personalisation. Jour-
nal of Database Marketing & Customer Strategy Man-
agement, 9 (2), 137–142.
Press, G. (2016). Retrieved June 27, 2017, from
https://www.forbes.com/sites/gilpress/2016/11/
01/forrester-predicts-investment-in-
artificial-intelligence-will-grow-300-in-
2017/691466b35509
Quarteroni, S., & Manandhar, S. (2009). Designing an inter-
active open-domain question answering system. Natu-
ral Language Engineering, 15 (01), 73–95.
Rodden, K., Hutchinson, H., & Fu, X. (2010). Measuring the
user experience on a large scale: user-centered metrics
for web applications. In Proceedings of the sigchi con-
ference on human factors in computing systems (pp.
2395–2398).
Sauro, J., & Lewis, J. R. (2016). Quantifying the user expe-
rience: Practical statistics for user research. Morgan
Kaufmann.
Shaikh, A., Phalke, G., Patil, P., Bhosale, S., & Raghatwan,
J. (2016). A survey on chatbot conversational systems.
International Journal of Engineering Science, 3117 .
Shawar, A., Atwell, E., & Roberts, A. (2005). Faqchat as
in information retrieval system. In Human language
technologies as a challenge for computer science and
linguistics: Proceedings of the 2nd language and tech-
nology conference (pp. 274–278).
Smiers, L. (2017). How can chatbots meet expectations?
introducing the bot maturity model. Retrieved May 5,
2017, from http://www.content-loop.com/how-can-
chatbots-meet-expectations-introducing-the-
bot-maturity-model/
Someren, M. v., Barnard, Y. F., Sandberg, J. A., et al.
(1994). The think aloud method: a practical approach
to modelling cognitive processes. Academic Press.
Tatai, G., Csordás, A., Kiss, Á., Szaló, A., & Laufer, L.
(2003). Happy chatbot, happy user. In International
workshop on intelligent virtual agents (pp. 5–12).
Walker, M. A., Litman, D. J., Kamm, C. A., & Abella, A.
(1997). Paradise: A framework for evaluating spoken
dialogue agents. In Proceedings of the eighth confer-
ence on european chapter of the association for com-
putational linguistics (pp. 271–280).
Wigdor, D., & Wixon, D. (2011). Brave nui world: designing
natural user interfaces for touch and gesture. Elsevier.
APPENDIX
A. TASK ANALYSIS

Figure 5: Task analysis of three Dutch banks

B. CONVERSATIONAL USER FLOWS

17
Figure 6: Conversational user flow: blocking bankcard

18
Figure 7: Conversational user flow: financial advice

19
Link chatbot 1: m.me/1522158034481823
Link chatbot 2: m.me/1905225986423545
Link chatbot 3: m.me/272303553179105
Link chatbot 4: m.me/127510844467523 .

C. SCREENSHOTS CHATBOT 2

Figure 10: Screenshot of Blocking Bank Card Chatbot

Figure 8: Screenshot of Blocking Bank Card Chatbot

Figure 9: Screenshot of Blocking Bank Card Chatbot

Figure 11: Screenshot of Blocking Bank Card Chatbot

20
D. CHATBOT METRICS
Table 6: Chatbot Metrics (Hung et al., 2009)

Metric Type Data Collection Method

Total elapsed time Efficiency Quantitative Analysis
Total number of user/system turns Efficiency Quantitative Analysis
Total number of system turns Efficiency Quantitative Analysis
Total number of turns per task Efficiency Quantitative Analysis
Total elapsed time per turn Efficiency Quantitative Analysis
Number of re-prompts Qualitative Quantitative Analysis
Number of user barge-ins Qualitative Quantitative Analysis
Number of inappropriate system responses Qualitative Quantitative Analysis
Concept accuracy Qualitative Quantitative Analysis
Turn correction ratio Qualitative Quantitative Analysis
Ease of usage Qualitative Questionnaire
Clarity Qualitative Questionnaire
Naturalness Qualitative Questionnaire
Friendliness Qualitative Questionnaire
Robustness regarding misunderstandings Qualitative Questionnaire
Willingness to use system again Qualitative Questionnaire

E. QUESTIONNAIRE
Table 7: Questions from the self-made questionnaire where + means positively asked questions and - means negatively asked
questions.

Question Type -/+ questions

What is your name? Demographics
What is your age? Demographics
What is your education? Demographics
How much do you use a computer? Demographics
How confident do you feel about your computer usage? Demographics
Have you ever used a chatbot before? Demographics
The possibilities of this chatbot meet my requirements. Usability +
Using this chatbot is a frustrating experience. Usability -
This chatbot is easy to use. Usability +
I waste to much time on correcting things in this chatbot. Usability -
Because of this chatbot, I can quickly execute this task. Usefulness +
This chatbot makes it hard to execute the task. Usefulness -
Because of this chatbot, I can effectively execute this task. Usefulness +
This chatbot is useless. Usefulness -
This chatbot is fun to use. Satisfaction +
I miss functionalities in this chatbot. Satisfaction -
I would recommend this chatbot to a friend. Satisfaction +
I am unsatisfied about this chatbot. Satisfaction -

21
 F. CODING SCHEME

Figure 12: Coding scheme used to make notes of the user experience during the qualitative user tests

22

View publication stats