Market Data / Supplier Selection /

Event Presentations / User Experience

Benchmarking / Best Practice /
Template Files / Trends and Innovation


SAMPLE: User
Experience Survey
Report 2013

In association with WhatUsersDo

SAMPLE: User
Experience
Survey Report
2013
In association with
WhatUsersDo

Published February 2013 Econsultancy London Econsultancy New York

4th Floor, Farringdon Point Ste. 307, 350 7th Avenue
29-35 Farringdon Road New York, NY 10001
London EC1M 3JF United States
All rights reserved. No part of this publication may be United Kingdom
reproduced or transmitted in any form or by any means,
electronic or mechanical, including photocopy, recording Telephone: Telephone:
or any information storage and retrieval system, without +44 (0)20 7269 1450 +1 212 699 3626
prior permission in writing from the publisher.
http://econsultancy.com
Copyright © Econsultancy.com Ltd 2013 help@econsultancy.com
Like this sample report?
Access over 500 reports
with Silver membership.
The reason members join us - and stay with us
Econsultancy’s entire research library is available to our members, providing unlimited and
unrivalled access to over 500 reports (and counting). In an environment where market trends are
constantly changing, gaining up to the minute insider knowledge and expertise is invaluable.

Comprehensive, authoritative and easy to read, our award-winning research offers practical advice
to marketers on all aspects of digital marketing and e-commerce.

Whatever you need to know, you just found it

Just a few of
Best Practice
 Our definitive ‘how-to’ guides across key topic
our reports...
areas for digital marketing professionals.
• SEO Best Practice Guide
Market Data
 Statistics, facts and figurees; great for
• Facebook Pages for Business
Best Practice Guide
presentations! • Online Video
Best Practice Guide
Supplier Selection
 Understand the market, latest trends and find • Global Internet Statistics
Compendium
the right supplier for your needs.
• RTB Buyer’s Guide

 Template Files
Save time on RFPs, web and digital marketing
• Email Marketing Platforms
Buyer’s Guide
projects.
• Digital Marketing
Trends and Innovation Template Files
 What’s happening out there, what’s new, • Innovation Report
what’s next? • Quarterly Digital
Intelligence Briefings

Find out more about research and

get access now with Silver membership

To view our full range of 500+ reports, please visit

econsultancy.com/reports

If you’re ready to get your hands on our reports and use them for
all they’re worth, we recommend becoming an Econsultancy Silver
member. For more details, please visit econsultancy.com/join

If you have any questions about membership, get in touch by

emailing membership@econsultancy.com
Contents

1. Executive summary and highlights ................................. 1

2. Foreword by WhatUsersDo ............................................. 4
2.1. About Econsultancy .................................................................. 5
2.2. About WhatUsersDo ................................................................. 6

3. Methodology .................................................................... 7
4. Findings ........................................................................... 8
4.1. Approach to user experience ..................................................... 8
4.1.1. Commitment ............................................................................... 8
4.1.2. Benefits of user experience ....................................................... 11
4.1.3. Rating of user experience .........................................................12
4.1.4. Internal organisation of user experience ................................14
4.1.5. Decision making and senior management understanding .... 17
4.1.6. Measuring user experience .......................................................21
4.1.7. Barriers to improving user experience ................................... 22
4.1.8. Opinions on user experience ................................................... 23
4.2. Testing .................................................................................... 28
4.2.1. Extent of testing ....................................................................... 28
4.2.2. Return on investment .............................................................. 30
4.2.3. Aspects and methods of testing............................................... 33
4.2.4. Objections to user experience testing ......................................41
4.2.5. Budgets for testing ................................................................... 42
4.3. Mobile devices......................................................................... 46

5. Appendix ........................................................................ 48

User Experience Survey Report 2013 In association with

WhatUsersDo

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage
and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2013
1. Executive summary and highlights
This is the first User Experience Survey Report, published by Econsultancy in association
with WhatUsersDo. The research is based on a survey of more than 1,400 digital and and
ecommerce professionals working for brands, agencies and specialist user experience firms.

The survey, which was conducted in November and December 2012, was promoted to
Econsultancy’s user base and also to WhatUsersDo clients.

The report looks in detail at the approaches, benefits, budgets and opinions of those involved in
user experience (UX) today, with a particular focus on user experience testing. It also looks to
understand internal and political issues which may be hindering the advancement of user
experience best practice within organisations.

This research identifies the following key trends:

Companies are enthusiastic about providing a quality user experience but

are yet to achieve best practice in testing and improvements

Although user experience is not a new topic within digital marketing and web design, the survey
reveals a growing appetite for improving UX across digital assets, particularly as consumers
embrace more digital touchpoints.

 Nearly eight out of 10 (78%) client-side respondents stated that their company
was ‘extremely’ or ‘quite’ committed to delivering the best online user
experience.
 In addition, 95% agreed with the statement that ‘good user experience just makes
sense’.
 93% believe that ‘optimising the user experience will improve conversions’.
Qualitative responses confirmed these results; when asked why their company was committed to
user experience, a major theme emerging was that a good user experience leads to “customer
satisfaction, brand loyalty, and increased sales”.
However, when asked about the quality of the user experience on their own digital properties,
only a minority of respondents (44%) rate their own companies’ performance in this area as
‘excellent’ or ‘good’.

SAMPLE: User Experience Survey Report 2013 In association with WhatUsersDo

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage
and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2013
2. Foreword by WhatUsersDo
I’m going to make an assumption about you.

I’m going to assume that you, like me, usually skip read the forewords of reports like these. My
hunch is that you find them (when written by a VP of this, or a Director of that) nothing more
than a subtle, and oftentimes not so subtle, plug for their companies that ticks all of the “on-
message” boxes.

And, while that’s not offensive and is clearly accepted practice, isn’t it all a bit too “corporate” and
“impersonal”?

You are, after all, a person. A person who, like the rest of us, is getting their head around a digital
landscape that’s changing faster than it ever has and becoming more complex. Grappling with
not only how you can improve user experience, but how you find the budget, get board buy-in,
embed it and prove the ROI (as evidenced in this report).

Or, you’re in the minority of organisations that have already cracked it. You’re already obsessing
over UX and the board don’t need convincing of its importance (they get that digital experience =
brand). Users are at the heart of your organisation and you’ve based your 2013 development
roadmap on user insight and not your own hunches or colleagues’ opinions.

However, judging from the results of this survey, that’s probably not you.

The report shows that the majority of organisations have a long way to go – even though it seems
the desire and appetite is there. Don’t get me wrong, I am delighted that WhatUsersDo (there’s
me ticking an “on-message” box) ranks in the top three most insightful methods (considering
we’ve only been trading for three years). But, so few organisations (barely a third) have embedded
UX testing in their processes and nearly half don’t conduct any UX testing at all.

What’s really telling in this report is the prevalence given to ubiquitous tools like Google Analytics
and content testing (e.g. MVT). If they truly offered the answers, then how come over 60% of
companies are still making improvements to user experience based upon their own hunches?

It comes down to a single word: “Why?”

As digital professionals we need to answer “Why?”. We need to understand the why of user and
customer behaviour so we can move beyond the incremental improvements that MVT ultimately
leads to and step beyond the how many of Google Analytics by applying user insight (and not
hunches) to our decision making.

That message, understanding the why of user behaviour, is a real opportunity for digital
professionals to act upon in 2013.

I’d like 2013 to be the year where we convince organisations that user experience = brand and
that real improvement and change can only really come about by understanding users.

Enjoy the report.

Lee Duddell
Founder
WhatUsersDo

SAMPLE: User Experience Survey Report 2013 In association with WhatUsersDo

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage
and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2013
2.1. About Econsultancy
Econsultancy is a global independent community-based publisher, focused on best practice digital
marketing and ecommerce, and used by over 400,000 internet professionals every month.

Our hub has 185,000+ subscribers worldwide from clients, agencies and suppliers alike with over
90% subscriber retention rate. We help our subscribers build their internal capabilities via a
combination of research reports and how-to guides, training and development, consultancy, face-
to-face conferences, forums and professional networking.

For the last ten years, our resources have helped subscribers learn, make better decisions, build
business cases, find the best suppliers, accelerate their careers and lead the way in best practice
and innovation.

Econsultancy has offices in London, New York, Sydney and Singapore and we are a leading
provider of digital marketing training and consultancy. We are providing consultancy and custom
training extensively across Europe, Asia and the US. We trained over 5,000 marketers and ran
over 200 public training courses in 2012.

Join Econsultancy today to learn what’s happening in digital marketing – and what works.

Call us to find out more on +44 (0)20 7269 1450 (London) or +1 212 971 0630 (New York). You
can also contact us online.

SAMPLE: User Experience Survey Report 2013 In association with WhatUsersDo

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage
and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2013
2.2. About WhatUsersDo
Founded three years ago by Head of UX, Lee Duddell, WhatUsersDo is an online User
Experience Testing Service. WhatUsersDo is for ecommerce, marketing, customer insight
and digital people who want to bring change to how they make digital decisions. We help you
remove your hunches and act with greater confidence through user driven actionable proof. Join
us in shaking off your internal focus with greater “user centric” insight. We urge you to improve
your digital assets with our “why” based analysis and insight that compliments your existing
“what” based analytics.

We are growing rapidly and now provide our services to over 1500 digital (ecommerce, research,
customer insight, mobile, marketing) professionals internationally. In the last three years our
collective mission has been to bring the “user’s voice” centrally to operating businesses wishing to
improve their digital channels.

Clients include SMEs, agencies and brands including: EasyJet, Andrews Aldridge, Lipsy, Ocado,
Tesco, Hobbs, Jam, LateRooms and Standard Life.

Our international testing platform makes it possible to watch real people completing tasks on
mobile, tablet or fixed websites thereby gaining actionable insights to improve.

Brands and agencies use our testing service to capture “in the moment” user feedback from our
international panel of "real people" or even their own customers (private panels). Our platform
records the screens and spoken thoughts of users into online videos as they complete tasks on our
clients’ digital channels. Insight from the resulting online videos reveals the why of user
behaviour so that our clients can make informed decisions on how to improve their digital
channels. Quickly and cost effectively. As standard, we test in nine countries with 26 others on
demand.

We provide two levels of service: ‘Pay As You Go’ and ‘Managed Service’. Our ‘Pay as You Go’
service is for SME clients who need occasional user insight. The ‘Managed Service’ is for
companies who are serious about making user insight ‘business as usual’.

Our ‘Managed Service’ will help you and your budget work more efficiently from day one of
signing up. From the outset we work with you to understand your objectives so you can bring the
user’s voice into your business and make decisions on digital channels with confidence and real
evidence from your users.

Our Managed Service includes everything from scoping, designing and setting up user tests for
you. We even provide expert UX analysis on the results which so you can start working with the
insights within days of going live instead of having to watch many hours of test footage yourself.
You can, of course, supplement our findings with your existing “what” and “how many” data
sources such as web stats and embed user insight into your digital design and development
processes.

We work on a rolling contract basis meaning there is no lengthy tie in period for either party but
you still benefit from a deep relationship and features like those outlined below.

 UX workshop to set testing objectives

 Testing on desktop as well as tablet and smartphone devices
 Test with our panel or use your own customers and recruit on demand for 26 countries
 Rapid expert analysis on the results that are turned around in seven working days
To find out more, contact Steve Porthouse: steven@whatusersdo.com / 0845 302 4783
www.whatusersdo.com

SAMPLE: User Experience Survey Report 2013 In association with WhatUsersDo

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage
and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2013
3. Methodology
This is the first User Experience Survey Report published by Econsultancy in association with
WhatUsersDo.

There were more than 1,400 respondents to our 2012 research request, which took the form of an
online survey carried out in November and December 2012. Respondents included client-side
professionals (including marketers and UX professionals), those working for agencies, and those
working for specialist agencies or as a UX consultant.

Information about the survey, including the link, was emailed to Econsultancy’s user base and
promoted on Twitter. The incentive for taking part was access to a complimentary copy of this
report just before its publication on the Econsultancy site.

Further information on the background of the respondents is included in the appendix at the end
of this report.

The main points of note are:

 52% of respondents said they were “part of an in-house team”, while 29% said they worked for
an agency (including full service or design and build) and 19% described themselves as
working for a “specialist user experience agency or [as a] consultant”.
 For in-house respondents, 49% stated their company was mostly focused on B2C commerce,
with 26% working in the B2B space. A quarter (25%) said their business was “equally focused”
on both B2C and B2B marketing.
 44% of in-house marketers stated they worked for a company with a turnover greater than
£50 million, while 35% gave their company turnover as less than £10 million.
 For in-house marketers, the best represented sectors were retail / mail order (22%), financial
services (12%), publishing (9%) and travel (7%).
Where relevant, percentages may not add up to 100% due to rounding.

If you have any questions about the research, please email Econsultancy’s Research Director,
Linus Gregoriadis (linus@econsultancy.com).

SAMPLE: User Experience Survey Report 2013 In association with WhatUsersDo

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage
and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2013
4. Findings
4.1. Approach to user experience
4.1.1. Commitment
Figure 1 below illustrates how committed companies are to delivering the best possible user
experience. The majority (73%) state that their organisation is either “extremely” or “quite”
committed. Just 8% answered that their organisation is “not committed” to delivering the best
possible user experience.

Company respondents
Figure 1: How committed is your organisation to delivering the best possible online
user experience?

Respondents: 602

What the experts say

“This level of commitment is a very positive sign for the whole field of UX. This very much reflects the trend
we’ve seen over the last few years; organisations no longer see activities such as UX testing (in all its guises) as a
nice-to-have or luxury service, but as a necessity for continually driving up conversion rates and keeping ahead of
the competition.”
Chris Gibbins, Head of User Experience, RedEye

SAMPLE: User Experience Survey Report 2013 In association with WhatUsersDo

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage
and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2013
 In order to explore why companies were or were not committed to delivering the best possible
user experience, respondents were asked to explain their answer 1.

In total, 468 of the respondents gave further details; the tag clouds illustrate the frequency of
words in the collated responses and a selection of quotes is provided.

For companies committed to user experience, the main theme of the responses was that focusing
on the customer through user experience led to better business performance, whether through
improved conversion, satisfaction, or brand loyalty.

For companies not committed to delivering the best possible user experience (as shown on the
following page), the main reasons given concerned cost and a lack of resource. However, common
themes also included political issues and a lack of awareness around the business benefits of
having an improved user experience.

Company respondents
Tag cloud: Why is your company committed to user experience?

What company respondents said – “Why is your company committed to user

experience?”
“Bad UX leads to poorer conversion, more confusion, leading to more queries and dissatisfaction.”

“Good UX leads to customer satisfaction, brand loyalty, and increased sales.”

“So we can strengthen the brand, improve customer satisfaction and improve sales and conversion.”

“We have a large website, with potential customers embarking on what can be a long information
search/collection period before making a decision to buy, so providing good user experience is paramount to the
success of our web presence. If users can't find their way around our site easily, they won't stick around to get the
information they need to inform their purchase decision.”

“User experience is akin to customer service. How a user is serviced online will have a direct impact on how the
customer views the brand and if they will ever come back.”

“A better user experience spreads a good ethos surrounding the brand. A better experience is more greatly shared
socially, whether online or offline. It increases conversion, sales, gives the brand and website a good reputation.
Ideally the customers come away from the website having experienced something highly positive that they will
recommend to friends and revisit.”

SAMPLE: User Experience Survey Report 2013 In association with WhatUsersDo

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage
and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2013
4.1.2. Benefits of user experience
Respondents were asked to list what they would describe as their three most important business
benefits of improved user experience (Figure 2). Answers from both client-side and agency
respondents were broadly similar.

Although increased sales / conversions was the most commonly cited business benefit, reducing
costs featured less, with only 16% of client-side respondents and 17% of agency respondents citing
reducing cost-per acquisition (CPA) as one of their three most important benefits. This seems to
reflect a lack of awareness as to how an improved user experience can positively affect the bottom
line by reducing costs as well as by increasing revenue.

Company and agency respondents

Figure 2: What do you view as the most important business benefits of improved
user experience?

Client-side respondents: 619

Agency respondents: 556
Note: respondents could select up to three options.

SAMPLE: User Experience Survey Report 2013 In association with WhatUsersDo

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage
and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2013
4.2. Testing
4.2.1. Extent of testing
Figure 16 shows how many companies and agency clients are currently conducting user
experience testing. Despite the high esteem in which user experience is held (see Figure 14 and
Figure 15), there are still a significant number of organisations which do not currently appear to
be doing any user experience testing (45% for in-house respondents, and 53% for agency clients).
However, the majority of companies not doing user experience testing now are planning to do so
in the next 12 months, indicating that there is an appetite for testing within companies.

Reasons why companies are not conducting user experience testing are shown in Figure 17. Much
like the barriers to improving user experience generally (Figure 13), budget is the most commonly
listed barrier to testing However, given the emergence of lower cost user experience testing
methods, and the fact that most marketers see the value in user experience improvement, this
issue is expected to diminish with time as the business case for improving UX becomes stronger.

Combined company and agency respondents

Figure 3: Do you/ your clients conduct any user experience testing?

Client-side respondents: 563

Agency respondents: 299
Note: respondents who described themselves as working for a “specialist user experience agency or [as a]
consultant” as detailed in the methodology were not included.

SAMPLE: User Experience Survey Report 2013 In association with WhatUsersDo

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage
and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2013
Company and agency respondents
Figure 4: Why don’t you/your clients carry out any user experience testing?

Client-side respondents: 246

Agency respondents: 168

SAMPLE: User Experience Survey Report 2013 In association with WhatUsersDo

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage
and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2013
4.2.2. Return on investment
For those companies who performed user experience testing (and those non-specialist agencies
whose clients did the same), the survey asked them to list the top three user experience testing
techniques that they perceived to provide the greatest return on investment. Their responses are
shown in Figure 18.

Despite its potential for higher costs, in-person / lab user experience testing was perceived to
offer the greatest return on investment, with around half of all respondents (49% on client-side,
51% at agencies) listing this as one of their top three techniques. The other two most commonly
cited top techniques were again the same for both client-side and agency respondents, namely
content testing (e.g. MVT, A/B) and remote user experience testing (e.g. WhatUsersDo).

The two techniques where there was the greatest difference in opinion between agencies and
companies were expert reviews (26% agency vs 18% client-side) and guerrilla user testing (21%
agency vs 13% client-side).

Combined company and agency respondents

Figure 5: Please select the top three user experience testing techniques that you
perceive to provide the best ROI (even if you have not used any of them)

Client-side respondents: 546

Agency respondents: 455
Note: respondents could select up to three options.

SAMPLE: User Experience Survey Report 2013 In association with WhatUsersDo

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage
and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2013
5. Appendix
The charts in this appendix detail the profile of the respondents who took part in this survey.
Details on how the survey was promoted to respondents are described in the Methodology section
of this report.

All respondents
Figure 6: Which of the following most accurately reflects your role?

Respondents: 1,434

SAMPLE: User Experience Survey Report 2013 In association with WhatUsersDo

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage
and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2013