Вы находитесь на странице: 1из 11

White Paper

The What and How of Digital

Sponsored by

The value of experiences is have out-innovated and out-executed all competition in the
increasing. Consumer behavior has digital domain. Harvard Business Review previously outlined
fundamentally changed, becoming the essential need for digital experimentation in their 2017
progressively digital and mobile- article “The Surprising Power of Online Experiments.”
centric. Unsurprisingly, technology
and content have taken center The aforementioned companies have achieved this state of
stage and are fundamental to how digital competence and excellence through continued focus
CARL TSUKAHARA brands connect and enhance the and investment in their experimentation programs. We
CHIEF MARKETING customer experience. Simply using recommend that in order to achieve similar results, every
historical data to make decisions organization must consider how to:
about new digital experiences has • Set up and define the core business outcomes desired by
diminishing returns in a world where your overall digital programs or transformation initiatives.
customer expectations and behaviors
are changing more rapidly. Brands must build and deploy • Apply experimentation at pace to the full life cycle of a
experiences that are forward-thinking and differentiated digital experience, implementing rapid ideation, testing,
and can directly drive improvements in measurable business and learning at every stage.
objectives, customer loyalty, and conversion. • Utilize an enterprise-caliber experimentation software
platform that enables massive scale and helps grow
Critical to this is the concept of digital experimentation, digital maturity.
or more simply, the concept of rapidly innovating, testing,
and learning which digital product features, campaigns, or • Embed this skill set in the development and deployment of
commerce experiences will predictably drive the desired products, commerce, and campaign initiatives across both
business outcome. Experimentation guides the proper the product and marketing teams.
product and marketing decisions by providing statistical • Engage with the proper partners that can help companies
evidence and highly reliable outcome forecasting that implement technology components, strategies, and
eliminate the guesswork in digital decision making. organizational structure to evolve or expand this evolution
to an experimentation-led culture.
Leading companies apply digital experimentation to get—
and stay—ahead of the competition. European media giant Organizations big or small must look to experimentation to
Sky notes that frequent testing is necessary to keep up with drive continuous iteration and innovation across all areas
constantly evolving customer expectations. Data gleaned of their digital business. To keep pace with accelerating
from frequent iteration has also guided executive decision digital transformation, companies must act now to reap
making on omnichannel innovation at HP, a cultural shift from the handsome rewards that hundreds, even thousands, of
reliance on experience and intuition. The experimentation incremental improvements can bring to the bottom line.
program at Gap Inc. has helped them address strategic
questions throughout their brand portfolio and grow their To learn more about how Optimizely can help you achieve
online business, while the Wall Street Journal is boosting digital leadership, please visit www.optimizely.com.
subscription renewals through testing.

Done correctly, the experimentation examples outlined in this

paper can have a massive, cumulative effect on the business
performance of any traditional or emerging company. To
take this a step further, this discipline is no longer a “nice to
have”— it’s a competitive “must have.” Through large-scale
experimentation programs, giants such as Amazon and Netflix
The What and How of Digital Competitiveness

Why experimentation is more than testing

Many organizations are scrambling to boost their analytics capabilities and digitally
transform their businesses. They are also under intense pressure to improve their ENTERPRISE-WIDE EXPERIMENTATION:
innovation acumen in the quest to find the next breakthrough idea before anyone Test and learn how the market
else does. will react to new product features,
But these aren’t the only paths to digital competitiveness. They can even cause marketing messages, and user
companies to miss the boat. Analytics, for example, focuses on past history, and experiences and gather statistical
even the most sophisticated predictive capabilities can miss market changes evidence to make decisions.
happening right now. Although breakthrough ideas are powerful, they are
exceptionally difficult to come up with and bring to market. Enter enterprise-wide
experimentation programs—where companies test and learn how the market will
react to new product features, marketing messages, and user experiences and
gather statistical evidence to make decisions.
Experimentation is emerging as the next major digital capability that will
separate business winners from losers. When done at scale, an enterprise-wide
experimentation program can increase performance up to fivefold in revenue, share
of wallet, prospect conversion, risk mitigation, or product deployment efficiency.
Even small improvements can add up quickly and beat competitors to the punch.
Some companies have been at it for a long time. Microsoft, Amazon, and Facebook,
for example, conduct 10,000 or more experiments annually, according to Stefan
Thomke, a professor at Harvard Business School and an expert on experimentation
and optimization. However, digital experimentation isn’t the domain of only
digitally native companies. A recent MIT Sloan Management Review1 study
found that digitally maturing entities across all industries are more likely than
their less-mature peers to experiment at the enterprise level. Enterprise-wide
experimentation is, in fact, a mark of digital maturity. FIGURE 1
As products, services, and customer experiences move online, businesses have
broad new vistas in which to experiment with millions of customers. And they can
do it quickly and frequently. Moreover, this capability can identify the most current
market changes in near-real time—a major challenge for most companies.

Experimentation: The What and How of Digital Competitiveness Harvard Business Review Analytic Services 1
10 20 30 40 50 60 70 80 90 100

Rita Gunther McGrath, a professor
IMPLEMENTATION OF EXPERIMENTS at Columbia University’s business
school and creator of Discovery-
When my organization implements digital initiatives, they tend to start as:
Driven Planning, stresses that the
• MATURING speed of business and technology’s
ability to eliminate or reduce friction
Mostly small experiments in the market is forcing companies
58% to experiment aggressively and
quickly. “The typical cycles of design,
developing requirements, and creating
Mostly big enterprise-wide experiments technical specifications just takes
8% too long,” she says. “The gradual
disappearance of so many frictions
in business is driving much of the
Both small experiments and big enterprise-wide efforts intense speed.” As an example, she
20% points to audio books, which are
becoming more popular than the
written versions. Before the Internet,
Charts do not equal 100% due to “do not know” responses. people had to carry around a device
with cassettes or disks. Now they can
stream books in seconds anywhere.
Ryan McManus, investor, startup
advisor, board director, and previously
founder of Accenture’s digital strategy
Enterprise-wide experimentation also
transformation business—points
bolsters innovation by reaching out
to several other seminal market
across the organization to generate as
changes that call for enterprise-
many ideas as possible. As employees
wide experimentation programs.
see their ideas put to the test and
First, products are becoming
produce revenue, the momentum
quickly commoditized. To keep
builds. This original research report,
their customers, businesses have to
based on interviews with executives
constantly improve digital products
and other subject matter experts,
and experiences. As Jeff Bezos,
delves into the rise of experimentation
founder of Amazon, Inc., said in a
as the next major digital corporate
recent earnings call with investors,
capability and how companies are
“One thing I love about customers is
making it work.
that they are divinely discontent. Their
expectations are never static—they go
Why Experiment in Overdrive? up. It’s human nature.”2
Testing an idea—building a prototype,
Waning customer loyalty adds to the
running direct mail tests, conducting
pressure as clientele are easily swayed
focus groups and user tests—has
to opt for expensive products or go
been standard procedure for decades.
for a better deal, including products
But in recent years, enterprise-wide
enhanced with new digital capabilities.
experimentation has burgeoned into
Customer expectations are advancing
a core capability that drives digital
at breakneck speeds, and most
competitiveness. With the pervasive
companies have inadequate sensors to
digitalization of business, companies
spot changes in time to react. “What
can quickly and accurately test their
was right a year ago when you were
web site experiences, back-end
planning is not necessarily right today,”
operational effectiveness, product
says Abdul Mullick, head of digital
2016 offers, and marketing
transformation at European media and
telecommunications company Sky.

2 Harvard Business Review Analytic Services Experimentation: The What and How of Digital Competitiveness
The rise of approaches such as agile
is another harbinger of the growing
“What was right a year ago when you were
importance of experimentation. “Many
companies are moving toward an agile
planning is not necessarily right today,”
enterprise and changing the whole way
that they think about making changes
says Abdul Mullick, head of digital
to everything they do,” says Paul
Wilmott, a senior partner at McKinsey.
transformation at Sky.
“They are moving away from
traditional project timelines toward the
speed that agile provides. And that is
arguably all about experimentation.”

Experimentation programs can also

The Impact at Scale address thorny strategic questions.
The number of companies where
Gap Inc.—which owns Gap, Banana
enterprise-wide experimentation
Republic, Old Navy, Athleta, and
is having a significant impact on
Intermix—was mired in a common
performance is steadily climbing.
debate: Can the company encourage
Businesses are using it to develop
customers to shop across all its
rapid-fire improvements in everything
brands from each web site without
from digital product features and
cannibalizing sales? Noam Paransky,
commerce to overall business
senior vice president of digital, found
outcomes, including revenue, prospect
out that it could be quite profitable.
conversion rates, and profitability.
“We experimented with the
Microsoft’s Bing search engine, for prominence of the brand logos across
example, added an eye-opening the header of our website and found
$50 million in annual revenue by that by just making them 5% larger,
experimenting with different ad sizes. the website drove significantly more
Advertisers wanted larger ads. But referral traffic to all the sites,” he
these take up more web real estate and says. “We know that customers who
can also decrease user satisfaction. shop more than one of our brands
So the company ran experiments have 10 times the value of a ‘one and
testing larger ad sizes while not done’ customer, and that multiplier
increasing the amount of space increases significantly over their
devoted to advertising. The result of lifetime.” Focus on new experiences
showing fewer but larger ads: hefty online has had strong results; fourth-
contributions to annual revenue. quarter online sales alone leapt by
30% compared to the previous year.
Amazon added tens of millions of
dollars annually to its bottom line by Results such as the above don’t
experimenting with the placement come from a few experiments
of credit card offers on its web site. here and there. According to
Through a series of experiments, Thomke, companies with robust
Amazon found that many more experimentation programs
customers take up a credit card offer if get better hit rates—Microsoft
it is on the shopping cart page versus averages about 30%, according to
the home page. Thomke—while companies that are
limiting themselves to a handful of
Since it began a formal experimentation
experiments face very small odds of
program in 2015, Dow Jones has been
success. These companies also face
able to boost the new order conversion
even greater odds of being trounced by
rate for the Wall Street Journal by 64%
a competitor with a more aggressive
through online tests. According to Peter
experimentation program.
Gray, who heads the experimentation
program for subscriptions and
memberships, the revenue impact has
been in the millions.

Experimentation: The What and How of Digital Competitiveness Harvard Business Review Analytic Services 3
“CEOs need to coach company leaders and inspire them to
encourage new ideas and experiments and assure that results are
promptly turned into action,” says Professor Rita Gunther-McGrath,
Columbia University.

Yahoo! and Google are prime

examples, according to Thomke.
What Does Good Look Like?
Enterprise-wide experimentation
Yahoo! wanted to challenge Google’s
programs do more than collect reliable
leading position in search. Google had
and accurate data for decisions.
a formal experimentation program,
By-products of the effort include
and Yahoo! established one as well.
overcoming debates about the
Although Yahoo! developed some
accuracy of data and improving the
compelling innovations, Google
organization’s innovation culture.
simply outran them. Google simply
outran its competitor.
Ending Data Debates
One of the biggest challenges
The Power of Incremental companies face in using data for
Improvement decisions and innovation is the
Companies that are successful with frequent disagreement about what
enterprise-wide experimentation the numbers mean and what’s behind
conduct experiments on a very large them. Experimentation can supply
scale. Global entities with established a healthy stream of very clear and
programs may conduct 10,000 or more reliable market data if experiments are
experiments a year. But even nascent conducted properly and tied directly to
programs run several hundred. The measurable business outcomes.
results of these experiments are often
Some experts lament that much
small. But because of their sheer
experimentation in business and other
volume, they can lead to dramatic
fields does not consist of controlled
performance improvements.
experiments. The usual process
Experts often complain that takes data, looks for correlations
companies are locked into incremental between variables, and surmises that
improvements that don’t move the correlations indicate that one variable
needle. But not all companies need caused change in another. Sometimes
to or should rely on blockbuster the number of correlations is
new ideas. overwhelming and convincing, such as
the relationship between smoking and
As an example, one expert describes
various diseases. But more often than
a company that conducts 1,000
not, correlations can be misleading.
experiments per year. If an idea
meets muster every three days and
provides as little as a 1% annual
improvement, the company will double
its performance in only one year. And
it will also have a powerful view of
the market that can help it repeatedly
move the needle.

4 Harvard Business Review Analytic Services Experimentation: The What and How of Digital Competitiveness
For example, medical science has The CEO should step center stage
suffered embarrassment based on to drive the effort. It is his or
uncontrolled studies—basically her responsibility to champion
observational studies of an entire experimentation, and the CEO should
population over time—that made frequently ask about what experiments
inaccurate assumptions about are being conducted and champion
causality. For example, eating eggs was the successes. CEOs also need to
once frowned upon because there was build buy-in on the part of other
a correlation between egg consumption senior executives whose roles in the
and high cholesterol. But scientists organization will change. “Executive
later discovered that avoiding eggs had leadership devolves from its traditional
little or no positive effect on cholesterol role,” says McGrath. “They no longer
levels. If the body’s cholesterol goes too make all the decisions. The data
low, it manufactures more to reach a assumes that role. CEOs need to coach
basic level. company leaders and inspire them to
encourage new ideas and experiments
There are also plenty of examples in
and assure that results are promptly
business. A major retailer, for example,
turned into action.”
wanted to test the impact of a new
store design. So it tried the new ideas in Sometimes experimentation will
a number of locations, and the results be warmly received by executives
were strong. But when the company if experiments are conducted with
rolled out the new design to more rigor and the surrounding data is
of its stores, the results were much reliable. “Before we launched our
lower. The retailer failed to account for global experimentation program,
different variables affecting different executives had very little data and had
locations: competitors opening new to rely on their own preferences and
stores nearby, factories or other sense of intuition,” says Herriot Stobo,
major employers moving, and the director of omnichannel innovation
corresponding impact on shopping. and solutions at HP Inc. “They were
actually happy to have definitive data
Controlled experiments eliminate
and were ultimately very open to
the problems of not knowing all the
changing direction based on what our
variables at play and confusion about
customers were telling us.”
what the numbers mean. They divide
a test population into two different Pontus Siren, a partner at the
groups. One is given the new idea. innovation consultancy Innosight,
The other group, which has the same points out that many executives may
characteristics as the first, is not shown be comfortable using data but have
or given the change. Thus, the results little experience in creating it. “An
can be nearly definitive, since the experiment is an effort to create unique
experiment measures any change in data that other companies don’t have,”
behavior among identical groups. he says. “Business leaders have to
create the data, which is a very different
A New Take on Creating a Culture exercise. They need to understand the
of Innovation basics of experimentation and how to
Although discovery is the main act on the results.”
objective, an enterprise-wide
experimentation program also changes
the culture and senior executive
involvement in innovation. The data
from experimentation replaces position
in the corporate hierarchy as the basis “AN EXPERIMENT IS AN EFFORT TO CREATE UNIQUE
for decisions. In addition, the need to
feed the experimentation program with
a steady flow of compelling ideas brings
practically everyone in the organization
into the innovation fold. CONSULTANCY INNOSIGHT.

Experimentation: The What and How of Digital Competitiveness Harvard Business Review Analytic Services 5
To feed the enterprise-wide A platform to manage multiple
experimentation program, experiments across the enterprise
organizations need to harness new is critical, says McManus.
ideas from across the enterprise. Enterprises need to know exactly
Experiments generate near-real–time what experiments are being run,
insights into market changes, which how they are managed, criteria for
spur the creation of even more new continuing with or disengaging from
innovations, especially as people see experiments, and how other changes
their ideas being taken seriously. At in the enterprise affect the innovation
Sky, for example, all ideas are put to the portfolio. Without that knowledge,
test unless they would be impractical to variables could change without the
implement or run afoul of regulations. experimenter realizing it, which will
produce “useless data,” says McManus.
Experimentation programs need
Platforms can also include tutorials
analytics professionals. In addition to
and templates for managers to conduct
designing experiments, they can help
some of their own experiments, which
the CEO and other executives build
can be monitored on the platform.
momentum by demonstrating to the
company what good experimentation Tying experimentation into overall
looks like and the results it can deliver. corporate goals and measures is
Enterprise-wide experimentation also critical according to Gap Inc.’s
often starts as a centralized unit, Paransky. For Gap Inc., enterprise-
with analytics professionals working wide experimentation is the vehicle
with marketers, usability specialists, that drives the company’s digital
product managers, and business transformation. “Experimentation
unit strategists. They can then grow is embedded in almost everything
into a center of excellence to make we do as a company,” he says. “We
experimentation a part of every have enterprise goals around digital
business unit and function. transformation that cascade through
the organization. Testing and learning
By and large, the skills of middle
drive our digital evolution and
managers don’t need to change
that much. But these managers
must become deeply familiar with They are also the anchor of the
technology and understand the basic company’s continuous learning
discipline of experimentation and process. “We’re constantly using
analytics. That allows them to work experimentation to drive our
effectively with analytics professionals continuous improvement,” he says.
or use platforms that provide “The key is that we’re focusing on the
self-guiding tools and templates. biggest wins, the biggest levers in terms
“Managers often enter experimentation of what we’re experimenting with.”
programs sleepwalking,” says Mullick.
“That can result in oceans of poorly
constructed experiments, which will
affect the results.”

“Experimentation is embedded in almost everything we do as a company.

We have enterprise goals around digital transformation that cascade through
the organization. Testing and learning drive our digital evolution and
transformation,” says Noam Paransky, senior vice president of digital, Gap Inc.

6 Harvard Business Review Analytic Services Experimentation: The What and How of Digital Competitiveness
Experimentation also plays a seminal
role in helping organizations spot and
keep pace with radical change in their
markets. According to a 2018 MIT Sloan
Management Review report,3 many
companies simply throw money at
technology disruption in the quest for
the next big idea. Experimentation
hones market knowledge and also
provides key insights for the success of
existing businesses.
Finally, organizations that are most
successful in leveraging enterprise-
wide experimentation programs have
conquered much of the fear of failure
Top Down or Bottom Up
that plagues many company cultures.
CEOs and senior executives often
The sheer volume of experiments
drive experimentation programs. Dow
makes it easy to show how each idea
Jones, Gap Inc., and HP Inc. started by
leads to another and reaches a positive
creating a centralized unit with senior
outcome. The team looking after Sky’s
managers, fostering collaboration
web-based self-serve platform, Sky.
between the unit and the rest of
com, leverages the volume of tests
the organization. Sky started with a
it does and its high success rate to
center of excellence and is planning
create a “safe zone,” where failures are
to federate the experimentation
celebrated and the stigma of coming
program. Executives are making
up with an idea that doesn’t pan out is
sure that managers understand the
significantly lessened.
fundamentals. They have also installed
a platform that will provide “self-serve”
Getting the Ball Rolling capabilities for basic experiments.
Despite the power of enterprise-
McGrath points out that bottom-up
wide experimentation as the primary
efforts can also be successful. She
competitive tool in a digital age,
cites an air company where the
most companies seem to struggle to
chief innovation officer built the
get out of the starting gate. To get
experimentation program from the
the ball rolling, McGrath suggests
ground up and under wraps. Senior
that companies focus on three core
executives began noticing that they
were receiving proposals with a similar
Ability to create ideas: Businesses format and based on experimentation
need people who can come up with data. The quality of the proposals
ideas. These individuals should prompted the C-Suite to find out how
come from all parts of the company, they were being created, and then it
especially those on the front line. launched an experimentation program
across the enterprise.
Incubation: Companies need
sophisticated experimentation
capabilities and the ability to act
on data.
Acceleration: Many ideas will have
operational implications for everything
from customer care to regulatory
compliance. Businesses need to be
adept at adjusting their operation
quickly and effectively.

Experimentation: The What and How of Digital Competitiveness Harvard Business Review Analytic Services 7
Conclusion Although managing by data changes
Controlled experiments are rising the role of executives and managers,
as the primary core capability of companies are achieving broad buy-in
digital competitiveness. Analytics are and creating innovative cultures.
important and foundational, but they As the results pour in, they build
are based on the past and may not credibility, and new ideas add to the
accurately identify what is changing company’s fortunes.
at this very moment. Customer
Amazon’s Jeff Bezos once extolled the
ENTERPRISE-WIDE research can be valuable but needs to
be validated.
importance of experimentation with
an analogy to baseball. In baseball,
EXPERIMENTATION CAN Dow Jones, for example, posed a an “at bat” can score four runs at

TURN INCREMENTAL simple research question on the

home page of the Wall Street Journal
the most (runners on all bases). “In
business, every once in a while you

IMPROVEMENTS INTO asking if readers wanted a print-only

offer or a package that also included
can score 1,000 runs. This long-tailed
distribution of returns is why it’s
DRAMATIC GROWTH. online access. To the company’s
astonishment, the vast majority
important to be bold. Big winners pay
for many experiments.”4
selected print only. But then Dow Jones
ran an actual test and found that four
times as many customers opted for
the print and digital package instead
of print alone. “Research can be very
powerful directionally,” says Dow
Jones’ Gray. “But your entire premise
or your interpretation of the research
could be in fact quite inaccurate.”

1 “Achieving Digital Maturity,” MIT Sloan Management Review research report, July 13, 2017
2 Amazon 2017 letter to shareholders, April 18, 2018
3 “Coming of Age Digitally,” MIT Sloan Management Review research report, June 5, 2018
4 Sonia Thompson, “Jeff Bezos Credits Amazon’s Success to This One Thing. Too Bad Most Companies
Aren’t Willing to Do It,” Inc., August 1, 2017

8 Harvard Business Review Analytic Services Experimentation: The What and How of Digital Competitiveness

hbranalyticservices@hbr.org Copyright © 2018 Harvard Business School Publishing.