DIGITAL CHANGE AGENTS

Lessons, Tactics,
and Stories from
World-Class
Experimenters

The marketer’s role is changing. And it’s
Prefac e
changing fast. The same can be said for product
managers. In an increasingly digital world
where competition is fierce, it’s not enough to
simply rely on our hunches to drive our product
Companies that
and marketing decisions. It’s high time we
continuously test new
replace digital guesswork with evidence-based
messaging, features,
optimization. Our success depends on it.
and user experiences
will achieve the best
business outcomes Companies that continuously test new
and outperform their messaging, features, and user experiences
competitors. will achieve the best business outcomes and
outperform their competitors. You may already
be familiar with the concept of A/B testing
– comparing two separate approaches to
determine the optimal result. Digital experience
optimization takes this concept to an entirely new
level. Organizations today use experimentation
to transform themselves into businesses that
continue to learn, continue to improve, and
continue to succeed. Simply put, because they
out-experiment, they outperform.
2
In the past, only a few companies — Amazon,
Facebook, and Google — made digital
experience optimization a fundamental business
practice, launching tens of thousands of
experiments each year. Fast forward to 2018,
and experimentation is quickly becoming the
new normal for digital businesses. From digital-
first companies like Atlassian and Box, to global
industry giants like Visa and Sky Media, there
is a widespread urgency to build out dedicated
digital experience optimization teams.
In this ebook, we will take you behind the
scenes to see how both the disruptors and
the transformers are building out their digital
experience optimization teams. You’ll discover
how they approached and ultimately resolved
their most pressing product development and
marketing questions. So whether you’re an A/B
testing novice, or an experienced experimenter,
this informative ebook is certain to put you on
the path to becoming a digital change agent in
your own organization.
 3

Tom Tsao is Atlassian’s Head of eCommerce. constant. Every one of them was thinking
In t ro
To him, the benefit of experimentation is plainly well beyond basic call-to-action button
apparent. “It’s simple,” Tsao says, “Experimentation color or landing page copy tests. They were
lets us make better business decisions.” Mr. considering critical business questions and
Tsao — an industry veteran who’s helped guide aligning their experimentation programs to
It’s not about
companies such as Netscape, Yahoo, and address those concerns, to establish buy-in,
button colors. It’s
Walmart — confesses that in the past decisions and to demonstrate the impact of their efforts.
about aligning
were often made based on the HIPPO (or highest In this ebook, we will take a closer look at
experimentation with
key business initiatives. paid person’s opinion) in the room. That seldom the novel ways each of these change agents
happens at Atlassian these days because successfully integrated experimentation into
experimentation now gives his team a framework their businesses.
to evaluate every idea as it arises. Today, they
only implement the ones proven impactful by
concrete data. Ramkumar Ravichandran, the
Director of Experimentation at Visa, echoes
Tsao’s sentiment, viewing experimentation as a
win-win, “If your hypothesis was correct, you now
have metrics on just how much value your new
product or feature is delivering to the business.
If your hypothesis was incorrect, you’ve saved
valuable time and avoided launching a product
that would have hurt your business.”

Throughout all of our interviews with change

agents at leading experimentation companies,
we discovered that each one had their own
unique perspective on the power of
experimentation. Still, one thing remained

Pa r t 1
I T E R AT E TO P E R F E C T I O N
4
The most obvious reason to start experimenting
is to choose correctly from a set of potential
options. But the decision itself isn’t the sole
benefit of testing. There is greater value in what
you learn from the process and how you can
then apply that knowledge to future testing.
The team at shoe maker Wolverine Worldwide
wanted to create a new mobile homepage for
one of their brands, Merrell footwear. Their
mobile experience was hard to navigate and
busy; the goal was to make it free of distractions
and clutter.
They hypothesized that a simple and clean
homepage would help nudge users to view
shoe products more quickly and add them to
their carts. The team debated various designs,
disagreeing on which one would elicit the user
behavior they wanted. So, they decided to
build three variations, run an experiment, and
evaluate the results. The design that induced
the greatest number of visitors to view, and
ultimately add shoes to their carts, would be the
undisputed winner. Naturally, things didn’t turn
out so simple.
 5

Version A Version B Version C

Wolverine used experimentation to test three very different landing page

experiences for their mobile shoppers. Version B, with the large category sections,
was the winner for their goal of increasing views to the product details page.

Par t 1
The lesson here is
that even small learnings
from secondary metrics
can give your team insights
into user behavior and help
position future test ideas.
The design that the product lead thought
would convert the best (the large visual grid
displaying shoe categories centered above)
didn’t have a measurable impact on their
primary metric of getting users to add shoes
to their carts. However, the design did induce
the highest number of people to click through
to the product pages. What the team learned
from this experiment was that this specific
design was good at nudging visitors in the right
direction, but it didn’t induce users to make
purchases. This new knowledge informed their
design process as they continued to iterate on
mobile versions that would help improve their
company’s sales. The lesson here is that even
small learnings from secondary metrics can give
your team insights into user behavior and help
position future test ideas.
Similarly, Mike Loveridge, who leads the
experimentation program for Intuit’s TSheets
time management software product, was
looking to refine and focus their software’s
core messaging from a list of options. His
team wanted to differentiate TSheets’
communications in a way that would help them
stand out from competitors. While considering
how they could accomplish this, the team
debated everything. The length of their tagline.
6
Whether to focus on features rather than
benefits. Promoting awards and reviews. They
even debated which adjectives might work best.
With their original slogan ‘Simplify Your business,
Save Money’ representing their core brand story,
they decided to put other taglines to the test.
They tested numerous versions of their value
proposition.
What they eventually uncovered was far more
than just a winning slogan. Through their testing,
they revealed that potential customers were
motivated primarily by peer reviews. Every one
of the slogans that mentioned their reviews
outperformed the lines promoting features and
benefits. This newfound understanding provided
them with deeper insight into their overall
marketing strategy. They now realized they could
be doing much more to validate their product by
highlighting peer reviews at the top of the funnel.
What began as a simple debate over their core
value proposition didn’t just lead them to a
tagline validated by data. It also opened their
eyes to other possibilities. They now were armed
with a vital customer insight that would influence
their future marketing strategy going forward—
an insight they may never have stumbled upon if
not for experimentation.
 7

V3: Time tracking with V4: PC Magazine Editors’

500,000 active users Choice “Best Time
and counting Tracking Software”

V2: 246,897,755
timesheets processed V5: G2 Crowd Winner
and counting for “Best Time
Tracking Software”

V6: TSheets, the highest-

rated time tracking app
V1: 250 million timesheets
on top review hubs
processed and counting

Control: Time tracking with

V7: Easy time tracking with
over 8,000 five-star reviews
over 20,000 five star reviews
(WINNER)

TSheets tested numerous versions of their value proposition on the home page. Pictured
above you can see eight different variations they tested during this iterative phase,
including the winning variant. They learned through this process that marketing the
number of five-star reviews was a key element to their value proposition.
 8

“With Optimizely we’ve been

able to increase our velocity to
40 tests, installed Optimizely
on 6 of our brand sites, and
set-up our team to scale
experimentation across our
portfolio of brands.”

Jill Brown Thomas had the challenge of building a testing program across 12 very

Jill Brown Thomas different brands. Wolverine Worldwide operates as the parent company for brands
Sr. Manager of User Experience like Keds, Merrell, and Saucony. From alignment on people, to building process,
Wolverine to buying technology, to changing and establishing new culture, Jill has been the
change agent as Wolverine goes through a digital transformation.
 9

“We’ve learned the importance

of iterative testing. We build a
lot of variations in a systematic
fashion, which helps us learn
a lot quickly. With Optimizely
it was really easy to turn this
around.”

Mike Loveridge Mike Loveridge is not new to running experimentation teams. Before
Head of Conversion Rate Optimization Intuit, he built out programs at Ancestry.com, GE, Humana, and
Intuit CheapOair. He’s an expert at making experimentation work at high
velocity, even in traffic-challenged situations.
 10

Pa r t 2 Predicting human behavior is hard. Many

experimenters use this idea of testing to uncover
what users really want, before completely
building or getting rid of an entire feature. For
Ramkumar Ravichandran’s team at Visa, the

M A K E B E T T E R D EC I S I O N S value of experimentation has been in answering

B A S E D O N C U STO M E R B E H AV I O R critical product questions without committing

unnecessary time and resources to build
something that ultimately doesn’t work the way
they wish.

So, when they launched their online and mobile

payment product, Visa Checkout, they were able
to test a prototype before making a full-scale
investment. Roger Chang, a Decision Scientist at
Visa, confirmed that viewpoint. “The experience
might not be 100 percent perfect,” he admits, “but
it should be close enough so that when we are
testing to see whether or not the new experience
works, we’ll have a sense before spending our
engineering resources.”

During a redesign of their Visa Checkout

product, the team had discussed the idea of
adding their partner merchants’ logos to the Visa

Pa r t 2 Checkout widget. The partner merchants were
all for it. They wanted to see their branding
on the product, and the Visa team agreed
the partner logos would make the product
appear more official and secure. Upon initial
testing, however, Ravichandran’s team started
to see alarming data that led them to believe
something wasn’t working in their new design.
They inferred that having the branding of each
individual merchant might actually be confusing
because customers were actually entering the
merchant’s credentials rather than using their
Visa Checkout username and password.
Variation A
11
It was clearly time for another experiment. So the
Visa team removed the merchant logo, and tested
again. Despite the product’s new stripped down
look and lack of co-branding, the team immedi-
ately began seeing much better conversion
results. “That was really an interesting test for
me to see,” Ravichandran asserts, “because
even though it’s something that we had as a
value proposition for our merchants —namely,
that they would get their brand shown on our
product — it actually turned out that it was better
for both parties to not have it. And we actually
got much better conversion because of this
Variation B

Pa r t 2
One of Sky Media’s big
initiatives was to improve the
operational efficiency of their
call centers.
design change.” As a result, Ravichandran’s team
now continues to test prototypes to their user
interface, iterating and perfecting as they go,
before rolling out official changes to their product.
The team at Visa is not alone. Similarly, the
product group at Sky Media, Europe’s largest
media and telecommunications company, also
aligns their experimentation practice with their
Agile development process. Last year, one of
Sky Media’s big initiatives was to improve the
operational efficiency of their call centers where
customers receive technical assistance when
resolving problems with their set-top boxes.
Typically, calling in isn’t a positive customer
experience. It’s also costly for Sky Media to staff
these call centers effectively as the volume of
problems surge. Curious about experimentation,
they wondered if they could use it to tackle this
challenge at their digital help facilities.
As a rule, Sky Media always begins any
improvements with Discover, Design, Develop,
their in-house test framework that aligns
with the way their product managers and
engineers build and deploy new features. So,
while trying to find ways to improve their online
help experience, they discovered something
12
unexpected — one of the main reasons people
contact their call center is to get help
understanding their bills.
One component of their online help experience is
an interactive page that breaks down the details
of the customers’ monthly bills. For obvious
reasons, the team at Sky set out to understand
why customers weren’t getting the answers they
needed online. After looking at heatmaps and
behavioral data on their website, they learned
that customers often come to the billing page,
navigate away from the page, and then come
back at a later time. This process would repeat
itself until the customer would eventually call in
for support.
When the Sky Media team dug deeper they
realized that a large sidebar on the interactive
page featuring “related articles” — a module
they assumed, but never validated with data, to
be a best practice for keeping visitors on your
site — might be the culprit. Their hypothesis was
that this “related articles” section was actually
directing users to pages that didn’t address their
problems.

Pa r t 2 So, before they built an entirely new product
experience, the team wanted to validate this
assumption with an experiment. They created
a new variation of the page without the “related
articles” sidebar and observed what impact it
might have on their call volume. It turned out
Version A
13
their assumption was correct. This new version
increased “goal completion”, a Sky Media
metric that measures their help site’s customer
satisfaction. It also resulted in a decreased
number of customer calls regarding their
billing process.
Version B
 14

“If your hypothesis was correct, you now have

metrics on just how much value your new
product or feature is delivering to the business.
If your hypothesis was incorrect, you’ve saved
valuable time and avoided launching a product
that would have hurt your business.”

Ramkumar Raavichandran Ram has been in charge of building Visa’s experimentation program:
Director of A/B Testing from selling the value, to laying down the vision, the roadmap and
Visa success criteria, to creating the right team and driving engagement.
 15

“There is no secret. We fail

fast, we test constantly, and
we repeat that process to
learn more and more about our
customers.”

In 2015 Sky begun heavily investing in digital innovation for Sky.com. With teams
and resources scattered they could quickly see that they needed a focus. They
Jonathan Crowder
formed the Sky Digital Centre of Excellence which was chartered for defining their
Senior Digital Optimisation Analyst
digital vision moving forward. Central to that vision was to establish a data-driven
Sky
culture where decision making was powered by reliable data and insight. Today,
Jonathan leads a team of analysts at their center for excellence in Leeds, UK.

Pa r t 3
O P T I M I Z E YO U R
C U STO M E R J O U R N E Y
16
There are countless questions that stand
between you and the things your organization
hopes to accomplish. Should you offer a free
trial? Should you require a credit card number?
How should we package our product offerings?
The list goes on and on, but each question,
if answered correctly, leads to an informed
decision that brings your business one step
closer to achieving its ultimate objectives.
Atlassian is not like most business-to-
business (B2B)software companies. While
most enterprise software companies secure
significant revenue by employing large sales
teams, Atlassian’s revenue is earned solely
online, through its website. They operate in the
same way that a typical eCommerce site does.
From the very beginning, it has been absolutely
essential that their website be optimized, and
that it deliver visitors the best user experience
possible. The product and marketing teams at
Atlassian experiment on everything — from their
bundled product offers, to their personalized
landing pages, to their free trial offers
eCommerce chief Tom Tsao is very clear about
one thing — the majority of their experiments,
regardless of their nature, are designed to help
his team validate and make better business
decisions

Pa r t 3
One hypothesis Atlassian
formed was that if they
removed the credit card from
the initial sign-up process,
they would reduce friction in
the process and secure more
participants.
Take Atlassian’s free trial, for example. The
trial offer is a huge driver of new leads. It’s an
ideal way for customers to test their product
before formally committing to the software. For
Atlassian, it provides the perfect opportunity to
make a positive first impression and possibly
embed themselves into an organization’s
daily workflow. The Atlassian team has tested
everything about their trial offer, including trial
length, product bundling, as well as the timing
of their credit card request. One hypothesis they
formed was that if they removed the credit card
from the initial sign-up process, they would
reduce friction in the process and secure more
participants.
17
They were right. But what also occured when
they tested this hypothesis surprised them. In
addition to signing up more trial users, removing
the credit card request had no negative impact
on their down funnel conversion metrics.
Requesting a credit card is widely viewed as
a way to qualify prospects. The logic goes, if
they’re willing to give you a number, they must
be reasonably serious about your product.
Those just casually interested generally won’t
enter their credit card numbers. Upon removing
this step, they worried they might get less-
qualified buyers. Their experiment proved,
however, that these less-qualified users were
still, in fact, converting from free trial users to
paying customers.
The resulting spike in new trial participants
generated more traffic which allowed Atlassian
to run even more in-trial tests with new offers
and promotions. In other words, the influx
of new users provided them with even more
opportunities to broaden, activate, and educate
their customer base.
Another B2B change agent is Renny Chan, who
runs a growth team at file sharing service Box
Inc. in Redwood CIty. While Box has a sales team
 18

Pa r t 3
Business Plus was a
plan that appealed to select
Box customers who needed
additional features but weren’t
quite ready to commit to a large,
that sells their software to enterprise businesses,
they also serve a segment of the SMB market. So,
while their website is primarily a marketing tool,
it also services as the business engine for SMB
customers. These customers can purchase a plan
from the Box website and begin sharing files right
away.
Originally, Box had a Starter plan, a Business
Chan’s team then ran a test that measured the
impact of adding this fourth plan to their website.
It was important, they felt, to also calculate the
measurable loss to the Business plan, since many
potential Business plan customers might opt for
the Business Plus plan instead. After running the
test for one month, they did observe a decrease in
Business plan purchases. The spike in Business
Plus purchases, however, was more than enough

enterprise license.
plan, and an Enterprise offering that required to offset the difference.
customers to contact the sales team for more
details. What Chan uncovered was that there was
a fourth Box plan in existence that wasn’t on the
website called Business Plus.

Business Plus was a plan that appealed to select

Box customers who needed additional features
but weren’t quite ready to commit to a large,
enterprise license. As it turns out, this plan was
popular among customers who talked with the
sales team. As you might expect, Chan’s group
considered the value of including the Business
Plus plan on their website. Doing so, they
presumed, would accomplish two things — one, it
would free the sales team to focus on larger, more
strategic deals and, two, it would provide SMB
customers with a faster, more flexible option.
 19

Pa r t 3

Adding business plus results in

an 11% increase in ARR for Box.

Business Plus plans were converting at a their annual recurring revenue (ARR). In the end,
1.57% traffic-to-trial rate vs. a 1.50% rate for the Chan’s group prompted a measurable impact on
Business plan. Further down the funnel, the team their subscription business while also increasing
was also able to measure the impact on ARR by 11%. All, with this one, simple test.
 20

“We saw a measurable impact on

our subscription business while
also increasing ARR by 11%.
All, with this one, simple test.”

Last year the growth team at Box revisited a design layout from their Plans page.
A page that they had been excited about redesigning in the past, but had failed to
RENNY CHAN show the measurable improvements needed to key business metrics. With a new
Sr. Manager, Growth
hypothesis and experiment implementation — the team found that this redesign not
Box
only delivered more sign ups, it also made a significant improvement to their annual
recurring revenue (ARR).
 21

“It’s simple,” Tsao says,

“Experimentation lets us make
better business decisions.”

Today, Atlassian runs experiments and personalization campaigns across

TOM TSAO all their products, campaigns, and commerce. But it wasn’t always like this.
Head of Global eCommerce Atlassian started small and at their own pace, grew across the enterprise. Tom
Atlassian Tsao, the head of eCommerce at Atlassian, credits early wins in the product that
helped evangelize their program and get buy-in from senior leadership.

C on cl u si o n At Optimizely, we’ve seen that the organizations
that out-experiment their competition, ultimately
outperform the competition. Granted, moving
from a company that performs periodic A/B tests
to a company that truly builds experimentation
into the fabric of its culture takes time. But it all
starts with a few simple wins, and then using
those wins to evangelize experimentation across
other teams in your business.
The chief principle we hope you take away from
this ebook is that experimentation isn’t just
about button colors or optimizing conversion
rates on your landing pages. It’s about the way
digital change agents use experimentation to
achieve significant business objectives. The
common theme we see among all of these
visionary individuals is that they see the bigger
picture, and they align their experimentation
22
programs to meet the needs of their business
initiatives. At companies like Visa, using
experimentation helps validate their product
roadmap. For Merrell, experimenting allows
them to constantly refine the mobile shopping
experience. And at organizations such as
Atlassian, experimentation empowers team
members to make better business decisions.
You too can become one of these change
agents. You can make your organization more
competitive. You can propel its future success.
All you need to do is embrace experimentation
and align it with key business initiatives.
Digital Change Agents
optimizely.com/digital-change-agents
The Webinar Series

Want to learn more about the individual stories of Atlassian, Visa, Wolverine, Sky, Box and Intuit?
The Digital Change Agents webinar series goes behind the scenes to show you the journey
of each of these change agents in six different 45-minute sessions.