Regardless of whatever ‘Call to Action’ you use to try and influence sales on

your website, there is probably one that statistically works better than others.
Find out which one performs better using the Ad Copy Analysis.

1
I’ve been trying different selling pitches and I want to know if one performs
better than the others. How do I find out if one Ad Copy is statistically better
at influencing orders placed than others?

We want to know whether our orders increase when we had “Save 20% Off!” written at
the top of our website versus other Ad Copy. Suppose weekly orders were monitored for
16 weeks on your website. For 7 weeks the “Save 20% Off!” logo was placed at the top
of the web page. For 5 weeks the “$15 Off!” logo was placed at the top of the web page.
Finally, for 4 weeks the “Click Here to Order!” logo was placed at the top of the web
page. Resulting orders are contained in the worksheet named Insight in the file Location
of Measuring different Ad Copy.xls, which is shown in the figure below. Does the data
indicate that the location of the “Save 20% Off!” logo has a significant affect on orders
placed?

2
We assume that during these 16 weeks the same products and content were featured on
the website. This assumption allows us to use one-way ANOVA because we believe that
one factor (the Ad Copy on the Top of Web Page) is affecting orders placed. (If different
products were being sold on various days, we would need to analyze our data with two-
way ANOVA, which we’ll discuss after this).

To analyze the data, choose Tools, Data Analysis, and then select ANOVA: single
factor. Fill in the dialog box as shown in the figure below.

 The data for our input range, including labels, is in cells B3:D8.
 Select the labels option because the first row of our input range contains labels.
 The Columns option was selected because the data is organized in columns.
 Cell E12 was chosen as the upper-left corner of the output range.
 The selected alpha value is not important. You can use the default value.

After clicking OK, we obtain the results shown below.

3
In cells H16:H18, we see average orders depending on the different Ad Copy written.
When the “Save 20% Off!” Ad Copy is at the top of the Web Page, average orders are
572.81; when the “$15 Off!” Ad Copy is used, orders average 616.06; and when the
“Click Here to Order!” Ad Copy is used, orders average 490.87. Because our p-value
of 0.834 (in cell J23) is greater than 0.10, we can cannot conclude that these means
are significantly different. In other words, it does not matter what Ad Copy you use as
they do not influence orders placed. If the p-value were less than 0.10, we could say
“we are 90% confident that the means are significantly different”.

Okay, wait a second. Orders average 616.06 when the “15$ Off!” Ad Copy is located
at the Top of the Web Page versus 490.87 when the “Click Here to Order!” Ad Copy
is located at the Top of the Web Page. Clearly it is better to use the “15$ Off!” Ad
Copy at the Top of the Web Page rather than using the “Click Here to Order!” Ad
copy right?

Unfortunately it does not make a difference which Ad Copy you use. The results
of performing a One-Way ANOVA between orders for when the “$15 Off!” Ad Copy
was used versus “Click Here to Order!” are located in the output range: E33:K46.
Because the p-value of 0.509 (in cell J43) is greater than 0.10, we cannot conclude
that these means are significantly different.

4
Can you explain further?

The reason for this strange result is that in this data set, we have a tremendous amount
of variation in orders when the different Ad Copies were used. Looking at the Ad
Copy “$15 Off!”, for example, the variation in orders is between 200 and 954 and
“Click Here to Order!” the variation in orders is between 184 and 675. The large
variation within the data points for each Ad Copy masks the variation between the Ad
Copy themselves and makes it impossible to conclude that the difference between
orders resulting from different Ad Copied used is significant.

Can I see an example of when a specific Ad Copy does influence orders?

Please open the worksheet named Signif-One Way. In this data set we are
analyzing 12 weeks of data and again are assuming that during these 12 weeks the
same products and content were featured on the website. See example below.

To analyze the data, choose Tools, Data Analysis, and then select ANOVA:
Single Factor. Fill in the dialog box as shown in the figure below.

5
 The data for our input range, including labels, is in cells B3:D8.
 Select the labels option because the first row of our input range contains labels.
 The Columns option was selected because the data is organized in columns.
 Cell E12 was chosen as the upper-left corner of the output range.
 The selected alpha value is not important. You can use the default value.

After clicking OK, we obtain the results shown below.

6
 In cells H16:H18, we see average orders depending on the location of the different
Ad Copies used. When the “Save 20% Off!” Ad Copy is at the top of the Web Page,
average orders are 919.6; when the “$15 Off!” Ad Copy is used, orders average
722.5; and when the “Click Here to Order!” Ad Copy is used, orders average 450.
Because our p-value of 0.0004 (in cell J23) is less than 0.10, we can conclude that
these means are significantly different. In other words, the “Save 20% Off!” Ad Copy
located on Top of the Web Page positively influences orders more so than the other
Ad Copies used.

In addition to testing different Ad Copies, I was also selling different

products on various days. How can I incorporate the different products being
sold into my analysis?

In this situation, two factors (the Ad Copy and the product being sold) are influencing
orders. When two factors might influence a dependent variable, two-way ANOVA
(analysis of variance) can be used to determine which, if any, of the factors have a
significant influence on the dependent variable.

So now we want to determine how a product and the Ad Copy influence orders
placed. To answer the question in this example, we can have each of our products spend a
few weeks selling when all 3 Ad Copies were shown on the Top of the Web Page. The
resulting orders are given in the worksheet Two Factor as shown in the results below.

7
 To analyze this data in Excel, click Tools, Data Analysis, and then select the
option ANOVA: Two-Factor Without Replication. Then fill in the dialog box as shown in
the figure below.

8
 The data for our input range, including labels, is in cells C5:F9.
 Select the labels option because the first row of our input range contains labels.
 The Columns option was selected because the data is organized in columns.
 Cell B12 was chosen as the upper-left corner of the output range.
 The selected alpha value is not important. You can use the default value.

The output we obtain is shown in the figure below.

To determine whether the row factor (Product) or column factor (Ad Copy used)
has a significant effect on orders placed, just look at the p-value. If the p-value for a
factor is low (less than 0.10), the factor has a significant effect on sales. The row p-
value (0.06099279) and column p-value (0.0072378) are both less than 0.10, so both
product being featured and type of Ad Copy used have a significant effect on orders
placed. In this case, a combination of selling Designer Jeans and using the “Save 20%
Off!” Ad Copy at the top of the Web Page generates the highest volume of orders
placed.

9
 BONUS: Based on the knowledge of Products and Ad Copy used, can I
forecast orders?
We can predict orders during a week by using the equation shown here:

Predicted orders = Overall average + (Product effect) + (Ad Copy Used)

In this equation, product effect equals 0 if the product factor is not significant. If
the product factor is significant, product effect equals the mean for the given rep
minus the overall average. Likewise, type of Ad Copy equals 0 if the Ad Copy factor
is not significant. If the Ad Copy factor is significant, the Ad Copy used equals the
mean for the given district minus the overall average.

Let’s jump to the Bonus worksheet. I computed the overall average (73.58) in cell
D33 of the Bonus worksheet by using the formula =AVERAGE(D6:F9). The product
and type of ad copy effects are computed by copying from cell G15 to G16:G22 the
formula E15-$D$33. As an example, you can compute predicted orders by selling
Seinfeld DVD Sets when the “Click Here to Order!” Ad Copy is used as 73.58 -4.92 -
20.83 = 47.83. This value is computed in cell D37 with the formula D33 + G18 +
G22.

10
 The standard deviation is the square root of the mean square error shown in cell E29.
I computed this standard deviation in cell E30 with the formula =SQRT(E29). Thus,
we are 90 percent sure that if selling a Seinfeld DVD set when the “Click Here to
Order!” Ad Copy is used; weekly orders would average about 47.83 or in between
27.43 and 68.24. These limits are computed in cell D38 and D39 with the formulas
D37 – 2*E30 and D37 + 2*E30, respectively.

Legal Disclaimer

The opinions and analyses provided here represent my own and not those of my Clients.
Bears repeating: The tutorials do not represent the thoughts, intentions, plans, actual data
or strategies of my clients or employer.

11