Confidence Level 90% 95% 97% 97.

5% 98% 99%

0.1 0.05 0.03 0.025 0.02 0.01

/2 0.050 0.025 0.015 0.0125 0.010 0.005

Z(/2) 1.645 1.96 2.17 2.24 2.33 2.575

Chapter 7 You need confidence intervals b/c your test and your roll-our will never be 100% identical. with samples there is always a level of associated error variance // the higher the error variance of a test the more likely your results will be off in your roll-out. CI can tell you if theres a statistically significant difference between the test and the control and what the lower- and upper-bound results are (best and

worst case scenario). CI for sample mean (x) - n30, sample mean is equal to the population mean, is normally distributed, but has a larger st. dev. CI for sample proportion (p) - np5 and n*(1-p) 5; the sampling distribution of a sample proportion is distributed normally such that CI for difference in means (x x ) both n30; the difference between two sample means for both n30 is distributed normally such that CI for difference in proportions (p p ) np5 and n*(1-p) 5; the linear combination of sample percentages is distributed normally, such that -- even if we know that one option is better than the other, we need to consider if its better enough to actually breakeven Single estimate: Is the range small or big? Is there a lot of error? How accurate is my data? Is it a good prediction for what the outcome will be? Difference between two means/proportions: Is there statistically significant difference between the two options (if 0 is in the interval, there is no difference, they need to be the same sign)? What is ? Its the level of significance of a given test the level of uncertainty; representation of the confidence level, e.g. = 0.1, confidence level = 90% If you need a narrower CI, you can either lower your confidence level () or increase the sample size. If you can afford to increase sample, you should do that b/c it wont compromise your confidence Your results are only valid if no outside circumstances change from test to rollout (e.g. seasonality, major competitor emerging, increased competition, bad press, major natural disaster, changes in promotion or offer). Setting confidence levels how much risk am I willing to take? Chapter 8 H0: p1p2=d H1: p1p2d Terminology hypothesis test vs. its alternative: we are testing a control treatment to a group of subjects with a known (usually) outcome in response and profitability versus a test treatment to a group of subjects from the same population with an unknown outcome. Type 1 error () you reject H0 when its true associated with confidence interval Type 2 error you accept H0 when its false lost opportunity You always have to look for anomalies and outliers in your tests!!! Left-Tailed: Right-Tailed: Two-Tailed: Step1: state your hypothesis Step 2: Compute TS Step 3a: If youre given the level use decision rule Step 3b: If you dont have an level, calculate p-value Step 4: Make a conclusion TS is always a standard normal random variable (with a mean of 0, and st dev of 1). Therefore if TS is more than 2-3, you automatically reject H0 at all standard confidence levels and assume there is a difference. Difference between means ($, time on site, page views), for n>30 Based on CLT difference between two means is normally distributed with a mean equal to the difference between the two population means and a st dev equal to the linear combination of the individual population st devs

Difference between proportions (order rates, attrition rate, CTRs, conversion rate), for p>5 Based on CLT difference between two sample proportions is normally distributed for n1p1, n2p2, n1*(1-p1), n2*(1-p2) with a mean equal to the difference between two population proportions p1-p2 and a st dev equal to the linear combination of the individual population st deviations p-value the level of (confidence) which just causes the rejection of H0 // the probability of observing a more extreme TS reject H0 if pvalue accept H0 if pvalue for TS>3, p-value is approaching 0 or >0.0001 For TS = 1.70 P(z>1.70)+P(z<-1.70) = 2* [0.5 - P(0<z<1.7)] p-value = 0.089 1-p-value = level (1-0.089 = 91.1%) p-value < - reject H0 0.089 < 0.1 ; 0.089>0.05 ; 0.089 >0.01 reject H0 for 90%, accept for 95% and 99% Why do we care? In real life you rarely get the confidence level & when you use the standard ones you get limited data do you wantto roll out a campaign? Is it worth the risk? Decision rule: One tailed if TS> Z - reject H0 Two tailed: if TS>Z(/2) reject H0 // if TS< Z(/2) accept H0 One-Tailed vs. Two-Tailed two-tailed if you want to know if there is a difference between the two, more conservative // onetailes if youre testing if one outcome is more or less than the other (e.g. if profit will increase sufficiently or if payment levels wont decline more significance in concluding the alternative hypothesis) Confidence Interval Hypothesis Testing *Create an interval over single mean or proportion or a *Can test whether there is a difference between two means or difference between two means or proportions. proportions. *If 0 is in, there is a statistically significant difference. *If TS is large, there is a statistically significant difference. *You need a set confidence level, hard to establish exact level of *You dont need a pre-set level of significance b/c you can read significance. and interpret the TS and calculate the p-value. *Best & worst case scenarios to use in P&Ls for determining ROI *Stating differences and the significance of difference *Can only be compared with two-tailed HT *Examples of inferential statistics the kind which allows for generalization about data and decision making * Rooted in the CLT and use the property of linear combinations of normally distributed random variables * Always work together and never contradict each other 1) Test AOS = $75 Control AOS = $72 2) directionally Test is better, but we dont know if its better enough 3) Test for statistical significance (H0: x1=x2 H1: x1x2) 4) if there is stat. sig. it is always in the original direction! Case 1 TS = 5.26 pvalue<0.0001 reject H0 for all levels Case 2 TS=1 pvalue = 0.3174 accept H0 for all levels Case 3 TS=2.01 pvalue = 0.0444 reject H0 for = 0.1 and 0.05, accept for = 0.02 and 0.01

CI: single mean Tinterval, single prop 1-PropZInt Difference in means 2SampTInt, difference in prop 2-PropZInt HT: mean 2-SampTTest, proportion 2-PropZTest 1. Total Revenue: Test = 365.33 * 253 = 92,428.49 (AOS * #orders) Control = 353.33 * 240 = 84,799.20 7,629.29 / by #orders in test (30.15) looks like enough to pay shipping costs Create CI @ 99% (6.083, 17.917)