A/B Testing

1. Do’s

1.1. Always test both versions simultaneously.

1.2. Always split traffic between two versions.

1.3. Know how long to run a test before giving up.

1.4. Show repeat visitors the same variations.

1.5. Make your A/B test consistent across the whole website.

1.6. Do many A/B tests.

2. Don'ts

2.1. Don’t surprise regular visitors.

2.2. Don’t conclude too early.

2.3. Don’t let your gut feeling overrule test results

2.4. Don’t reject the results because of your arbitrary judgment.

3. Pros & Cons of A/B Testing

3.1. Pros

3.1.1. 1. Fast - matter of splitting traffic to the two pages

3.1.2. 2. Tests reality - obtaining real results from real users doing real things

3.1.3. 4. Accurate - assuming you have statistically significant data.

3.1.4. 3. Quantifiable - provides actual numbers

3.2. Cons

3.2.1. 1. Can Hurt Web Site Results - make some bad decisions from time to time

3.2.2. 2. Missing the “Why” - you won’t know for sure

3.2.3. 3. Not Predictive -can’t be used to predict future design change impacts

3.2.4. 4. Needs Traffic -need a pretty good amount of traffic

4. What is A/B Testing?

4.1. 2 versions of an element (A and B) and a metric that defines success.

5. What to test?

5.1. Dependent on your goals

6. Set up an A/B test in one of two ways

6.1. Replace the element to be tested before the page loads

6.2. Redirect to another page