A/B Testing

1. Creating the A/B Test

1.1. Tools

1.1.1. Google Website Optimizer

1.1.2. Visual Website Optimizer

1.1.3. A/Bingo and Vanity

1.1.4. Unbounce and Performable

1.1.5. Vertster, SiteSpect, Webtrends Optimize and Omniture’s Test&Target

1.2. Setting up

1.2.1. Replace the element to be tested before page loads

1.2.2. Redirect to another page

2. Do's And Don'ts

2.1. Don'ts

2.1.1. Never wait to test the variation until after testing the control, always test both simultaneously

2.1.2. Don't conclude too early. Statistical confidence determines if test results are significant

2.1.3. Don't surprise regular visitors, only test on new visitors

2.1.4. Don't let your gut feeling overrule test results

2.2. Do's

2.2.1. Know how long to run a test before giving up

2.2.2. Show repeat visitors the same variations

2.2.3. Make A/B test consistent across the website

2.2.4. Do many A/B tests

3. Pros of A/B Testing

3.1. Fast, takes little time to create a modified version

3.2. Tests reality, not theory, obtaining real results from real users doing real things

3.3. Quantifiable, providing actual numbers that can be compared

3.4. Accurate, assuming you have significant data

4. Cons of A/B Testing

4.1. Can Hurt Web Site Results, making bad decisions from time to time

4.2. Missing the "Why", A/B testing does not explore the rational behind decisions made by visitors

4.3. Not Predictive, can't be used to predict future design change impacts

4.4. Needs Traffic, to provide quick, consistent and reliable results

5. What is A/B Testing

5.1. 2 Versions of an element (A/B) and a metric that defines success

5.2. Subject both versions to experimentation simultaneously to determine which version is better

5.3. Select the better version for real world use

6. What to test

6.1. Depends on the goals

6.2. Call to action's wording, size, color and placement

6.3. Headline or product description

6.4. Form’s length and types of fields

6.5. Layout and style of website

6.6. Product pricing and promotional offers

6.7. Images on landing and product pages

6.8. Amount of text on the page (short vs. long)