A/B Testing
by Kim Meng Ng
1. What is A/B Testing? And how it works?
1.1. A/B Testing is a type of metric that defines success.
1.2. A/B Testing works by offering two versions of a website to site visitors. Assuming, Version A is the existing design and Version B is the new design.
1.3. The best version is determined by using performance metrics, such as conversion rate, sales, bounce rate and etc.
1.4. By comparing the performance metrics of two different websites, the version with better performance metrics will be chosen.
2. Do's and Don'ts of A/B Testing
2.1. Do's
2.1.1. Knowing how long to run a test, before giving up.
2.1.2. Show repeat visitors the same variations.
2.1.3. Make sure that the A/B test is consistent across the whole website.
2.1.4. Do many A/B tests.
2.2. Don'ts
2.2.1. Never ever, wait to start testing the second version only after the first version has completed testing.
2.2.2. Don't conclude too early.
2.2.3. Don't surprise regular visitors, by allowing them to use the old version if they wanted to.
3. What to Test?
3.1. - The button's wording, color, size, placement - Headline or product description - Form's length and type of fields - Layout and style of website - Product pricing and promotional offers - Images on landing and product pages - Amount of text on the page
3.2. The goal of A/B Testing is to figure out what prevents the user from accomplishing a goal.
4. Pros of A/B Testing
4.1. Fast. It takes very little time to create a modified version of an existing web page.
4.2. Tests reality, not theory. The actual results are based on what users are doing on the web site.
4.3. Quantifiable. Provides actual numbers as results, so that it can be compared, sliced and diced. Therefore, no guessing is required.
4.4. Accurate. It is 100% accurate assuming there is statistically significant data.
5. Cons of A/B Testing
5.1. Can Hurt Web Site Results. In order to get accurate A/B testing results, visitors may have to go through either or all of the different versions of the web site. A bad decision in a particular version may end up hurting overall web site results.
5.2. Missing the "Why". A/B Testing does not explore the rationale behavioral decisions that are being made by the visitors of the web site.
5.3. Not Predictive. A/B Testing can't be used to predict future design change impacts.
5.4. Needs Traffic. There needs to be more visitors in order to have valid analysis.
6. Tools used for A/B Testing
6.1. Google Website Optimizer
6.2. A/Bingo and Vanity
6.3. Unbounce and Performable
6.4. Vertster, SiteSpect, WebTrends Optimize and Omniture's Test&Target
