1. what is A/B testing
1.1. You have two designs of a website: A and B. Typically, A is the existing design and B is the new design. You split your website traffic between these two versions and measure their performance using metrics that you care about (conversion rate, sales, bounce rate, etc.). In the end, you select the version that performs best.
2. what to test
2.1. The call to action’s (i.e. the button’s) wording, size, color and placement, Headline or product description, Form’s length and types of fields, Layout and style of website, Product pricing and promotional offers, Images on landing and product pages, Amount of text on the page (short vs. long).
3. DOs
3.1. Know how long to run a test before giving up. Giving up too early can cost you because you may have gotten meaningful results had you waited a little longer.
3.2. Show repeat visitors the same variations. Your tool should have a mechanism for remembering which variation a visitor has seen.
3.3. Make your A/B test consistent across the whole website. Showing one variation on page 1 and another variation on page 2 will skew the results.
3.4. Do many A/B tests. The key to optimizing conversion rates is to do a ton of A/B tests, so that all positive results add up to a huge boost to your sales and achieved goals.
4. DON'Ts
4.1. never ever wait to test the variation until after you’ve tested the control. Always test both versions simultaneously. If you test one version one week and the second the next, you’re doing it wrong.
4.2. Don’t conclude too early. There is a concept called “statistical confidence” that determines whether your test results are significant. It prevents you from reading too much into the results if you have only a few conversions or visitors for each variation.
4.3. Don’t surprise regular visitors. If you are testing a core part of your website, include only new visitors in the test.
4.4. Don’t let your gut feeling overrule test results. The winners in A/B tests are often surprising or unintuitive.
5. pros
5.1. fast
5.1.1. it takes very little time to create a modified version of an existing web page that includes a modified item and put in on the webpage. Results can be gained quickly too, by splitting traffic 50-50 between the existing and the test version of the pages you can measure the results in short order
5.2. test reality
5.2.1. obtaining real results from real users doing real things. Not using theory, estimates, forecasts
5.3. quantifiable
5.3.1. A/B web site testing provides actual numbers that can be compared, sliced and diced to evaluate results
5.4. accurate
5.4.1. A/B testing is 100% accurate if have statistically significant data.
6. cons
6.1. hurt website result
6.1.1. In A/B testing that bad decision, may go terribly wrong. When that happens you’re going to end up hurting your overall web site results
6.2. miss the "why"
6.2.1. A/B web site testing does not explore the rationale behavioral decisions that are being made by the web site visitors. You won’t know for sure WHY all those web site visitors picked the item they picked. You may have theories, but you won’t know for sure.
6.3. not predictive
6.3.1. A/B testing is great and all, but it can’t be used to predict future design change impacts. To a certain extent this means that you’re always stuck doing A/B testing
6.4. needs traffic
6.4.1. In order to provide quick, consistent and reliable results, you’re going to need a pretty good amount of traffic to your web page to run an A/B test. A/B testing works great in high-traffic areas, but becomes more problematic in lower-traffic areas.