1. Pros
1.1. Fast
1.1.1. It takes very little time to create a modified version of an existing web page that includes a modified item
1.2. Tests reality, not theory
1.2.1. A/B testing on a live web site is you’re obtaining real results from real users doing real things.
1.3. Quantifiable
1.3.1. A/B web site testing provides actual numbers that can be compared, sliced and diced to evaluate results. Interaction, conversion, number of abandonments
1.4. Accurate
1.4.1. A/B testing is 100% accurate assuming you have statistically significant data.
2. Cons
2.1. Can Hurt Web Site Results
2.1.1. Thought was an excellent B test item in an A/B test, may go terribly wrong.
2.2. Missing the “Why”
2.2.1. A/B web site testing does not explore the rationale behavioural decisions that are being made by the web site visitors.
2.3. Not Predictive
2.3.1. A/B testing is great and all, but it can’t be used to predict future design change impacts.
2.4. Needs Traffic
2.4.1. Need a pretty good amount of traffic to your web page to run an A/B test, in order to provide quick, consistent and reliable results
3. What is A/B Testing?
3.1. Two version of an element (A and B)
3.2. Metric that define success
3.3. Experiment both versions simultaneously to determine which version is better
3.4. Measure which version was more successful and select that version for real-world use
4. What to test?
4.1. The call to action’s (i.e. the button’s) wording, size, color and placement
4.2. Headline or product description
4.3. Form’s length and types of fields
4.4. Layout and style of website
4.5. Product pricing and promotional offers
4.6. Images on landing and product pages
4.7. Amount of text on the page (short vs. long)
5. How to create A/B Test
5.1. Replace the element to be tested before the page loads
5.1.1. When testing a single element on a Web page, create variations of that button (in HTML)
5.1.2. When test is live, the A/B tool will randomly replace the original button on the page with one of the variations before displaying the page to the visitor
5.2. Redirect to another page
5.2.1. If want to A/B test an entire page, need to create and upload a new page on your website
5.2.2. When the test runs, your tool will redirect some visitors to one of your alternate URLs
6. Do’s And Don’ts
6.1. DON’TS
6.1.1. When doing A/B testing, never ever wait to test the variation until after you've tested the control. Always test both versions simultaneously.
6.1.2. Don’t conclude too early. There is a concept called “statistical confidence” that determines whether your test results are significant
6.1.3. Don’t surprise regular visitors. If you are testing a core part of your website, include only new visitors in the test.
6.1.4. Don’t let your gut feeling overrule test results. The winners in A/B tests are often surprising or unintuitive.
6.2. DO’S
6.2.1. Know how long to run a test before giving up. Giving up too early can cost you because you may have gotten meaningful results had you waited a little longer.
6.2.2. Show repeat visitors the same variations. Your tool should have a mechanism for remembering which variation a visitor has seen.
6.2.3. Make your A/B test consistent across the whole website.
6.2.4. Do many A/B tests. An A/B test can have only three outcomes: no result, a negative result or a positive result.