1. Pro's of A/B Testing
1.1. Quantifiable
1.1.1. Provides actual numbers that can be compared, sliced and diced to evaluate results. Interaction, conversion, number of abandonments – all those numbers are accessible during and after testing.
1.2. Fast
1.2.1. it takes very little time to create a modified version of an existing web page that includes a modified item (like a new picture, new copy or other new element) and throw it up on your site)
1.3. Accurate
1.3.1. Understanding error rate and statistical significance and all those other statistics terms
1.4. Tests reality, not theory
1.4.1. Obtaining real results from real users doing real things.
2. Con's of A/B Testing
2.1. Can Hurt Web Site Results
2.1.1. In A/B testing that bad decision, meaning what you thought was an excellent B test item in an A/B test, may go terribly wrong.
2.2. Missing the “Why”
2.2.1. No understanding of the reasons to the results
2.3. Not Predictive
2.3.1. Can’t be used to predict future design changes
2.4. Needs Traffic
2.4.1. Provide quick, consistent and reliable results
2.4.2. Need a pretty good amount of traffic to your web page to run an A/B test.