1. Cons
1.1. 1. Can Hurt Web Site Results
1.1.1. Bad decision can lead to hurting the overall web site results
1.1.2. Splitting a smaller portion of traffic will reduce potential huge catastrophe, but also takes longer to get enough data to make an accurate evaluation
1.2. 2. Missing the "Why"
1.2.1. Web site managers will not know the reasons WHY web site visitors picked a particular item
1.2.1.1. No definite answers, only assumptions and theories
1.3. 3. Not Predictive
1.3.1. Cannot be used to predict future design change impacts
1.4. 4. Needs Traffic
1.4.1. Need an amount of traffic to provide quick, consistent and reliable results when running A/B test
1.4.2. If testing on low-volume pages, it defeats the "fast" & "accurate" benefits of A/B testing
2. Definition
2.1. Craig's
2.1.1. Simultaneously testing an existing item against a modified version (or versions) of the item by splitting traffic evenly between the items, and measuring results
2.2. Wikipedia's
2.2.1. Method of advertising testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates
3. Pros
3.1. 1. Fast
3.1.1. takes little time to create a modified version of an existing web page that includes modified item & throw it up on the site
3.1.1.1. Modified item: New picture/copy/other element
3.1.2. Results can be gained quickly too
3.1.2.1. If traffic split into 50-50, can measure results in shorter order
3.1.2.2. If traffic split into 10/20, test result will be longer
3.1.2.2.1. This defeats the purpose of using A/B testing
3.1.2.2.2. Web site managers prefer to keep the percentage of traffic low, for the test page
3.2. 2. Test Reality, not theory
3.2.1. Obtains real results from real users, doing real things
3.2.2. No theory, estimates, forecasts, predictions are needed
3.3. 3. Quantifiable
3.3.1. Provides actual numbers that can be compared, sliced and diced to ecaluate results
3.3.1.1. Can determine the interaction, conversion, number of abandonments
3.3.2. No guessing and prediction are required
3.4. 4. Accurate
3.4.1. Have statistically significant data to understand error rate & statistical significance etc
3.4.2. 100% accurate
4. What to Test?
4.1. Depends on the company's goals
4.1.1. E.g. Increase the no. of sign-ups
4.2. Elements that are usually tested
4.2.1. The call to action’s (i.e. the button’s) wording, size, color and placement
4.2.2. Headline or product description
4.2.3. Form’s length and types of fields
4.2.4. Layout and style of website
4.2.5. Product pricing and promotional offers
4.2.6. Images on landing and product pages
4.2.7. Amount of text on the page (short vs. long)