A/B Testing

1. Advantages

1.1. 1. Fast Of all the test types, A/B is way, way fast. That’s because it takes very little time to create a modified version of an existing web page that includes a modified item (like a new picture, new copy or other new element) and throw it up on your site. Then, it’s just a matter of splitting traffic to the two pages.

1.2. 2. Tests reality, not theory The good news about A/B testing on a live web site is you’re obtaining real results from real users doing real things. That means you’re not using theory, estimates, forecasts, predictions, your Horoscope or Fairy cards to base decisions on.

1.3. 3. Quantifiable A/B web site testing provides actual numbers that can be compared, sliced and diced to evaluate results. Interaction, conversion, number of abandonments – all those numbers are accessible during and after testing. No guessing required!

1.4. 4. Accurate Unlike other forms of web site testing, A/B testing is 100% accurate ASSUMING you have statistically significant data. Understanding error rate and statistical significance and all those other statistics terms you were supposed to be learning in Statistics class is very important. You were paying attention in Statistics class, right? If not, find someone who was and have them examine your results before assuming you’ve got accurate data

2. Disadvantages

2.1. 1. Can Hurt Web Site Results Unless you always win at everything you do (in which case I’m instantly suspicious of you, or want to go to Las Vegas with you – either way) you’re going to make some bad decisions from time to time.

2.2. 2. Missing the “Why” Have you ever noticed a dog or cat staring blankly into space, at apparently absolutely nothing. WHY are they doing that?! What could possibly be in their minds?! Well, that’s the same feeling you’ll get when you use A/B testing. A/B web site testing does not explore the rationale behavioral decisions that are being made by the web site visitors.

2.3. 3. Not Predictive A/B testing is great and all, but it can’t be used to predict future design change impacts. To a certain extent this means that you’re always stuck doing A/B testing. At some point it would be handy to be able to predict if a whole new web page, web site or application will (or won’t) work, based on fairly accurate predictions of use, without the hassle of actually having to create the web page, web site or application then test it.

2.4. 4. Needs Traffic In order to provide quick, consistent and reliable results, you’re going to need a pretty good amount of traffic to your web page to run an A/B test. Remember that for the time the test is running, that traffic split will be siphoning off a percentage of your visitors from the existing page to the test page.

3. What Is A/B Testing?

3.1. At its core, A/B testing is exactly what it sounds like: you have two versions of an element (A and B) and a metric that defines success. To determine which version is better, you subject both versions to experimentation simultaneously. In the end, you measure which version was more successful and select that version for real-world use.

3.2. This is similar to the experiments you did in Science 101. Remember the experiment in which you tested various substances to see which supports plant growth and which suppresses it. At different intervals, you measured the growth of plants as they were subjected to different conditions, and in the end you tallied the increase in height of the different plants.

4. What to Test?

4.1. •The call to action’s (i.e. the button’s) wording, size, color and placement,

4.2. •Headline or product description,

4.3. •Form’s length and types of fields,

4.4. •Layout and style of website,

4.5. •Product pricing and promotional offers,

4.6. •Images on landing and product pages,

4.7. •Amount of text on the page (short vs. long).