1. What Is A/B Testing?
1.1. At its core, A/B testing is exactly what it sounds like: you have two versions of an element (A and B) and a metric that defines success. To determine which version is better, you subject both versions to experimentation simultaneously. In the end, you measure which version was more successful and select that version for real-world use.
1.2. This is similar to the experiments you did in Science 101. Remember the experiment in which you tested various substances to see which supports plant growth and which suppresses it. At different intervals, you measured the growth of plants as they were subjected to different conditions, and in the end you tallied the increase in height of the different plants.
1.3. A/B testing on the Web is similar. You have two designs of a website: A and B. Typically, A is the existing design (called the control), and B is the new design. You split your website traffic between these two versions and measure their performance using metrics that you care about (conversion rate, sales, bounce rate, etc.). In the end, you select the version that performs best.
2. What To Test?
2.1. Your choice of what to test will obviously depend on your goals. For example, if your goal is to increase the number of sign-ups, then you might test the following: length of the sign-up form, types of fields in the form, display of privacy policy, “social proof,” etc. The goal of A/B testing in this case is to figure out what prevents visitors from signing up. Is the form’s length intimidating? Are visitors concerned about privacy? Or does the website do a bad job of convincing visitors to sign up? All of these questions can be answered one by one by testing the appropriate website elements.
2.2. Even though every A/B test is unique, certain elements are usually tested.
2.2.1. The call to action’s (i.e. the button’s) wording, size, color and placement,
2.2.2. Headline or product description,
2.2.3. Form’s length and types of fields,
2.2.4. Layout and style of website,
2.2.5. Product pricing and promotional offers,
2.2.6. Images on landing and product pages,
2.2.7. Amount of text on the page (short vs. long).
3. Create Your First A/B Test
3.1. Once you’ve decided what to test, the next step, of course, is to select a tool for the job. If you want a free basic tool and don’t mind fiddling with HTML and JavaScript, go with Google Website Optimizer. If you want an easier alternative with extra features, go with Visual Website Optimizer (disclaimer: my start-up). Other options are available, which I discuss at the end of this post. Setting up the core test is more or less similar for all tools, so we can discuss it while remaining tool-agnostic.
3.2. You can set up an A/B test in one of two ways.
3.2.1. Replace the element to be tested before the page loads
3.2.1.1. If you are testing a single element on a Web page—say, the sign-up button—then you’ll need to create variations of that button (in HTML) in your testing tool. When the test is live, the A/B tool will randomly replace the original button on the page with one of the variations before displaying the page to the visitor.
3.2.2. Redirect to another page
3.2.2.1. If you want to A/B test an entire page—say, a green theme vs. a red theme—then you’ll need to create and upload a new page on your website. For example, if your home page is http://www.example.com/index.html, then you’ll need to create a variation located at http://www.example.com/index1.html. When the test runs, your tool will redirect some visitors to one of your alternate URLs.
3.3. Once you have set up your variations using one of these two methods, the next step is to set up your conversion goal. Typically, you will get a piece of JavaScript code, which you would copy and paste onto a page that would represent a successful test were a visitor to arrive there. For example, if you have an e-commerce store and you are testing the color of the “Buy now” button, then your conversion goal would be the “Thank you” page that is displayed to visitors after they complete a purchase.
3.4. As soon as a conversion event occurs on your website, the A/B testing tool records the variation that was shown to the visitor. After a sufficient number of visitors and conversions, you can check the results to find out which variation drove the most conversions. That’s it! Setting up and running an A/B test is indeed quite simple.
4. Do’s And Don’ts
4.1. Even though A/B testing is super-simple in concept, keep some practical things in mind. These suggestions are a result of my real-world experience of doing many A/B tests (read: making numerous mistakes).
4.2. DON’TS
4.2.1. When doing A/B testing, never ever wait to test the variation until after you’ve tested the control. Always test both versions simultaneously. If you test one version one week and the second the next, you’re doing it wrong. It’s possible that version B was actually worse but you just happened to have better sales while testing it. Always split traffic between two versions.
4.2.2. Don’t conclude too early. There is a concept called “statistical confidence” that determines whether your test results are significant (that is, whether you should take the results seriously). It prevents you from reading too much into the results if you have only a few conversions or visitors for each variation. Most A/B testing tools report statistical confidence, but if you are testing manually, consider accounting for it with an online calculator.
4.2.3. Don’t surprise regular visitors. If you are testing a core part of your website, include only new visitors in the test. You want to avoid shocking regular visitors, especially because the variations may not ultimately be implemented.
4.2.4. Don’t let your gut feeling overrule test results. The winners in A/B tests are often surprising or unintuitive. On a green-themed website, a stark red button could emerge as the winner. Even if the red button isn’t easy on the eye, don’t reject it outright. Your goal with the test is a better conversion rate, not aesthetics, so don’t reject the results because of your arbitrary judgment.
4.3. DO’S
4.3.1. Know how long to run a test before giving up. Giving up too early can cost you because you may have gotten meaningful results had you waited a little longer. Giving up too late isn’t good either, because poorly performing variations could cost you conversions and sales. Use a calculator (like this one) to determine exactly how long to run a test before giving up.
4.3.2. Show repeat visitors the same variations. Your tool should have a mechanism for remembering which variation a visitor has seen. This prevents blunders, such as showing a user a different price or a different promotional offer.
4.3.3. Make your A/B test consistent across the whole website. If you are testing a sign-up button that appears in multiple locations, then a visitor should see the same variation everywhere. Showing one variation on page 1 and another variation on page 2 will skew the results.
4.3.4. Do many A/B tests. Let’s face it: chances are, your first A/B test will turn out a lemon. But don’t despair. An A/B test can have only three outcomes: no result, a negative result or a positive result. The key to optimizing conversion rates is to do a ton of A/B tests, so that all positive results add up to a huge boost to your sales and achieved goals.
5. Pros and Cons of A/B Testing
5.1. Pros of Web Site A/B Testing
5.1.1. Fast Of all the test types, A/B is way, way fast. That’s because it takes very little time to create a modified version of an existing web page that includes a modified item (like a new picture, new copy or other new element) and throw it up on your site. Then, it’s just a matter of splitting traffic to the two pages.
5.1.2. Tests reality, not theory The good news about A/B testing on a live web site is you’re obtaining real results from real users doing real things. That means you’re not using theory, estimates, forecasts, predictions, your Horoscope or Fairy cards to base decisions on.
5.1.3. Quantifiable A/B web site testing provides actual numbers that can be compared, sliced and diced to evaluate results. Interaction, conversion, number of abandonments – all those numbers are accessible during and after testing. No guessing required!
5.1.4. Accurate Unlike other forms of web site testing, A/B testing is 100% accurate ASSUMING you have statistically significant data. Understanding error rate and statistical significance and all those other statistics terms you were supposed to be learning in Statistics class is very important. You were paying attention in Statistics class, right? If not, find someone who was and have them examine your results before assuming you’ve got accurate data. More on this in the Cons.
5.2. Cons of Web Site A/B Testing
5.2.1. Can Hurt Web Site Results Unless you always win at everything you do (in which case I’m instantly suspicious of you, or want to go to Las Vegas with you – either way) you’re going to make some bad decisions from time to time. Remember that horrible hair style you just HAD to have that one time? Ugh!
5.2.2. Missing the “Why” Have you ever noticed a dog or cat staring blankly into space, at apparently absolutely nothing. WHY are they doing that?! What could possibly be in their minds?! Well, that’s the same feeling you’ll get when you use A/B testing. A/B web site testing does not explore the rationale behavioral decisions that are being made by the web site visitors.
5.2.3. Not Predictive A/B testing is great and all, but it can’t be used to predict future design change impacts. To a certain extent this means that you’re always stuck doing A/B testing. At some point it would be handy to be able to predict if a whole new web page, web site or application will (or won’t) work, based on fairly accurate predictions of use, without the hassle of actually having to create the web page, web site or application then test it.
5.2.4. Needs Traffic In order to provide quick, consistent and reliable results, you’re going to need a pretty good amount of traffic to your web page to run an A/B test. Remember that for the time the test is running, that traffic split will be siphoning off a percentage of your visitors from the existing page to the test page.