Chapter 3

The Difference Between AB Testing and Multivariate Testing

Written By : Khalid Saleh

A/B and multivariate testing share the same concept: splitting visitors between several designs of your website or webpage in order to determine which of these designs generates more conversions.

If you are deciding which type of test best suits your conversion rate optimization, read on to find out more about particular characteristics of A/B and multivariate testing.

Discover the best scenarios for each test and check five must implement steps when conducting split tests.

AB Testing

AB testing

AB Testing (or Split testing) allows you to test one or more variations for each page on your website against each other.

For example, you can create different designs (layouts) for an e-commerce product page to test whether the product image should be placed on the left or the right side of the page.

For a lead generation website, you can use AB testing to create multiple designs of a landing page switching headlines or images.

For a blog subscription page, you can test different designs displaying assorted color and text on the subscribe button.

There are two distinct approaches to conducting AB tests:

  1. Testing new, radical designs of an entire page or process. As an example, testing three entirely different and distinct designs of a home page;
  2. Testing small changes on a page: each design makes a slight change to the same element. For example, each variation will test different colors of a call to action button.

Smaller split tests (where you are testing one or two variations against an original design) do NOT require a lot of traffic or conversions to conduct. On the other hand, when using AB testing to run radical designs against each other, you need significant traffic before you can identify what element causes a change in your conversion rate.

AB testing is excellent for measuring the impact of two to five new designs compared to the original. The more designs you introduce to a test, the longer the test will take to conclude.

Multivariate Testing

multivariate testing

Multivariate testing allows you to test several variations of multiple elements on a webpage at once.

For example, you can test multiple options of the product name on an e-commerce product page, along with various alternatives of the product image and diverse types of add-to-cart buttons.

You can also use multivariate testing to evaluate various headlines and several designs of the CTA on a lead generation landing page.

MVT testing allows you to zoom in and focus on the changes you are making at an element level.

While multivariate testing is great in doing element level analysis, it has three drawbacks:

  1. It requires intense traffic or conversions to conclude a test;
  2. If you are not careful when designing your MVT test, you can end up testing thousands of designs against the original;
  3. Since multiple elements are simultaneously switched within a single test, it is hard to isolate the exact reason a particular design performs a certain way.

MVT testing is very powerful. However, you must be careful when using it to optimize your conversion rates. Most companies forget about the large volume of visitors and conversions required to complete a test. As a result, they find themselves running tests for several weeks without bringing them to a conclusion.

Additionally, since testing software allows them to do so, many companies tend to alternate elements randomly without thinking about the rationale behind the change. This mistake alone is enough to kill the benefits of any testing program.

Multivariate Test vs. an A/B Test: what type of test should you use?

Should you start with an A/B or multivariate test for a particular page? There is no correct answer to this question. A/B tests are good for testing alternate designs of an entire page or a process. We usually recommend using them while deciding high-level changes or radical changes to the optimized area. MVT testing, on the other hand, allows for fine-grained testing on a particular page. They are helpful in determining the most impactful values on visitors.

AB testing is highly recommended in the following scenarios:

  • If you are just starting out with the testing process;
  • If you have limited number of website visitors;
  • If you have limited number of conversions;
  • If you are looking for radical departure from your existing designs;
  • If you are starting with a new page (you do not have a page that has been launched for some time).

Multivariate Testing is highly recommended in the following scenarios:

  • If you have been conducting AB tests for at least 6-8 months;
  • If you have large number of monthly visitors (more than 100,000 visitors);
  • If you have large number of monthly conversions (more than 1200 conversions per month);
  • If you are looking to fine tune existing designs;
  • If you already have launched your landing page.

For most companies, we recommend starting with an AB experiment to assess major design changes to the website.

However, if you already have an existing page, we start with a small multivariate test (less than 12 or so different scenarios). The goal of this initial test is to determine which of the elements (headline, image, benefit list, etc.) resonates most with visitors. Analysis of the first test results will help guide the need for further MVT or A/B tests.

Five must implement steps when conducting testing on your website:

  1. Determine which of your website pages are good candidates for optimization, by finding which of your pages are leaking the most visitors. Calculate the commercial revenue you can gain from fixing the leak.
  2. Check the number of visitors to your webpage. The number of visitors who will go through the actual webpage or process tested impacts the agility of the test results. Although your site may have 500,000 visitors a month, a particular page you want to test may receive fewer
  3. Do not run your tests for more than four weeks.
  4. Pick the right conversion rate. Not every test should have the goal of increasing the macro conversion Many successful tests help in increasing micro conversion rates. This is particularly important if your website or landing age does not receive enough conversions. Starting out with micro conversion tests allows you to conduct tests on a smaller scale.
  5. Examine what elements you should test. Not all elements on a page will have the same impact on your conversion rate. Determine which elements will have the most impact on your bottom line based on marketing data, personas, and analytics.

Join 25,000+ Marketing Professionals

If you enjoyed this post, please consider subscribing to the Invesp blog feed to have future articles delivered to your feed reader. or,receive weekly updates by email: