Image

Chapter 3

The Difference Between AB Testing and Multivariate Testing

Written By : Khalid Saleh

Editor Note: We highly recommend that you implement the different ideas in this blog post through AB testing. By doing that, you will figure out which of these ideas works for your website visitors and which don’t.  Download Invesp’s “The Essentials of Multivariate & AB Testing” now to start your testing program on the right foot.

A/B and multivariate testing share the same concept: splitting visitors between several designs of your website or webpage in order to determine which of these designs generates more conversions.

If you are deciding which type of test best suits your conversion rate optimization, read on to find out more about particular characteristics of A/B and multivariate testing.

Essentials of AB Testing

Essentials of AB Testing

Discover the best scenarios for each test and check five must implement steps when conducting split tests.

Editor Note: We highly recommend that you implement the different ideas in this blog post through AB testing. By doing that, you will figure out which of these ideas works for your website visitors and which don’t.  Download Invesp’s “The Essentials of Multivariate & AB Testing” now to start your testing program on the right foot.

AB Testing

Editor Note: We highly recommend that you implement the different ideas in this blog post through AB testing. By doing that, you will figure out which of these ideas works for your website visitors and which don’t.  Download Invesp’s “The Essentials of Multivariate & AB Testing” now to start your testing program on the right foot.

AB testing

AB Testing (or Split testing) allows you to test one or more variations for each page on your website against each other.

For example, you can create different designs (layouts) for an e-commerce product page to test whether the product image should be placed on the left or the right side of the page.

Essentials of AB Testing

For a lead generation website, you can use AB testing to create multiple designs of a landing page switching headlines or images.

For a blog subscription page, you can test different designs displaying assorted color and text on the subscribe button.

Essentials of AB Testing

There are two distinct approaches to conducting AB tests:

  1. Testing new, radical designs of an entire page or process. As an example, testing three entirely different and distinct designs of a home page;
  2. Testing small changes on a page: each design makes a slight change to the same element. For example, each variation will test different colors of a call to action button.

Smaller split tests (where you are testing one or two variations against an original design) do NOT require a lot of traffic or conversions to conduct. On the other hand, when using AB testing to run radical designs against each other, you need significant traffic before you can identify what element causes a change in your conversion rate.

Editor Note: We highly recommend that you implement the different ideas in this blog post through AB testing. By doing that, you will figure out which of these ideas works for your website visitors and which don’t.  Download Invesp’s “The Essentials of Multivariate & AB Testing” now to start your testing program on the right foot.

AB testing is excellent for measuring the impact of two to five new designs compared to the original. The more designs you introduce to a test, the longer the test will take to conclude.

Multivariate Testing

Editor Note: We highly recommend that you implement the different ideas in this blog post through AB testing. By doing that, you will figure out which of these ideas works for your website visitors and which don’t.  Download Invesp’s “The Essentials of Multivariate & AB Testing” now to start your testing program on the right foot.

multivariate testing

Multivariate testing allows you to test several variations of multiple elements on a webpage at once.

For example, you can test multiple options of the product name on an e-commerce product page, along with various alternatives of the product image and diverse types of add-to-cart buttons.

Essentials of AB Testing

You can also use multivariate testing to evaluate various headlines and several designs of the CTA on a lead generation landing page.

MVT testing allows you to zoom in and focus on the changes you are making at an element level.

Essentials of AB Testing

While multivariate testing is great in doing element level analysis, it has three drawbacks:

  1. It requires intense traffic or conversions to conclude a test;
  2. If you are not careful when designing your MVT test, you can end up testing thousands of designs against the original;
  3. Since multiple elements are simultaneously switched within a single test, it is hard to isolate the exact reason a particular design performs a certain way.

MVT testing is very powerful. However, you must be careful when using it to optimize your conversion rates. Most companies forget about the large volume of visitors and conversions required to complete a test. As a result, they find themselves running tests for several weeks without bringing them to a conclusion.

Editor Note: We highly recommend that you implement the different ideas in this blog post through AB testing. By doing that, you will figure out which of these ideas works for your website visitors and which don’t.  Download Invesp’s “The Essentials of Multivariate & AB Testing” now to start your testing program on the right foot.

Additionally, since testing software allows them to do so, many companies tend to alternate elements randomly without thinking about the rationale behind the change. This mistake alone is enough to kill the benefits of any testing program.

Multivariate Test vs. an A/B Test: what type of test should you use?

Editor Note: We highly recommend that you implement the different ideas in this blog post through AB testing. By doing that, you will figure out which of these ideas works for your website visitors and which don’t.  Download Invesp’s “The Essentials of Multivariate & AB Testing” now to start your testing program on the right foot.

Should you start with an A/B or multivariate test for a particular page? There is no correct answer to this question. A/B tests are good for testing alternate designs of an entire page or a process. We usually recommend using them while deciding high-level changes or radical changes to the optimized area. MVT testing, on the other hand, allows for fine-grained testing on a particular page. They are helpful in determining the most impactful values on visitors.

AB testing is highly recommended in the following scenarios:

Essentials of AB Testing
  • If you are just starting out with the testing process;
  • If you have limited number of website visitors;
  • If you have limited number of conversions;
  • If you are looking for radical departure from your existing designs;
  • If you are starting with a new page (you do not have a page that has been launched for some time).

Multivariate Testing is highly recommended in the following scenarios:

Essentials of AB Testing
  • If you have been conducting AB tests for at least 6-8 months;
  • If you have large number of monthly visitors (more than 100,000 visitors);
  • If you have large number of monthly conversions (more than 1200 conversions per month);
  • If you are looking to fine tune existing designs;
  • If you already have launched your landing page.

For most companies, we recommend starting with an AB experiment to assess major design changes to the website.

However, if you already have an existing page, we start with a small multivariate test (less than 12 or so different scenarios). The goal of this initial test is to determine which of the elements (headline, image, benefit list, etc.) resonates most with visitors. Analysis of the first test results will help guide the need for further MVT or A/B tests.

Editor Note: We highly recommend that you implement the different ideas in this blog post through AB testing. By doing that, you will figure out which of these ideas works for your website visitors and which don’t.  Download Invesp’s “The Essentials of Multivariate & AB Testing” now to start your testing program on the right foot.

Five must implement steps when conducting testing on your website:

  1. Determine which of your website pages are good candidates for optimization, by finding which of your pages are leaking the most visitors. Calculate the commercial revenue you can gain from fixing the leak.
  2. Check the number of visitors to your webpage. The number of visitors who will go through the actual webpage or process tested impacts the agility of the test results. Although your site may have 500,000 visitors a month, a particular page you want to test may receive fewer
  3. Do not run your tests for more than four weeks.
  4. Pick the right conversion rate. Not every test should have the goal of increasing the macro conversion Many successful tests help in increasing micro conversion rates. This is particularly important if your website or landing age does not receive enough conversions. Starting out with micro conversion tests allows you to conduct tests on a smaller scale.
  5. Examine what elements you should test. Not all elements on a page will have the same impact on your conversion rate. Determine which elements will have the most impact on your bottom line based on marketing data, personas, and analytics.

Join 25,000+ Marketing Professionals

If you enjoyed this post, please consider subscribing to the Invesp blog feed to have future articles delivered to your feed reader. or,receive weekly updates by email: