Editor note: Please note that we use button colors, fonts, headlines as a way to clarify the concept of testing. Succesful AB and multivariate tests will include more sophisticated changes to your page.

Multivariate Testing or MVT testing is the process of testing multiple variations of multiple elements on a webpage with the goal of determining the best combination of different elements on the page to increase conversions.

By using MVT testing software, you can test different variations of any element on your page (headlines, images, buttons, etc.) to measure their impact on your conversion rates. The following image displays an example of how MVT testing software works.

What is Multivariate Testing
In this example, the software tests different variations of the page headline, image, and the “call to action” button:
  • The original headline is tested against three other possible headlines, for a total of four possible headlines on the page
  • The original image is tested against two other possible photos, for a total of three possible pictures on the page
  • Three different buttons are tested against the original button on the page, for a total of four possible buttons on the page

As a visitor arrives at a page, the software picks one of the four headlines, one of the three images, and one of the four buttons to display.

Your team does not have to create all of the 48 designs; the software will swap the different variations and create the designs automatically and create all 48 possible variations. The following image shows four of the 48 possible designs the testing software can generate.

mvt-design-variations
The total number of testing variations (also called challengers) depends on the number of elements you will test on a page (headline, image, buttons, etc.) and the number of variations you will be testing for each of these elements.

You can calculate the total number of challengers in a multivariate test multiplying the number of different variations of each of the elements.

For a webpage in which we will be testing (N) number of elements, we calculate:

Total number of page variations = Number of variations of 1st element  x Number of variations of the 2nd element x Number of variations of the 3rd element x …x Number of variations of the Nth element

The number of page variations can grow very fast. Some testing software allows you to tens of thousands (sometimes millions) of variations of a single page.

MVT Test Examples

Let’s take the product page from apple.com as an example:

apple-multivariate-testing-example On this page, you can test:
  • different variations of the headline
  • displaying two MacBook models per line (currently, each model takes a line)
  • different product images
  • different pricing
  • CTA colors
  • CTA text

Let’s take an example from salesforce.com:

Essentials of AB Testing

salesforce-multivariate-testing-example On this page, you can test:

  • different variations of the headline
  • displaying the side navigation or not showing it
  • different hero images
  • CTA colors
  • CTA text

How do to create a successful multivariate test?

Multivariate testing software allows marketers to create and start simple tests in a few hours.

But that is the easy part!

Many companies ultimately fail when designing successful test scenarios, assessing results, and creating meaningful follow-up tests.

Poorly designed experiments can take years to conclude. Even worse, they might not provide accurate insights into what elements convert more visitors into customers.

Imagine a case where you plan to test different headlines on a page. You start by coming up with ten different possible variations to the headlines. Which of these ten possible headlines should you test against your original headline? Why not test all of them? Why not test variations of images, buttons, and layouts?

You will most likely find yourself relying on guesswork to determine which versions to include in the test. The same logic, of course, applies to all elements you want to test on a page.

Without being judicious with test scenarios, you might end up attempting to test millions of combinations.

Testing is an essential component of any conversion optimization project. However, it should not be the only component. Testing should only take place after the conclusion of other equally critical optimization stages, such as persona development, voice of customer research (including polls and surveys), heuristic evaluation, usability testing, site analysis, and design and copy creation. Each of these elements provides a building block towards a highly optimized website that converts visitors into customers.

To create a successful test, you must go through the following steps:

  • Evaluate the page, looking for possible problems in it
  • Prioritize the issues identified on the page in terms of their impact on your conversion rate
  • Create a hypothesis of how to fix some of the top issues on the page and the effect your fix will have on your conversion rate
  • Assert the validity of your hypothesis through multivariate or AB testing
  • Analyze the results of the test to determine the correctness of the test hypothesis
  • Create a new test based on the test result.

The results from running multivariate testing

While MVT testing is powerful in helping online business increase conversions rates, the results you will achieve from running a single test may vary.

You can choose different approaches to design and create your multivariate test:

1.  Element level testing: In this type of testing, you test different variations of an element on the page. For example, you test different headline variations or several images. The goal of an “element level test” is to measure the impact of that element on your conversion rate.

3M Results

Element level testing is considered the easiest type of testing. It requires the least amount of effort. And in most cases, element level testing has minimum impact on your website conversion rates.

2.  Page level testing: in this type of testing, you test multiple page elements at the same time. As an example, you can test different page layouts, and/or a different combination of elements and so on. Page level testing requires more effort from the development team to implement and it generates a higher impact on your conversion rates compared to element level testing.

Carefully designed page-level testing can produce anywhere from 10% to 20% increase in conversion rates.

3. Visitor flow testing: in this type of testing, you test several navigation paths for visitors within your website. As an example, an e-commerce website might test single step vs. multi-step checkout. Another example is to test different ways visitors can navigate from category pages to product pages.

Visitor flow testing can get complicated quickly. It typically requires a higher level of effort from your development team to implement. Done correctly, this type of testing will have a higher impact on your conversion rates compared to page level testing.

The dangers of multivariate testing

If you are not careful with planning your multivariate tests, you will end up with weak quality tests that take too long to implement and produce neither results nor insights.

You must always remember that testing (AB or multivariate) is only one component of a conversion rate optimization work.

We have seen many companies that entirely relied on testing software without doing an in-depth analysis of what they were testing. Our 2007 article on the case against multivariate testing points out this example:

Let’s do some simple math.

Say you want to test six different elements on a page (headers, benefits list, hero shots, call to action, etc).

For each element, you will choose four different options. This means you will have a total of 4^6 = 4,096 possible scenarios that you will have to test.

As a general rule of thumb [being more aggressive], you will need around 200 conversions per scenario to ensure the data you are collecting is statistically significant. This translates into 4,096 * 200= 819,200 conversions.

If your website converts around 1%, you will need 819,200 * 100=81,920,000 visitors before you start gaining some confidence in your test results.

If testing 4,096 variations sound difficult, imagine how complicated matters will get by adding variation in campaigns, offers, products, and keywords. Yes, running that many test variatins is not unheard of for many larger websites.

When creating an MVT test, keep these possible problems in mind:

  • Be aware of creating the test without paying close attention to the hypothesis behind it
  • Be mindful of the number of variables you are testing and their dependency on one another
  • Be aware of the length of time it will take to complete the test to a statistical significance