Chapter 2

What is Multivariate Testing?

Written By : Khalid Saleh

Multivariate Testing or MVT testing is the process of testing multiple variations of multiple elements on a webpage with the goal of determining the best combination of elements to increase conversions.

By using MVT testing software, you can test different variations of any element on your page (headlines, images, buttons, etc) to measure their impact on your conversion rates. The following image displays an example of how MVT testing software works.

What is Multivariate Testing
In this example, the software tests different variations of the page headline, image and call to action button:
•  The original headline against three other possible headlines, for a total of four possible headlines;
• The original image against two other possible images, for a total of three possible images;
•  Finally, three different buttons are tested against the original button on the page.
For this test, that one page receives 48 possible variations. The total number of testing variations (also called challengers) depends on the number of elements you will test on a page (headline, image, buttons, etc.) and the number of variations you will be testing for each of these elements.
The total number of challengers is calculated by multiplying the number of different variations of each of the elements. For a webpage in which we will be testing (N) number of elements, we calculate:
Total number of page variations = Number of variations of element (1) x Number of variations of element (2) x  …x Number of variations of element (N)
The number of page variations can grow very fast. Some testing software allows for testing tens of thousands of variations of a single page.
As a visitor arrives at a page, the software will pick one of the four headlines, one of the three images, and one of the four buttons to display.
The following image shows four of the 48 possible designs the software can create. Your team does not have to create all of the 48 designs; the software will swap the different variations and create the designs automatically.

MVT Test Examples

Let’s take the product page from as an example:

On this page,you can test:
•    different variations of the headline;
•    displaying two MacBook models per line (currently, each model takes a line);
•    different product images;
•    different pricing;
•    CTA colors;
•    CTA text.

Let’s take an example from

On this page, you can test:
•    different variations of the headline;
•    displaying the side navigation or not displaying it;
•    different hero images;
•    CTA colors;
•    CTA text.

How Do You Create A Successful Multivariate Test?

Tools are useless without the people who properly run them. Most testing software allows marketers to create and start simple tests in a few hours.
But that is the easy part!

Many companies ultimately fail when designing successful test scenarios, assessing results, and creating meaningful follow-up tests.

Poorly designed experiments can take years to complete. Even worse, they might not provide concrete insights to what elements will convert more visitors into customers.

Imagine a case where you plan to test different headlines on a page. You start by coming up with ten different possible variations to the headlines. What criteria are you going to use to determine which of the headlines you should test? Why not test all of the ten different headlines?

You will most likely find yourself relying on guesswork to determine which versions to include in the test. The same logic, of course, applies to all elements you want to test on a page.

Without being judicious with test scenarios, you might end up attempting to test millions of combinations.

Testing is an important component of any conversion optimization project. However, it should not be the only component. Testing should only take place after the conclusion of other equally important stages of optimization work, such as persona development, site analysis, and design and copy creation. Each of these elements provides a building block towards a highly optimized website that converts visitors into clients.
To create a successful test, you must go through the following steps:
•    Evaluate the page, looking for possible problematic areas in it;
•    Prioritize the problems identified on the page, in terms of their impact on your conversion rate;
•    Create a hypothesis of how to fix some of the top problems on the page;
•    Create an AB or a multivariate test to assert the validity of your test hypothesis;
•    Analyze the results of the test to determine the correctness of the test hypothesis;
•    Create a new test based on the test result.

The Results From Running A Multivariate Test

While MVT testing is powerful in helping online business increase conversions rates, the results you will achieve from running a single test may vary.

You can choose different approaches to a test design:

1.  Element level testing: In this type of testing, you test different variations of an element on the page. For example, you test different headline variations or several images. The goal of an element level test is to measure the impact of that element on your conversion rate.

Element level testing is considered the easiest type of testing. It generally requires the least amount of effort. In most cases, the impact of this type of testing is limited on conversion rates. About 90% of element level testing will produce less than 3% increase in conversion rates.

2.  Page level testing: in this type of testing, you test multiple page elements at the same time. As an example, you can test different page layouts, different combination of elements and so on. This type of testing re-quires more effort from the development team to implement.

Done correctly, page level testing might bring a higher impact on your conversion rates compared to element level testing. Well-designed page level testing can produce anywhere from 10% to 30% increase in conversion rates.

3. Visitor flow testing: in this type of testing, you test several navigation paths for the visitor around the web-site. As an example, an e-commerce website might test single step vs. multi-step checkout. Another example is to test different ways visitors can navigate from category pages to product pages.

Visitor flow testing can get complicated very quickly. It typically requires a lot of effort from the development team to implement. Done correctly, this type of testing will have the highest impact on your conversion rates.

The three design types discussed above could be implemented as a multivariate or an AB test.

The Dangers of Multivariate Testing

If you are not careful with planning your split tests, they can be one of the main reasons that reduce the quality of optimization work.

You must always remember that testing (AB or multivariate) is only one component of conversion optimization.

We have seen many companies that completely relied on testing software without doing a deep analysis of what they were actually testing. Our article on the case against multivariate testing points out this example:

Let’s do some simple math.

Say you want to test six different elements on a page (headers, benefits list, hero shots, call to action, etc).

For each element, you will choose four different options. This means you will have a total of 4^6 = 4,096 possible scenarios that you will have to test.

As a general rule of thumb, you will need around 100 conversions per scenario to ensure the data you are collecting is statistically significant. This translates into 4,096 * 100= 409,600 conversions.

If your website converts around 1%, you will need 409,600 * 100= 40,960,000 visitors before you start gaining some confidence in your testing results.

If testing 4,096 variations sounds difficult, imagine how complicated matters will get by adding variation in campaigns, offers, products, and keywords. Running this many test scenarios is not unheard of for many larger websites.

When creating an MVT test, keep these possible problems in mind:

• Be aware of the dangers of creating the test without paying close attention to the hypothesis behind it;
• Be aware of the number of variables you are testing and their dependency on one another;
• Be aware of the length of time it will take to complete the test to a statistical significance.

Join 25,000+ Marketing Professionals

If you enjoyed this post, please consider subscribing to the Invesp blog feed to have future articles delivered to your feed reader. or,receive weekly updates by email: