Editor note: Please note that we use button colors, fonts, and headlines as a way to clarify the concept of testing. Successful AB and multivariate tests will include more sophisticated changes to your page.
Multivariate Testing or MVT testing is the process of testing multiple variations of multiple elements on a webpage with the goal of determining the best combination of different elements on the page to increase conversions.
By using MVT testing software, you can test different variations of any element on your page (headlines, images, buttons, etc.) to measure their impact on your conversion rates. The following image displays an example of how MVT testing software works.
- The original headline is tested against three other possible headlines, for a total of four possible headlines on the page
- The original image is tested against two other possible photos, for a total of three possible pictures on the page
- Three different buttons are tested against the original button on the page, for a total of four possible buttons on the page
As a visitor arrives at a page, the software picks one of the four headlines, one of the three images, and one of the four buttons to display.
Your team does not have to create all of the 48 designs; the software will swap the different variations and create the designs automatically and create all 48 possible variations. The following image shows four of the 48 possible designs the testing software can generate.
You can calculate the total number of challengers in a multivariate test multiplying the number of different variations of each of the elements.
For a webpage in which we will be testing (N) number of elements, we calculate:
The number of page variations can grow very fast. Some testing software allows you to tens of thousands (sometimes millions) of variations of a single page.
MVT Test Examples
Let’s take the product page from apple.com as an example:
- different variations of the headline
- displaying two MacBook models per line (currently, each model takes a line)
- different product images
- different pricing
- CTA colors
- CTA text
Let’s take an example from salesforce.com:
On this page, you can test:
- different variations of the headline
- displaying the side navigation or not showing it
- different hero images
- CTA colors
- CTA text
A/B Tests vs Multivariate Tests
Suppose you’re just getting started with running tests, it can be pretty confusing. On one hand, you’ve got A/B tests, on the other you’ve got multivariate testing. You don’t have to be confused about which to go with, here are the differences.
How to create a successful multivariate test?
Multivariate testing software allows marketers to create and start simple tests in a few hours.
But that is the easy part!
Many companies ultimately fail when designing successful test scenarios, assessing results, and creating meaningful follow-up tests.
Poorly designed experiments can take years to conclude. Even worse, they might not provide accurate insights into what elements convert more visitors into customers.
Imagine a case where you plan to test different headlines on a page. You start by coming up with ten different possible variations to the headlines. Which of these ten possible headlines should you test against your original headline? Why not test all of them? Why not test variations of images, buttons, and layouts?
You will most likely find yourself relying on guesswork to determine which versions to include in the test. The same logic, of course, applies to all elements you want to test on a page.
Without being judicious with test scenarios, you might end up attempting to test millions of combinations.
Testing is an essential component of any conversion optimization project. However, it should not be the only component. Testing should only take place after the conclusion of other equally critical optimization stages, such as persona development, voice of customer research (including polls and surveys), heuristic evaluation, usability testing, site analysis, and design and copy creation. Each of these elements provides a building block towards a highly optimized website that converts visitors into customers.
To create a successful test, you must go through the following steps:
- Evaluate the page, looking for possible problems in it
- Prioritize the issues identified on the page in terms of their impact on your conversion rate
- Create a hypothesis of how to fix some of the top issues on the page and the effect your fix will have on your conversion rate
- Assert the validity of your hypothesis through multivariate or AB testing
- Analyze the results of the test to determine the correctness of the test hypothesis
- Create a new test based on the test result.
The results from running multivariate testing
While MVT testing is powerful in helping online business increase conversions rates, the results you will achieve from running a single test may vary.
You can choose different approaches to design and create your multivariate test:
1. Element level testing: In this type of testing, you test different variations of an element on the page. For example, you test different headline variations or several images. The goal of an “element level test” is to measure the impact of that element on your conversion rate.
Element level testing is considered the easiest type of testing. It requires the least amount of effort. And in most cases, element level testing has minimum impact on your website conversion rates.
2. Page level testing: in this type of testing, you test multiple page elements at the same time. As an example, you can test different page layouts, and/or a different combination of elements and so on. Page level testing requires more effort from the development team to implement and it generates a higher impact on your conversion rates compared to element level testing.
Carefully designed page-level testing can produce anywhere from 10% to 20% increase in conversion rates.
3. Visitor flow testing: in this type of testing, you test several navigation paths for visitors within your website. As an example, an e-commerce website might test single step vs. multi-step checkout. Another example is to test different ways visitors can navigate from category pages to product pages.
Visitor flow testing can get complicated quickly. It typically requires a higher level of effort from your development team to implement. Done correctly, this type of testing will have a higher impact on your conversion rates compared to page-level testing.
Full Factorial or Fractional Factorial MVT. Which is best?
When people talk about multivariate testing, they’re usually referring to the full-factorial mvt. In this type of multivariate test, the traffic is distributed equally among every variation.
Let’s say for instance you’ve got 10 variations based on the number of variables you’re testing and the page has 1000 unique visitors. When you do your math, an equal distribution of traffic here means every variation gets 100 visitors.
Now, because each variation gets the same amount of traffic, this test type is best for determining which particular variation performed best.
Much more than finding out the winning variation, it allows you to single out the element in the variation that had the most impact in improving the conversion rate.
It’s important you know that in a winning variation, not all elements perform equally. The position of the testimonials might have had the most impact on the winning variation, while the headline pulled no weight.
Partial or fractional factorial mvt:
The name gives it away. Unlike the full factorial testing that requires all of the variables to get traffic to drive result, with fractional factorial, only a subset of the variations gets the traffic.
The other variations don’t get traffic while their conversion rates are inferred from the ones that got the traffic.
This mvt testing type requires some hard maths for conversion rate inference and assumptions for the variaitons that didn’t get traffic.
Professional tip: I’ll advise you always go with the full-factorial multivariate test. This provides you with data that is better than inferences.
Do’s and Don’ts Of Multivariate Testing
1. Decide which sections should be included in the test: not every element has the same impact on the conversion rate.
Suppose you included a headline, a testimonial section and a footer, you might come to realize that the footer section has little to no impact on the conversion rate of that page or user engagement.
It’s an important factor in the conversion impact of elements and sections.
2. Preview every combination: this is a mistake even mature experimenters make atimes. They forget to preview the product of the element variations.
This is important because you don’t want to have a variation where the header reads 20% off while the call-to-action button reads free samples. Both messages in this variation are incompatible. Previewing helps you detect these errors and remove them.
3. Estimate the traffic for significant results: having ten variations for a page that gets a hundred visitors will take a lot of time to achieve statistical significance.
To avoid running tests that the results would be invalid before it’s ready, learn to estimate the amount of traffic that will be required.
Here’s a simple method to use. Use you web analytics to get an idea of the traffic that page gets. Secondly, know how many sections/elements you want to test. Total number of page variations = Number of variations of 1st element x Number of variations of the 2nd element x Number of variations of the 3rd element x …x Number of variations of the Nth element.
Now, divide the number of traffic by the variations. If the number of traffic you get is small, then a multivariate test might not be a good fit for that page.
1. Don’t include a lot of sections or elements: the more elements and sections you’re testing, the more variations you get. The reason this is a big deal is that when you test a lot of elements that increase the number of variations, you’ll need a lot more traffic to get statistical significance.
The Pros Of Multivariate Testing
1. Ability to test more variations.
2. You better understand the impact of individual elements on conversion rate.
3. Saves your time because you don’t have to conduct individual A/B tests.
The downsides of multivariate testing
If you are not careful with planning your multivariate tests, you will end up with weak-quality tests that take too long to implement and produce neither results nor insights.
You must always remember that testing (AB or multivariate) is only one component of a conversion rate optimization work.
We have seen many companies that entirely relied on testing software without doing an in-depth analysis of what they were testing. Our 2007 article on the case against multivariate testing points out this example:
Let’s do some simple math.
Say you want to test six different elements on a page (headers, benefits list, hero shots, call to action, etc).
For each element, you will choose four different options. This means you will have a total of 4^6 = 4,096 possible scenarios that you will have to test.
As a general rule of thumb [being more aggressive], you will need around 200 conversions per scenario to ensure the data you are collecting is statistically significant. This translates into 4,096 * 200= 819,200 conversions.
If your website converts around 1%, you will need 819,200 * 100=81,920,000 visitors before you start gaining some confidence in your test results.
If testing 4,096 variations sound difficult, imagine how complicated matters will get by adding variation in campaigns, offers, products, and keywords. Yes, running that many test variatins is not unheard of for many larger websites.
When creating an MVT test, keep these possible problems in mind:
1. Be aware of creating the test without paying close attention to the hypothesis behind it
2. Be mindful of the number of variables you are testing and their dependency on one another
3. Be aware of the length of time it will take to complete the test to a statistical significance
4. Traffic allocation: as the possible number of variations goes up, you’ll need more traffic to achieve statistical significance.
In the case of an A/B test, you could easily split the traffic 50-50 between the control and variation, for a multivariate test, it’s not the same. Because you’ve got more variations to test, it’ll take a longer time to reach statistical significance because the traffic won’t be evenly split, it could be 5% to a variation, 10% to another variation, etc.
Professional tip: Before running the MVT project the sample size per variation. If the traffic on the page you want to test won’t be good enough to achieve statistical significance, I suggest you go for an A/B test.
5. Complexity in analyzing results: A/B tests are simpler to understand, especially in the analysis of the result. You have ann hypothesis driving the A/B test, you can easily deduce why the control won or didn’t.
This is not the case for a multivariate test. A variation has many elements working at the same time, so they analyze the result requires some mental gymnastics because you need to explain how the individual elements interact.