The Definitive Guide to

Multivariate and AB Testing

By Khalid Saleh & Ayat Shukairy


Most websites continue to struggle with increasing their online conversion rates.

On average, an e-commerce store converts around 3% of its traffic into customers. The remaining 97% of the visitors leave the website without ever placing an order.

Other types of websites struggle with conversion as well. On average, a lead generation or subscription websites convert around 15% of its traffic into customers. The remaining 85% leave the website without ever placing an order.

If you are looking to increase your website conversion rate, you will have to conduct AB and multivariate testing. The problem is that most testing programs start without any planning and thus fail with the first 6 months.

We know this first hand because we have been helping online companies conduct testing for over 12 years. Our goal from this guide to help you with a good introduction to multivariate and AB testing. We will continue updating the different chapters in the book on regular basis, so if you have comments or feedback, please let’s know (@invesp).

Chapter 1

What is A/B testing (split testing)?

If visitors are not converting on your website, then obviously, there is something wrong that is stopping them.

You can go ahead and ask your design team to create new designs but the question remains: how do you know that the new designs will convert more visitors compared to the original design?

That is where AB testing comes in handy.

AB testing (sometimes referred to as split testing) is the process of testing multiple variations or designs of webpage against the original page with the goal of determining which page generates more conversions.

The original design of a page is

Take Me To Chapter 1

Chapter 2

What is Multivariate Testing?

Multivariate Testing or MVT testing is the process of testing multiple variations of multiple elements on a webpage with the goal of determining the best combination of elements to increase conversions.

By using MVT testing software, you can test different variations of any element on your page (headlines, images, buttons, etc) to measure their impact on your conversion rates. The following image displays an example of how MVT testing software works.

In this example, the software tests different variations of the page headline, image and call to action button: •  The original headline against three other possible headlines, for

Take Me To Chapter 2

Chapter 3

The Difference Between AB Testing and Multivariate Testing

A/B and multivariate testing share the same concept: splitting visitors between several designs of your website or webpage in order to determine which of these designs generates more conversions.

If you are deciding which type of test best suits your conversion rate optimization, read on to find out more about particular characteristics of A/B and multivariate testing.

Discover the best scenarios for each test and check five must implement steps when conducting split tests.


AB Testing

AB Testing (or Split testing) allows you to test one or more variations for each page on your website against each

Take Me To Chapter 3

Chapter 4

Comparing Multivariate & AB Testing Tools

On the journey of increasing the conversation rate on your website, you need the precise help of your visitors and clients’ input. You need to evaluate the efficacy of hundreds of possible designs and compare dozens of different results.

You have heard there are tools out there to help you in your quest, but staggering questions haunt your decision.

•  What is the right split testing tool for my website?
•  By choosing a particular tool, will I affect the success of my testing or conversion optimization program?

From simple free tools to complex block testing based software, you

Take Me To Chapter 4

Chapter 5

The Process of Creating a Successful A/B Test

First, do not rely solely on a testing software to create successful tests.

You need people who can design efficient test scenarios, analyze results accurately, and create meaningful follow-up tests.

Poorly designed experiments might not provide concrete insights to conversion rates optimization. You need criteria to determine, for example, which elements on a page you should test, which external factors could affect the results, and in which ways to rearrange the designs for new phases of the test.

As much as testing is essential to any optimization project, it should only be conducted after the completion of equally important stages

Take Me To Chapter 5

Chapter 6

What Elements Should You Test In An A/B Test?

As you start the process of multivariate or AB testing, one of the important questions you will face is:
What elements to change and test on a page?

This article will suggest many elements that you can consider when creating a test. However, before you jump into selecting elements and creating variations for them, ask yourself:

Why are you selecting a particular element? Why are you testing the headline as opposed to the hero image? Why not test the benefits list or the call to action on the page?

The process of selecting elements to test on a page

Take Me To Chapter 6

Chapter 7

Multivariate and A/B Testing Best Practices

Conducting a multivariate test is exciting.

By using the right tool, you can quickly develop different designs for your website, direct visitors to each design and watch the conversions for each variation.

Done incorrectly, AB testing can result in a waste of money, a misuse of man-hours, and, even worse, a decrease in your conversion rates.

Here are nine best practices you must follow when conducting a multivariate test.


Planning Stage 1.  Set expectations correctly

Wrong expectations translate into disappointment and lost investment.

Many marketers start testing because they heard or watched a case study where a company achieved

Take Me To Chapter 7

Chapter 8

14 Beginner Mistakes That Will Kill Your A/B Testing (and what you can do about them)

AB testing mistakes

More and more companies are turning to AB testing to increase their online conversion rates.

The scene has changed a lot since we first started doing conversion optimization over ten years ago. Back then, it was difficult to explain to companies the impact of testing on websites’ performances. However, still today, most companies invest primarily in visitor-driving activities and set lower budgets to converting visitors into customers. CRO and split testing budgets in these cases represent a portion equivalent to 10% of the resources devoted to visitor driving.

If your company is starting with A/B or multivariate testing, the following

Take Me To Chapter 8