Editor Note: We highly recommend that you implement the different ideas in this blog post through AB testing. Use the guide to conduct AB testing and figure out which of these ideas in the article works for your website visitors and which don’t.  Download Invesp’s “The Essentials of Multivariate & AB Testing” now to start your testing program on the right foot.

On the journey of increasing the conversation rate on your website, you need the precise help of your visitors and clients’ input. You need to evaluate the efficacy of hundreds of possible designs and compare dozens of different results.

You have heard there are tools out there to help you in your quest, but staggering questions haunt your decision.

•  What is the right split testing tool for my website?
•  By choosing a particular tool, will I affect the success of my testing or conversion optimization program?

From simple free tools to complex block testing based software, you can expect to choose from a great variety of AB or multivariate testing tools.

We will help you in your decision by walking you through the pros and cons of the tools available, regarding price, complexity of tests applied as well as the intricacy of your website.

We will also suggest criteria to help you select the software that best fits your needs, considering ease of use, agility of results, cost of running, testing support, operational issues, and testing analytics.


The different types of AB or multivariate testing tools available

1. In-house AB testing tools

Few companies build their own in-house testing tools to conduct AB or multivariate testing. There are two main reasons for using an in-house AB testing tool:

1. legacy built testing tools;
2. complexity of website architecture.

In-house testing tools account for less than 2% of the overall market, and we expect this segment to shrink further as AB testing tools gain more popularity in the marketplace and provide a wider feature set addressing the complexities of varying sites.

2. Free AB Testing Tools

These tools provide the customer the ability of conducting testing free of charge, but they offer limited sets of features, including restricted reporting.
Implementing testing plans with these tools require medium to heavy involvement of technical teams.

The main player in this area is Google content experiments.

3. Point-and-Click AB Testing Tools

Point-and-Click testing tools provide users with an easy to use visual editor to create tests.

Essentials of AB Testing

While they require limited involvement from the customer’s technical team and create designs much faster than their counterparts, these packages do not have the full depth of features necessary to conduct deep testing or analysis.

These packages are great for creating new designs for one page, but they are not fully suited to conduct more complex testing such as visitor flow tests or multi-page tests (for example running a test on e-commerce product and category pages).

The cornerstone visual editor of these packages breaks when modifying websites that rely heavily on javascript or dynamic content.

These tools come with low-cost monthly subscriptions, starting at $50 per month and varying based on the number of page views a customer uses in a month.

Amongst Point-and-Click tools, you can find:

• Optimizely
• Visual Website Optimizer
• Convert

4. Block AB testing based software

These tools typically rely on the marking made by the customers on specific areas of a webpage to be tested. Variations of the marked areas are then created in the testing software by the IT team. Each new test created will require new involvement from the development team.

These packages frequently involve heavy custom configuration to retrieve meaningful visitor tracking data. They also require organizations to have full-time dedicated teams to manage the software and to analyze the results.

While these tools do not come with an easy to use visual editor to make quick changes on a webpage, they do provide detailed reporting and targeting features.

FigPii CRO platform

Monthly plans for these tools start around $1,000 and vary based on the number of page views a customer uses in a month. In addition to the software cost, we estimate a $100,000 annual in-house cost of running tests on enterprise testing packages.

These tools are useful in conducting complex testing such as multi-page testing or visitor flow testing. However, they are an overkill when conducting quick AB testing for a webpage design due to the substantial cost that is associated with the deployment of each test. The main player in this field is Test and Target by Adobe.

Editor note: we described the tools in this section as “block” testing tools in an attempt to describe their functionality. If you recommend a better description for these tools, please let’s know (@invesp)


Selecting the right testing tool for your website

With many tools available, we created the following criteria to help you select the best software to meet your requirements:

1. Ease of use: How fast can you deploy a test?

We evaluate the ability of a testing tool to deploy tests quickly as a way of measuring its ease of use.

When you are conducting a conversion optimization program, your goal is to implement tests efficiently and not to have technology as a barrier to your efforts of increasing conversion rates.

Testing tools vary tremendously when it comes to ease of use. Some of the more sophisticated tools require complex test setup, which can take days. Point-and-Click testing tools definitely outrank any of their counterparts in this area.

2.  How fast does the testing software take to determine a winner for a multivariate test?

After launching a test, the testing software will display the original and new variations to different visitors to determine the winning design. In order to select the winning design, the software will use either full factorial or fractional factorial testing:

Full Factorial Testing

3M Results

The testing software will test all of the different combinations of elements and their alternatives. So, if a test has four elements and each element has three different combinations in it, the testing software will test all possible 3^4= 81 designs.

Fractional Factorial testing (Taguchi method)

The testing software will select a subset of all possible combinations of the different elements and their alternatives. So, in our example above, the testing software will test a smaller set (less than 10) to determine the winner. Fractional factorial testing allows tests to run faster since they do not test all possible combinations. Critics of this method point out that it is less accurate compared to full factorial testing.

The debate between full factorial vs. fractional factorial has been going on for years.

Most testing software uses full factorial in determining the winner of an experiment or a mix between the two different methods. While this might be a sticking point for some testing experts, for 90% of online businesses, it is not a critical point in selecting which software to use.

3. What is the cost of running the AB testing software?

The cost of running testing software varies from zero (Google Analytics) to thousands of dollars per month.

In addition to the cost of running the software itself, there might be extra investments implicated. Some software is too complex for the business to use and will require involvement from the testing software company. That increases the cost of running tests.

 ToolInitial setup costMonthly cost
OptimizelyFree up to 50,000 page views

Free up to 50,000 page views – Higher cost for larger plans

VWOFree up to 2,000 page viewsStarts at $49/month.
ConvertFree 15 days trial up to 10,000 monthly viewsStarts from $125/month

4. Testing support

Sooner or later you will need some support from the testing software company. Many smaller companies ignore this one area when they are doing the initial assessment of the testing packages. We highly recommend evaluating the different alternatives available.

stop CRO frustrations

5. Operational issues (scale, performance, and high availability)

How long does it take the testing software to load up a particular design? Does the testing software use CDN (content delivery network) to deliver designs to visitors who live in different parts of the world?

We highly recommend evaluating the response time for each of the different software packages. Avoid any testing package that takes longer than 300 milliseconds to deliver a design.

6. Testing analytics

In addition to reporting the conversion rate for each variation in a test, different packages will also report the following metrics for each of the variations:

  • Average order value
  • Bounce rate
  • Exit rate
  • Visitor type (new vs. repeat)
  • Traffic source
  • Traffic medium

This data is important in understanding the impact of the new designs on all aspects of the visitor experience on your website.

Invesp can help you!