Guide To AB Testing: Everything You Need To Know

If your website visitors are not converting, something is stopping them.

You can ask your design team to create new designs, but How do you know the new designs will convert more visitors than the original design?

This guide to A/B testing covers everything you need to know: why you should consider AB testing, the categories of AB tests, how to launch an AB test, and more.

What Is AB Testing?

A/B testing (also known as split testing) involves comparing two versions of a webpage, app, or element to determine which one performs better. 

It involves testing different elements with similar audiences—such as a headline, landing page images, or CTA buttons. The goal is identifying which variant leads to higher conversions, engagement, or other desired outcomes.

Here’s how A/B testing works in action:

ALSO READ: The Difference Between A/B Testing and Multivariate Testing

The original design of a page is usually referred to as the control. The new designs of the page are usually referred to as the “variations,” “challengers,” or “recipes.”

The process of testing which page design generates more conversions is typically referred to as a “test” or an “experiment.”

A “conversion” will vary based on your website and the page you are testing. For an e-commerce website, a conversion could be a visitor placing an order. 

However, a conversion for a SaaS website could be a visitor subscribing to the service. For a lead generation website, a conversion could be a visitor filling out a contact form.

How Is AB Testing Performed?

Here’s a quick step-by-step breakdown of how A/B testing works: 

• Start with a Hypothesis: Before you make any changes, formulate a clear hypothesis about what you believe will happen when you make a specific alteration to your design or idea. This hypothesis should be testable and measurable. For example, “Changing the call-to-action button from blue to green will increase click-through rates by 10%.”
  
• Create Two Versions: You start with your original design or idea (Version A – the control) and then make a specific change to create a slightly different version (Version B – the variation). This change could be anything: a different headline, a new button color, an altered layout, or a new feature altogether.
  
  
• Split Your Audience: You randomly divide your audience into two groups. One group sees Version A, and the other group sees Version B. It’s crucial that this split is random to ensure fair results.
  
  
• Collect Data: As your audience interacts with each version, you track their behavior. This could mean measuring clicks, purchases, sign-ups, or other important metrics.
  
  
• Analyze and Decide: After a set period (long enough to gather meaningful data), you compare the results from both versions. Using statistical analysis, you determine if the difference in performance is significant enough to declare a winner. If so, you implement the winning version.

Here’s an A/B testing example to help you understand better: 

Let’s say your ecommerce site receives 100,000 visitors a month. To determine if there is a way to increase conversions, your design team creates a new design for the homepage.

They then use AB testing software to randomly split the homepage visitors between the control and the new challenger. 

• Control: 50,000 visitors see the original homepage.
• Challenger: 50,000 visitors see the new design.

Since we are testing which design generates more orders (conversions), we use the A/B testing software to track the number of conversions each design generates. 

This allows you to:

• Compare: See which design leads to more orders.
• Analyze: Understand the impact of the design change.

Decide: Choose the winning design based on data.

What Is The Main Purpose Of AB Testing?

A significant challenge ecommerce businesses face is the issue of a high cart abandonment rate. 

This is bad for a business because it usually signals the customer’s unhappiness. 

This is where the A/B test shines because it allows you to maximize your existing traffic without spending extra cash on acquiring new traffic.

Here are more reasons why you should conduct A/B tests:

1. Reduces bounce rates:

One advantage of running A/B tests is that they prevent you from launching untested new designs that could fail and dampen your revenue.

Nothing is more painful than working on your site design, making it public, and realizing site visitors are not engaging with your content.

Many ecommerce sites face this issue. To prevent this, create an A/B test before rolling out new designs. A/B testing will help you decide whether the new design is worth it. If it doesn’t show the right results, try more designs until you get favorable results. 

Split your traffic using an A/B testing tool like Figpii across the control and design and let your users decide.

2. Reduced cart abandonment rates:

One of the significant plagues eCommerce stores face is cart abandonment.

This means site visitors and customers add an item(s) to the cart and don’t complete the checkout process.

How A/B tests help you here is simple. 

The product checkout page has essential elements, such as the check-out page text, the shipping fee, etc. 

By creating a variation, combining these elements, and changing their location, you can see which of the pages (control or variation) helps decrease the cart abandonment rate.

With A/B testing your redesign ideas, you can guarantee that they will improve cart abandonment rates.

3. Increased conversion rates:

You can increase your conversion percentage if you see a decent conversion rate with A/B testing.

You can A/B test page layout, copy, design, location of the CTA button, etc. 

Without A/B tests, if you should make a design or copy change, there’s no guarantee of improvements.

4. Higher conversion values: 

The learnings you get from an A/B test on one of your product pages can be implemented or modified on the product pages of more expensive products.

This goes a long way in improving your customer AOV and revenue bottom line.

How To Launch an A/B Test: A Step-by-Step Guide 

Here’s a straightforward process to help you design and launch an A/B test from scratch. 

Step 1. Identify your goal 

Before you start, you must know what you’re trying to achieve. 

What’s your main goal? It could be: 

• Increasing website sales by 5%
• Getting 10% more people to click on a “Sign Up” button
• Reducing the number of people who abandon their shopping carts by 15%

Collecting quantitative and qualitative data is crucial in determining your goals and what to A/B test.

By looking closely at how people use your website and what they say about it in research, you’ll discover why they get frustrated.

One qualitative research method is analyzing heat maps and session recordings. 

This way, you can easily see visitor behavior data, including where they click, scroll depth, etc. All of it gives you ideas for future tests.

You can use behavior analytics tools like FigPii for that. 

Combine it with Google Analytics 4 to track the pages with the highest bounce rates and few user activities. These are pages you can improve.

Step 2. Form a hypothesis 

Once you have a goal, create a hypothesis. This is your best guess about what change might help you reach your goal.

An adequate A/B testing hypothesis consists of three key components:

• Identify an apparent problem or challenge.
• Offer a precise solution to address the problem.
• Describe the expected impact of the solution.

Here is an example of a solid AB testing hypothesis:

Certain customers abandon their shopping carts due to a lengthy checkout process (challenge). Simplifying the checkout form by reducing the required fields (specific solution) is expected to increase the conversion rate by 20% (assumed impact).

Step 3. Choose your A/B testing tool

There are many great tools out there for A/B testing. We’d suggest using something like FigPii—an all-in-one behavior analytics tool that helps you run A/B tests and gauge heat maps and session recordings for visitors’ behavioral insights.

This intuitive platform makes it easy to create and launch A/B tests, even if you don’t have any coding knowledge. It also has a visual editor that allows you to modify elements on your webpage and track results in real-time.

Other options include: 

• Optimizely: It is a robust platform with many features that can be expensive.
• VWO: A good choice for both beginners and advanced users, but it may require a learning curve. 

Step 4. Create your variations (A and B) 

Now, it’s time to make the changes you want to test. 

Example: You might create two versions of a webpage:

• Version A (the original) has a blue “Buy Now” button
• Version B (the variation) has a green “Buy Now” button

Using an A/B testing tool like FigPii, you can easily create the variation of the page you want to test. You can also make changes to the element you want to focus on. 

Step 5. Define your target audience

Who are you testing on? You can choose to show your test to everyone, or you can target specific groups of people.

For example, your target audience could consist of: 

• New visitors to your website
• People who have already looked at a particular product
• Visitors from a specific location

The more specific your audience, the more relevant your results will be.

Step 6. Split your audience (determine sample size and duration)

Randomly assign visitors to see either version A or B. 

Make sure your sample size is big enough to produce statistically significant results. There are calculators online to help you do this.

Check out this in-depth guide for more details on how to calculate A/B Testing sample size. 

As for duration, most tests run for at least a week, but some may last several weeks.

Step 7. Launch and monitor 

Now, you’re ready to start your test! Your testing tool will automatically split your traffic between the variations. 

The tool will also record and compute the interaction with the control or variation, determining how either is performed. 

Keep an eye on your results as they come in.

Pro Tip: Don’t stop your test until it hits the required sample size and achieves statistically significant results.

For the uninitiated, statistical significance means you’re sure a result isn’t just a random chance. It means the difference you see in an A/B test is likely absolute and not just a fluke due to the group of people you tested.

As a rule of thumb, an A/B test should yield statistically significant 90% (and above) results for the change to impact a website’s performance.

Like sample data, there are tools to calculate whether your results are statistically significant.

Step 8. Analyze and draw conclusions

Once your test is over, analyze the data. 

Did one variation perform better? Was the difference statistically significant (meaning it’s not just due to chance)? If so, you can confidently implement the winning change. If not, try a different hypothesis.

This post-test analysis gives you perfect clarity regarding A/B testing results.

How Does the A/B Testing Software Determine the Winning Design?

At its core, AB testing software tracks the number of visitors coming to each design in an experiment and the number of conversions each design generates. 

However, sophisticated A/B testing software would track so much more.

For example, FigPii helps you track the following for each variation:

• Conversions
• Pageviews
• Visitors
• Revenue per visit
• Bounce rate
• Exit
• Revenue
• Source of traffic
• Medium of traffic

The split testing platform uses different statistical modes to determine a winner in a test. 

The two popular methods for determining a winner are Frequentist and Bayesian models.

Frequentist Model:

This model uses two main factors to determine the winning design:

• The conversion rate for each design: This number is determined by dividing the number of conversions for a design by the unique visitors for that design.
  
  
• The confidence level for each design: A statistical term indicating the certainty that your test will produce the same result if the same experiment is conducted across many separate data sets in different experiments.

CRO experts favor frequentist methods for their simplicity and well-established statistical framework.

Bayesian Model:

This approach uses two main factors to determine the winning design:

• The conversion rate for each design: Similar to the Frequentist method 
• Historical performance: the success rate of previously ran A/B experiments on the web page.

Bayesian methods can be more flexible and allow for incorporating additional information, but they may require more computational resources and expertise.

Leonid Pekelis, Optimizely’s first in-house statistician, explains the essence of the Bayesian model,

“Bayesian statistics take a more bottom-up approach to data analysis. This means that past knowledge of similar experiments is encoded into a statistical device known as a prior, and this prior is combined with current experiment data to make a conclusion on the test at hand.”

Both Frequentist and Bayesian models have their strengths and weaknesses, and the choice often depends on the specific test scenario and preferences. 

However, both ultimately provide insights into the probability of one variation outperforming another.

Steps Involved in Determining a Winner:

Here’s a quick overview of the steps involved in determining  a winner for your A/B test: 

• Data Collection: The software randomly assigns visitors to variations and tracks relevant metrics (clicks, conversions, etc.).
  
  
• Statistical Analysis:
  
  
• Frequentist: Calculates p-values and confidence intervals to assess statistical significance.
• Bayesian: Updates prior beliefs based on data to generate posterior distributions.
  
  
• Hypothesis Testing:
  
  
• Frequentist: Tests the null hypothesis (no difference between variations) against the alternative hypothesis (a difference exists).
• Bayesian: Evaluates the probability of different hypotheses given the data and prior beliefs.
  
  
• Confidence Intervals: Both methods use confidence intervals to estimate the range within which the true difference in performance likely lies.
  
  
• Declaring a Winner: If the results show a statistically significant difference and align with the test goals, the variation with superior performance wins.

We’ve already talked about it in the above section. But here’s a quick overview of the important considerations when declaring an A/B test winner: 

• Make sure you have a sufficient sample size for reliable results.
• Tests should run long enough to gather enough data and account for fluctuations.
• Don’t overlook external events or seasonality when interpreting data.

Types Of A/B Testing?

AB tests come in different types and can be done in different environments.

1. A/A Test:

An A/A test is a control test that compares two identical webpage versions. It ensures that the testing process is working correctly. 

For example, in an A/A test, users are simultaneously shown both version A and version A’ of a webpage. If the results show significant differences between these identical versions, it suggests a problem in the testing setup.

When to use A/A testing: To validate your testing tool and ensure accurate results.

2. A/B Test:

A/B testing involves comparing two versions of the same web page (A and B) to determine which one performs better. 

For instance, in an ecommerce A/B test, version A might have a green “Buy Now” button, while version B has a red one. By analyzing user interactions, businesses can identify the color that leads to higher click-through rates and conversions.

When to use it:  For testing single changes, like headlines, user flow, designs, web elements, images, or email subject lines.

3. A/B/n Test:

A/B/n testing expands on A/B testing by comparing multiple webpage versions (A, B, C, etc.). For example, an online news platform might test different headlines (A, B, C) simultaneously to see which one attracts more clicks, providing insights into user preferences among multiple options.

When to use it: When you have several ideas for a single element and want to compare them all simultaneously.

4. Multivariate Test:

Multivariate testing involves testing multiple variations of different elements within a webpage. 

For instance, a travel website could test various combinations of images, headlines, and call-to-action buttons on its homepage to find the optimal mix that increases user engagement and bookings.

When to use multivariate tests: Run a multivariate test when you want to see how different elements interact with each other for optimal performance.

5. Targeting Test:

Targeting tests involve showing different versions of a webpage to specific audience segments. 

For example, an online clothing store might show different homepage versions to new visitors, returning customers, and newsletter subscribers, tailoring the user experience based on their preferences and behaviors.

When to use it: To personalize the website experience and tailor variations to different visitor groups.

6. Bandit Test:

Bandit tests, also known as Multi-Armed Bandit tests, dynamically allocate traffic to the best-performing versions during the testing period. 

For instance, an online gaming app might use a bandit test to optimize the display of in-game ads, ensuring that the most effective ad is shown more frequently to maximize revenue.

When to use it: To minimize the exposure of underperforming variations and maximize the winning one.

7. Split Page Path Test:

Split page path tests test different user journeys or paths within a website. 

For example, an e-learning platform could test two pathways for users to access course materials: a step-by-step guide and a video tutorial. 

The platform can optimize the learning experience based on the preferred path by comparing user engagement and completion rates.

When to use it: To optimize user flows and identify which path leads to higher conversions or desired outcomes.

Common A/B Testing Mistakes To Avoid

A/B testing takes time to plan, implement, and get learnings from the result. 

This means you can’t afford to make mistakes; otherwise, they might cost you revenue and precious time.

Here are some common A/B mistakes you want to avoid:

1. Testing without a clear hypothesis. 

Seasoned experimenters know only to test something if they form a hypothesis. 

Going straight to create an A/B test, skipping the step of insight gathering (qualitative and quantitative), and forming a hypothesis could negatively impact your site’s conversion rate.

What to do instead: Formulate a specific, testable hypothesis based on user data and insights. For example, “Changing the call-to-action button color from blue to orange will increase click-through rate by 10%.”

2. Testing too many elements at once.

Changing multiple elements simultaneously makes it impossible to know which change caused the observed effect.

What to do instead: Test one element at a time to isolate the impact of each change.

3. Changing parameters mid-test.

Changing your testing parameters midway is one absolute way to mess up your A/B test.

This needs to be corrected for your results.

Parameters you can mess up:

• Changing the allocated traffic mid-way.
• Changing your split test goals.

What to do instead: Changing your testing parameters spoils your results. If you must change something, start the test again.

4. Not allowing the test to run entirely and ignoring statistical significance 

Declaring a winner based on small, insignificant differences can lead to implementing changes without real impact. It might even negatively affect performance.

What to do instead: Allow the experiment to run to achieve statistical significance. This is the only way the results can’t be declared invalid.

5. Using tools that impact site performance. 

As A/B testing becomes more popular, many low-cost tools are flooding the market. Such tools put your test at risk and might negatively impact your site performance.

Both Google and your site visitors want your website to load fast, but some A/B test software adds an additional step in loading and displaying a page. 

This leads to the flicker effect, also known as the Flash of Original Content (FOOC), in which the site visitor sees the control page for some seconds before the variation appears.

This leads to a bad user experience, which slows the page load time, which ultimately impacts conversions because site visitors are known not to be patient.

What to do instead: Only use reliable and reputed A/B testing tools. Check their online reviews before making a decision. 

6. Not considering external factors. 

External events like holidays, sales, or website outages can skew A/B test results if not accounted for.

What to do instead: Be aware of any external factors influencing user behavior during your test, and consider pausing or adjusting your test if necessary.

A/B Testing FAQs

What Is A/B Testing?

AB testing, also known as split testing, compares two versions (A and B) of a webpage or marketing element to analyze user behavior and determine which version performs better in achieving specific goals, such as improving conversion or click-through rates.

How do I choose the correct elements for A/B testing?

Focus on landing page layout, ad copy, and subject lines that directly impact user engagement. Identify specific challenges in your marketing campaign to formulate compelling hypotheses for testing.

Why is statistical significance important in AB testing?

Statistical significance ensures that the differences observed in test results are not due to chance. It provides reliable data by confirming whether the changes observed in user behavior or website visitors are statistically significant results, not random fluctuations.

How do I determine the sample size for A/B testing?

Calculating an appropriate sample size is crucial. Use statistical methods to ensure the data collected is robust and representative of your target audience. Tools like Google Analytics can assist in understanding your visitor behavior data and guide your decisions.

Can A/B testing be applied to multiple pages of a website?

Yes, A/B testing can be conducted on multiple pages. Analyze user interactions across various pages to gain quantitative user insights. Ensure consistency in maintaining the same user experience to achieve accurate results.

How long should I run an A/B test to collect sufficient data?

Run the test until you achieve statistical significance. Factors like landing pages, test results, and user behavior impact the duration. Larger changes might show results quickly, while subtle ones require longer durations to gather enough data for analysis.