Category Archives:A/B Testing

  • Validity Threats to Your AB Test and How to Minimize Them

    an image of a gun being pointed to a mask depiciting validity threats to an A/B test

    Disclaimer: This section is a TL;DR of the main article and it’s for you if you’re not interested in reading the whole article. On the other hand, if you want to read the full blog, just scroll down and you’ll see the introduction.

    There are hundreds of case studies and examples of A/B testing. While A/B testing is important, it’s just a small fraction of the overall CRO process. AB testing isn’t foolproof and like anything in statistics, results can be inaccurate. But the more you know about what makes a test valid, and basic statistical concepts, the more likely

    Read More
  • 7 Reasons Your A/B Tests Fail And Their Solutions

    Test this, test that…and always be testing.

    As I listened to one speaker after the next, I wondered why a client would hire them in the first place. If you are a CRO agency, and the best advice you can provide to attendees in a conference is to test this and test that, then I am not sure why these companies would hire you in the first place.

    All you need to do is start AB testing, find that one fantastic test that will take you from absolute anonymity to an overnight web success…that is what brings most people to

    Read More
  • Server-Side A/B Testing Vs. Client-Side A/B Testing

    an image showing a fine car and a terribly kept car. This is trying to show the difference between server side vs client side a/b testing

    Wild guesses and opinions often only result in delusional outcomes. Though opinions are by far the most widely accepted truth, they rarely are factual. It is a better practice to lean on deductive reasoning and logic instead of guesswork. But how do you arrive at what is labeled as a fact or a valid statement? Experimentation is the key.

    Businesses realize the importance of scientific experimentation and testing. Many product-based enterprises have embedded experimentation in their organizational culture. The reason – innovation cannot result without experimentation. Moreover, though there are millions of ideas that spur every minute, every hour, it

    Read More
  • Bayesian A/B Testing vs Frequentist A/B Testing – What’s the Difference?

    an image of two super heroes super man and bat man portraying the difference between Bayesian and frequentist A/B testing

    Bayesian versus Frequentist Statisticians: the war is real

    Imagine that you wake up in the one morning and you don’t remember anything from your previous life. You’ve erased all memories from your previous days.

    Pretty intense and terrifying, huh?

    Well, that’s how Bayesian statisticians describe frequentist colleagues because frequentist statisticians do not use any prior knowledge. Everything you learn about the world is through the lens of events that are happening at the moment and currently.

    Michael Hochster explains;

    “The essential difference between Bayesian and Frequentist statisticians is in how probability is used.”

    Read More
  • How Long Should You Run an A/B Test for and How to Calculate Duration of A/B Test

    an image of a hour glass, with the sands spread evenly in between.

    As interest grows in conversion optimization and A/B testing, marketers are always searching for a new design that will generate significant uplifts in conversion rates. Because the majority of AB tests fail to produce any meaningful results, many marketers are too eager to declare a winner for a split test.

    So, even in the few instances where a testing software declares a winner, there is a good chance that you merely identified a false positive.

    How do you avoid that?

    You start by having a good handle on A/B testing statistics to ensure that your data is collected and analyzed

    Read More
  • What Is An A/A Test And Why You Should Run A/A Tests In Your Conversion Optimization Program

    an image of identical eggs in color in the nest depicting an AA test which means control and variant, no difference.

    Did you know anywhere from 80% to 90% percent of A/B tests do not produce any significant results?

    In fact:

    Only 1 out of 8 A/B tests drive significant change.

    Done incorrectly, some marketers have begun to question the value of A/B testing…

    …their A/B test reports an uplift of 20% and yet, the increase reported by the AB testing software never seems to translate into improvements or profits.

    The reason?

    “Most winning A/B test results are illusory.” (Source: Qubit)

    Furthermore, the majority of arguments that call for running A/A testing consider it a sanity check before you run

    Read More
  • 14 AB Testing Sample Size Issues and Mistakes That Can Ruin Your Test

    a cartoon image of a laptop screen with A/B test written ontop by the sides, a winning and losing A/B test.

    You must cover all the bases to get reliable results from your A/B tests.

    A/B testing mimics scientific experiments, and similarly will not provide you a 100% certainty in 100% of tests that you run. Only few marketers are aware of the limitations of the method and know how to run it to get valid results and minimize the risk of false positives and false negatives.

    That’s why Martin Goodson, now the Chief Scientist at Evolution AI, wrote a paper called “Most Winning A/B Test Results Are Illusionary.” In his paper, he explained that badly performed A/B tests are more

    Read More
  • A/B Testing Statistics Made Simple

    an image of a desktop screen with A and B split on the screen with statistics symbols as the background depicting A/B testing statistics made simple

    “Why do I need to learn about statistics in order to run an A/B testing?” You may be inclined to wonder, especially considering that the testing engine supplies you with data to make a judgement on the statistical significance of the test, correct?

    As a matter of fact, you have plenty of reasons to learn statistics.

    If you’re conducting A/B tests, you need to understand some basics about statistics to validate your tests and their results.

    Nobody wants to spend time, money and effort on something that will turn out useless at the end. To use A/B testing efficiently and effectively,

    Read More