{"id":11543,"date":"2018-06-25T14:34:03","date_gmt":"2018-06-25T19:34:03","guid":{"rendered":"https:\/\/www.invespcro.com\/blog\/?p=11543"},"modified":"2026-08-10T18:01:49","modified_gmt":"2026-08-10T18:01:49","slug":"validity-threats-to-your-ab-test-and-how-to-minimize-them","status":"publish","type":"post","link":"https:\/\/www.invespcro.com\/blog\/validity-threats-to-your-ab-test-and-how-to-minimize-them\/","title":{"rendered":"Validity Threats to Your AB Test and How to Minimize Them"},"content":{"rendered":"<span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\"> 14<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span>\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"11543\" class=\"elementor elementor-11543\" data-elementor-post-type=\"post\">\n\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-3672c48f elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"3672c48f\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-581e6616\" data-id=\"581e6616\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-73324b7a elementor-widget elementor-widget-text-editor\" data-id=\"73324b7a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p><i><span style=\"font-weight: 400;\">Disclaimer: This section is a TL;DR of the main article and it\u2019s for you if you\u2019re not interested in reading the whole article. On the other hand, if you want to read the full blog, just scroll down and you\u2019ll see the introduction.<\/span><\/i><\/p><ul><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">There are hundreds of case studies and examples of A\/B testing. While A\/B testing is important, it\u2019s just a small fraction of the overall CRO process.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">AB testing isn\u2019t foolproof and like anything in statistics, results can be inaccurate. But the more you know about what makes a test valid, and basic statistical concepts, the more likely it is that you will not face errors.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">\u00a0Validity threats are risks associated with certain uncontrollable or \u2018little-known\u2019 factors that can lead to inaccuracy in results and render inaccurate A\/B test outputs and they\u2019re categorized as type 1 and type 2 errors.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A null hypothesis is <\/span><b>a<\/b><span style=\"font-weight: 400;\">n assumption stating that there is absolutely no relation between two datasets. Hypothesis testing is done to either prove or disprove if an assumption is correct or wrong.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">In statistics, type 1 error is said to occur when a true null hypothesis is rejected, which is also called a \u2018false positive\u2019 occurrence. Results might indicate that Variation B is better than Variation A as B is giving you more conversions, but there might be a type 1 error causing this conclusion.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">In statistics, type 2 errors or a false negative occur when a false null hypothesis is retained or accepted. Or, in other words, when a test is inconclusive when in reality it is conclusive.\u00a0<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\"><strong>Flicker effect:<\/strong> this occurs when original content flashes for a brief time before the variation gets loaded onto the visitors\u2019 screens. This leads to visitors getting confused about content, and can result in conversions dropping.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\"><strong>History effect:<\/strong> this happens when an extraneous variable is introduced while a test is running, and leads to a skewing of results. It happens because an AB test is unlike a lab test and does not run in isolation. Therefore, AB tests are prone to be affected by external variables and real-world factors.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\"><strong>Instrumentation effect:<\/strong> these are errors related to your testing tool and code implementations. It happens when the tool you\u2019re using is faulty or the implemented the wrong code.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\"><strong>Selection effect:<\/strong> this bias or error occurs because the sample is not representative of your entire audience. One of the reasons why selection error happens is because of sample bias. Marketers conducting experiments get attached to the hypothesis that they have constructed. Everyone wants their hypothesis to win. So, it is easy to select a certain sample for testing and eliminate factors or variants that might result in their hypothesis being incorrect.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\"><strong>Novelty effect:<\/strong> error or changes in test results that are an outcome of introducing something unusual or new that the visitor is not used to. The novelty effect happens because something new is fed to visitors.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\"><strong>Statistical regression:<\/strong> this happens when you end a test too early. This leads to data being evened out over the time period. Most people end the test when a 90% significance level is reached, without reaching the required <a href=\"https:\/\/www.invespcro.com\/blog\/calculating-sample-size-for-an-ab-test\/\">sample size<\/a>. You cannot be sure of the AB test results only by reaching 90% significance. You must be able to reach the required sample size as well.<\/span><\/li><li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\"><strong>Simpson&#8217;s paradox:<\/strong> this happens because of changing the traffic splits for variants while the test is going on. It occurs when a trend that was being observed in different sets of data disappears by combining those groups.<\/span><\/li><\/ul><p>\u00a0<\/p><p><strong>Here\u2019s A Longer And More Detailed Version Of The Article.<\/strong><\/p><hr \/><p><span style=\"font-weight: 400;\">There are hundreds of case studies and examples on <a href=\"https:\/\/www.invespcro.com\/ab-testing\/\">AB testing<\/a>, explaining what makes it highly useful for <a href=\"https:\/\/www.invespcro.com\/cro\/\">conversion optimization<\/a>. While AB testing is important, it is a mere component of the <a href=\"https:\/\/www.invespcro.com\/cro\/process\/\">overall CRO process<\/a>. What\u2019s critical to keep in mind is not dive into <a href=\"https:\/\/www.invespcro.com\/ab-testing\/\">AB testing right away<\/a> until you know everything about interpreting, analyzing, and understanding test results. <\/span><\/p><p><span style=\"font-weight: 400;\">AB testing isn\u2019t foolproof and like anything in statistics, results can simply be wrong. But the more you know about what makes a test valid, and basic statistical concepts, the more likely it is that you will not face errors. This is where validity threats because of an important topic of discussion. If left unidentified or unaccounted for, they can lead you to take the wrong decision. <\/span><\/p><p><span style=\"font-weight: 400;\">What are validity threats, you ask? In simple words, validity threats are risks associated with certain uncontrollable or \u2018little-known\u2019 factors that can lead to inaccuracy in results and render inaccurate AB test outputs. Broadly speaking, validity threats can be categorized as type 1 and type 2 errors. But before we define these errors, let&#8217;s understand what a null hypothesis is. <\/span><\/p><\/div><\/div><\/div><\/div><\/div><\/section><\/div> <a href=\"https:\/\/www.invespcro.com\/blog\/validity-threats-to-your-ab-test-and-how-to-minimize-them\/#more-11543\" class=\"more-link elementor-more-link\"><span aria-label=\"Continue reading Validity Threats to Your AB Test and How to Minimize Them\">(more&hellip;)<\/span><\/a>","protected":false},"excerpt":{"rendered":"<p><span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\"> 14<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span>Disclaimer: This section is a TL;DR of the main article and it\u2019s for you if you\u2019re not interested in reading the whole article. On the other hand, if you want to read the full blog, just scroll down and you\u2019ll see the introduction. There are hundreds of case studies and examples of A\/B testing. While [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":11545,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[116,36],"tags":[103,568,87,569,570,245,571,572,109,573,574,575,565,566,576],"class_list":["post-11543","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ab-testing","category-cro","tag-ab-testing","tag-flicker-effect","tag-general","tag-history-effect","tag-instrumentation-effect","tag-intermediate","tag-novelty-effect","tag-null-hypothesis","tag-resource","tag-selection-effect","tag-simpsons-paradox","tag-statistical-regression","tag-type-i-error","tag-type-ii-error","tag-validity-threats"],"_links":{"self":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts\/11543","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/comments?post=11543"}],"version-history":[{"count":3,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts\/11543\/revisions"}],"predecessor-version":[{"id":101092,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts\/11543\/revisions\/101092"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/media\/11545"}],"wp:attachment":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/media?parent=11543"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/categories?post=11543"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/tags?post=11543"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}