{"id":15177,"date":"2021-09-17T07:41:41","date_gmt":"2021-09-17T12:41:41","guid":{"rendered":"https:\/\/www.invespcro.com\/blog\/?p=15177"},"modified":"2021-09-17T07:41:41","modified_gmt":"2021-09-17T12:41:41","slug":"a-b-testing-on-holidays-is-it-worth-it","status":"publish","type":"post","link":"https:\/\/www.invespcro.com\/blog\/a-b-testing-on-holidays-is-it-worth-it\/","title":{"rendered":"A\/B testing on Holidays:\u00a0Is it Worth it?\u00a0"},"content":{"rendered":"<span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\"> 7<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span><p><i><span style=\"font-weight: 400;\">Disclaimer: This section is a TL;DR of the main article, and it\u2019s for you if you\u2019re not interested in reading the whole article. On the other hand, if you want to read the full blog, just scroll down, and you\u2019ll see the introduction.<\/span><\/i><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Some online retailers take weeks, while some take months to plan out their holiday strategy.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">It seems impossible to develop a conversion strategy in a few days, but we did so in 2019 when a client approached us.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Seasonal events like Black Friday and Cyber Monday need to be approached thoughtfully, and you need all the customer data you can get to execute successful campaigns.<\/span><\/li>\n<\/ul>\n<h2><b>Why Do Companies Hate A\/B Testing During Holidays?<\/b><\/h2>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Most online retailers aren\u2019t willing to run A\/B tests during the holidays, and this is something we see with some of our clients.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">The fear of introducing new bugs into their site is one of the reasons why clients don\u2019t like running A\/B tests during the holiday season.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Whether it\u2019s a holiday season or not, there\u2019s a possibility of new bugs creeping into your variations when A\/B testing \u2013 especially when your A\/B testing has to do with the website functionality.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">The second reason many businesses are skeptical of conducting tests during the holiday season is the fear of revenue reduction due to introducing new designs.<\/span><\/li>\n<\/ul>\n<h2><b>Is Holiday A\/B Testing Worth It?<\/b><\/h2>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Holiday A\/B Tests come with their own set of risks and rewards<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Most online retailers experience an influx of visitors during holiday seasons, but it can be tricky to run tests if you\u2019re not experienced.<\/span><\/li>\n<\/ul>\n<h2><b>Why You Should A\/B Test During Holidays<\/b><\/h2>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Because you get an influx of new visitors.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">You get a lot of comparison shoppers who are driven by product price and availability.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">If you\u2019re not running tests, you\u2019ll never understand what kind of offers drive your customers during the holiday season.<\/span><\/li>\n<\/ul>\n<h2><b>Why You Shouldn\u2019t A\/B Test During The Holidays<\/b><\/h2>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A\/B testing is not just about maximizing revenue \u2013 at its core, it\u2019s about understanding how your visitors think and behave by analyzing the test results.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">The problem with running tests during the holidays is that the results might easily be skewed. You can\u2019t rely on holiday test results to permanently change your website.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Remember, during holidays; you\u2019re dealing with consumers driven by the cost and inventory \u2013 this means that their behavior is different from your usual customers during the year.\u00a0<\/span><\/li>\n<\/ul>\n<p><b>Here\u2019s a longer and more detailed version of the article<\/b><\/p>\n<hr \/>\n<p><span data-preserver-spaces=\"true\">How far out do you need to plan your holiday A\/B testing strategy?<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Some online retailers take two weeks to plan. Some take a couple of months. And, others need a few weeks to prepare.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">There seems to be no right or wrong answer to that question.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">At some point, I used to think that it was impossible to come up with and execute a successful holiday testing program in few days.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">That was before I witnessed our CRO team create and execute one three days before Black Friday.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Yes, it&#8217;s a true story!\u00a0<\/span><!--more--><\/p>\n<p><span data-preserver-spaces=\"true\">Three days before the Black Friday of 2019, an eCommerce company signed up with us and tasked us to handle their holiday testing program.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Seasonally events \u2013 like Black Fridays and Cyber Mondays \u2013 need to be approached thoughtfully, and you need all the customer data you can get to execute successful campaigns.\u00a0<\/span><\/p>\n<blockquote><p><span data-preserver-spaces=\"true\">&#8220;<\/span><em><span data-preserver-spaces=\"true\">If they hadn&#8217;t provided us with insights on their customers and inundated us with a ton of information &#8211; we would not have pulled off a successful campaign. We also needed to compare the traffic and their activity from the prior year to gauge customer behaviors during this period<\/span><\/em><span data-preserver-spaces=\"true\">.&#8221;<\/span><\/p><\/blockquote>\n<p><span data-preserver-spaces=\"true\">\u2013 says\u00a0<\/span><a class=\"editor-rtfLink\" href=\"https:\/\/www.linkedin.com\/feed\/update\/urn:li:activity:6600385441239744513\/\" target=\"_blank\" rel=\"noopener noreferrer\"><span data-preserver-spaces=\"true\">Ayat<\/span><\/a><span data-preserver-spaces=\"true\">, who was also part of our CRO team that created and executed a testing program in 3 days.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">In other words, If you&#8217;re entirely unfamiliar with the website, the market, and the customers, then your <a href=\"https:\/\/www.invespcro.com\/ab-testing\/\">A\/B testing program<\/a> is more likely to be a disaster.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">When it comes to holiday campaigns, it&#8217;s not about the promotions \u2013 but rather the marketing strategies will engage and persuade the visitor during and keep the visitor loyal post.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">In this article, we will look at why companies hate holiday testing and the pros and cons of running tests during holidays. We are also going to touch on the dos and don&#8217;ts of holiday testing.<\/span><\/p>\n<h2><strong><span data-preserver-spaces=\"true\">Why do companies hate A\/B testing during holidays?<\/span><\/strong><\/h2>\n<p><span data-preserver-spaces=\"true\">Most online retailers are not willing to run A\/B tests during holidays \u2013 and this is something we have seen with some of our clients.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">In our experience, there are two reasons why companies want to go through a code freeze during holidays:\u00a0<\/span><\/p>\n<h3><span data-preserver-spaces=\"true\">Fear of introducing new bugs<\/span><\/h3>\n<p><span data-preserver-spaces=\"true\">You launch a test with two or three variations against the control. And you think that your testing tool is collecting correct data. You then discover that one of the variations is bugged \u2013 and it&#8217;s not working for Safari users.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Whether it&#8217;s a holiday season or not, there&#8217;s a possibility of new bugs creeping in your variations when A\/B testing \u2013 especially when your A\/B testing has to do with the website functionality.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">In his article \u2013 about\u00a0<\/span><a class=\"editor-rtfLink\" href=\"https:\/\/www.invespcro.com\/ab-testing\/\" target=\"_blank\" rel=\"noopener noreferrer\"><span data-preserver-spaces=\"true\">A\/B testing sampling issues<\/span><\/a><span data-preserver-spaces=\"true\"> \u2013 Khalid writes:\u00a0\u00a0<\/span><\/p>\n<blockquote><p><span data-preserver-spaces=\"true\">&#8220;<\/span><em><span data-preserver-spaces=\"true\">Bugs are typically introduced while coding the test variations. Since most tests are coded using JavaScript and not served from the server-side, there is a good chance that one of your variations might contain bugs<\/span><\/em><span data-preserver-spaces=\"true\">.&#8221;<\/span><\/p><\/blockquote>\n<p><span data-preserver-spaces=\"true\">Fortunately, there&#8217;s a way to discover the bugs before the test is launched.\u00a0\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">We usually use extensive Quality Assurance (QA) on a staging environment and addressing any bugs (if any) before moving the test to production.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">In other words, no test has to be pushed to the production servers before the data is validated in a QA environment.\u00a0<\/span><\/p>\n<h3><span data-preserver-spaces=\"true\">Risk of reduction in revenue as a result of new designs<\/span><\/h3>\n<p><span data-preserver-spaces=\"true\">Whenever you launch a test, you are not 100% sure that it will increase conversions. That&#8217;s why you have to test it first.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">In other words, with every A\/B test you launch, there&#8217;s a risk of decreasing conversions.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">So some clients would instead go with the original design and not risk experimenting \u2013 especially during the holiday season where they&#8217;re expecting to cash in lots of revenue.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">So, how can this be mitigated, you ask?\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Well, the solution that we have for a lot of our clients is to do gradual testing.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">For example, instead of directing 50% of the traffic to the new design, we start by 10% and 90% to the original design. And then we gradually increase or decrease the traffic according to how o how the new design is performing.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">This way, our clients feel more comfortable running A\/B tests during holidays.\u00a0<\/span><\/p>\n<h2><strong><span data-preserver-spaces=\"true\">Holiday testing \u2013 Is it worth it?\u00a0<\/span><\/strong><\/h2>\n<p><span data-preserver-spaces=\"true\">Most online retailers experience an influx of visitors during holiday seasons.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Does that mean you should take advantage of this seasonal spike in traffic and launch some tests?\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Well, that&#8217;s a good but tricky question.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">I have to say, testing during holidays comes with its own set of risks and rewards.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">You have to be experienced in the field of CRO if you want your tests to be successful.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">ALSO READ:\u00a0<\/span><a class=\"editor-rtfLink\" href=\"https:\/\/www.invespcro.com\/blog\/what-they-dont-tell-you-about-a-b-testing-velocity\/\" target=\"_blank\" rel=\"noopener noreferrer\"><span data-preserver-spaces=\"true\">What they don&#8217;t tell you about A\/B testing<\/span><\/a><\/p>\n<h3><strong><span data-preserver-spaces=\"true\">Why you should AB test during Holidays<\/span><\/strong><\/h3>\n<p><span data-preserver-spaces=\"true\">It&#8217;s not every day that you get an increase in:\u00a0<\/span><\/p>\n<ul>\n<li><span data-preserver-spaces=\"true\">New visitors to your site,\u00a0<\/span><\/li>\n<li><span data-preserver-spaces=\"true\">Comparison shoppers, who are driven by product price and availability,<\/span><\/li>\n<li><span data-preserver-spaces=\"true\">Visitors from new or less popular sources (and sometimes locations),\u00a0<\/span><\/li>\n<li><span data-preserver-spaces=\"true\">And average order and cart size.\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-preserver-spaces=\"true\">\u2013 so, not running tests during holidays can feel like a missed opportunity.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">You might also be leaving more money on the table by not testing during holidays that you could have got through new variations.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">For instance, how do you know that your holiday offer is performing as well as it could be? If you are not running tests, you will never understand what kind of offer drives your customers during the holiday season.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">In such periods of high urgency, you can quickly learn how your customers think, behave, and what kind of offers drive them to take the desired action on your site.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">There are tons of\u00a0<\/span><a class=\"editor-rtfLink\" href=\"https:\/\/www.figpii.com\/blog\/top-10-ab-testing-tools-used-by-cro-experts-in-2021\/\" target=\"_blank\" rel=\"noopener noreferrer\"><span data-preserver-spaces=\"true\">A\/B testing tools<\/span><\/a><span data-preserver-spaces=\"true\">\u00a0that have editor WYSIWYG that allows you to create variations without touching your site&#8217;s code \u2013 especially when the variations don&#8217;t change the page functionality.\u00a0<\/span><\/p>\n<div class=\"blog_img\">\n<div class=\"blog_img\"><img fetchpriority=\"high\" decoding=\"async\" class=\"alignnone size-full wp-image-15181\" src=\"https:\/\/www.invespcro.com\/blog\/images\/blog-images\/Table-3.jpg\" alt=\"\" width=\"1600\" height=\"1144\" data-wp-pid=\"15181\" \/><\/div>\n<\/div>\n<p><span data-preserver-spaces=\"true\">Using the above tools, you don&#8217;t have to change the code to remove, add, resize, or even rearrange your web page copy.\u00a0\u00a0<\/span><\/p>\n<h3><strong><span data-preserver-spaces=\"true\">Why you shouldn&#8217;t A\/B test during Holidays<\/span><\/strong><\/h3>\n<p><span data-preserver-spaces=\"true\">A\/B testing is not just about trying to maximize revenue \u2013 at its core, it&#8217;s about understanding how your visitors think and behave by analyzing the test results.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">ALSO READ:\u00a0<\/span><a class=\"editor-rtfLink\" href=\"https:\/\/www.invespcro.com\/ab-testing\/results-analysis\/\" target=\"_blank\" rel=\"noopener noreferrer\"><span data-preserver-spaces=\"true\">How to analyze A\/B testing results and statistical significance<\/span><\/a><\/p>\n<p><span data-preserver-spaces=\"true\">But the problem with running tests during the holidays is that the results might easily be skewed. This means that you can&#8217;t rely on holiday test results to make permanent changes to your website.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Remember, during holidays; you&#8217;re dealing with consumers who are driven by the cost and inventory \u2013 this means that their behavior is different from your usual customers during the year.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">For instance, holiday shoppers tend to have a higher sense of urgency, lower bounce rates, and more engagement with particular items. They also tend to have more intent than regular browsing \u2013 meaning that there are more likely to purchase their first visit.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Most online retailers tend to increase their Ad spend during holidays to attract more customers. And this sometimes means bringing in an audience from new or less popular sources \u2013 this kind of audience is more likely to have a different mindset because they are not within your usual target audience.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">During the holiday season, what might have worked might not be as efficient during the rest of the year. In other words, your winning test during the holiday might not be the winner when you test it on regular days.\u00a0<\/span><\/p>\n<p>ALSO READ: <a href=\"https:\/\/www.invespcro.com\/blog\/holiday-marketing-strategies\/\">Holiday Marketing Strategies: 11 Tactics Every Small Business Should Leverage<\/a><\/p>\n<p><span data-preserver-spaces=\"true\">So, relying on the insights obtained during holiday testing is something you should carefully think about. Before you think of using the insights you gained from holiday tests, we recommend rerunning your holiday tests on regular days to validate your results.\u00a0<\/span><\/p>\n<h2><strong><span data-preserver-spaces=\"true\">Conclusion: Yes, holiday testing is worth it!<\/span><\/strong><\/h2>\n<p><span data-preserver-spaces=\"true\">Testing during the holiday is a good idea. You can still gain some valuable insights that you can use during high urgency periods.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">All you have to do is put in place an A\/B testing strategy and be mindful of the unique visitor behavior you will be dealing with.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Your testing strategy should also include conducting extensive quality assurance. You should also test high-impact elements \u2013 product page descriptions, value propositions, return policy, promoted products, etc. \u2013 and not any other elements.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Avoid testing variations that have to do with the functionality of your website, as they may cause some technical issues on your website.\u00a0<\/span><\/p>\n<p>ALSO READ: <a href=\"https:\/\/www.invespcro.com\/blog\/10-simple-ways-to-boost-your-holiday-conversion-rate\/\">10 simple ways to boost your holiday conversion rate<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p><span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\"> 7<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span>Disclaimer: This section is a TL;DR of the main article, and it\u2019s for you if you\u2019re not interested in reading the whole article. On the other hand, if you want to read the full blog, just scroll down, and you\u2019ll see the introduction. Some online retailers take weeks, while some take months to plan out [&hellip;]<\/p>\n","protected":false},"author":54,"featured_media":15178,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[116,11],"tags":[],"class_list":["post-15177","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ab-testing","category-ecommerce"],"_links":{"self":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts\/15177","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/users\/54"}],"replies":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/comments?post=15177"}],"version-history":[{"count":0,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts\/15177\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/media\/15178"}],"wp:attachment":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/media?parent=15177"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/categories?post=15177"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/tags?post=15177"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}