• A/B testing on Holidays: Is it Worth it? 

    An image showing an AB test variation to a website visitor trying to take advantage of the website’s holiday offer.

    How far out do you need to plan your holiday A/B testing strategy?

    Some online retailers take two weeks to plan. Some take a couple of months. And, others need a few weeks to prepare. 

    There seems to be no right or wrong answer to that question. 

    At some point, I used to think that it was impossible to come up with and execute a successful holiday testing program in few days. 

    That was before I witnessed our CRO team create and execute one three days before Black Friday. 

    Yes, it’s a true story! 

    Three days before the Black Friday of 2019, an eCommerce company signed up with us and tasked us to handle their holiday testing program. 

    Seasonally events – like Black Fridays and Cyber Mondays – need to be approached thoughtfully, and you need all the customer data you can get to execute successful campaigns. 

    If they hadn’t provided us with insights on their customers and inundated us with a ton of information – we would not have pulled off a successful campaign. We also needed to compare the traffic and their activity from the prior year to gauge customer behaviors during this period.”

    – says Ayat, who was also part of our CRO team that created and executed a testing program in 3 days. 

    In other words, If you’re entirely unfamiliar with the website, the market, and the customers, then your A/B testing program is more likely to be a disaster.

    When it comes to holiday campaigns, it’s not about the promotions – but rather the marketing strategies will engage and persuade the visitor during and keep the visitor loyal post.

    In this article, we will look at why companies hate holiday testing and the pros and cons of running tests during holidays. We are also going to touch on the dos and don’ts of holiday testing.

    Why do companies hate A/B testing during holidays?

    Most online retailers are not willing to run A/B tests during holidays – and this is something we have seen with some of our clients. 

    In our experience, there are two reasons why companies want to go through a code freeze during holidays: 

    Fear of introducing new bugs

    You launch a test with two or three variations against the control. And you think that your testing tool is collecting correct data. You then discover that one of the variations is bugged – and it’s not working for Safari users. 

    Whether it’s a holiday season or not, there’s a possibility of new bugs creeping in your variations when A/B testing – especially when your A/B testing has to do with the website functionality. 

    In his article – about A/B testing sampling issues – Khalid writes:  

    FigPii Heatmaps

    Bugs are typically introduced while coding the test variations. Since most tests are coded using JavaScript and not served from the server-side, there is a good chance that one of your variations might contain bugs.”

    Fortunately, there’s a way to discover the bugs before the test is launched.  

    We usually use extensive Quality Assurance (QA) on a staging environment and addressing any bugs (if any) before moving the test to production. 

    In other words, no test has to be pushed to the production servers before the data is validated in a QA environment. 

    Risk of reduction in revenue as a result of new designs

    Whenever you launch a test, you are not 100% sure that it will increase conversions. That’s why you have to test it first. 

    In other words, with every A/B test you launch, there’s a risk of decreasing conversions. 

    So some clients would instead go with the original design and not risk experimenting – especially during the holiday season where they’re expecting to cash in lots of revenue. 

    So, how can this be mitigated, you ask? 

    Well, the solution that we have for a lot of our clients is to do gradual testing. 

    For example, instead of directing 50% of the traffic to the new design, we start by 10% and 90% to the original design. And then we gradually increase or decrease the traffic according to how o how the new design is performing. 

    This way, our clients feel more comfortable running A/B tests during holidays. 

    Holiday testing – Is it worth it? 

    Most online retailers experience an influx of visitors during holiday seasons. 

    Does that mean you should take advantage of this seasonal spike in traffic and launch some tests? 

    Well, that’s a good but tricky question. 

    I have to say, testing during holidays comes with its own set of risks and rewards. 

    You have to be experienced in the field of CRO if you want your tests to be successful. 

    ALSO READ: What they don’t tell you about A/B testing

    Why you should AB test during Holidays

    It’s not every day that you get an increase in: 

    • New visitors to your site, 
    • Comparison shoppers, who are driven by product price and availability,
    • Visitors from new or less popular sources (and sometimes locations), 
    • And average order and cart size. 

    – so, not running tests during holidays can feel like a missed opportunity. 

    You might also be leaving more money on the table by not testing during holidays that you could have got through new variations. 

    For instance, how do you know that your holiday offer is performing as well as it could be? If you are not running tests, you will never understand what kind of offer drives your customers during the holiday season.

    In such periods of high urgency, you can quickly learn how your customers think, behave, and what kind of offers drive them to take the desired action on your site. 

    There are tons of A/B testing tools that have editor WYSIWYG that allows you to create variations without touching your site’s code – especially when the variations don’t change the page functionality. 

    Using the above tools, you don’t have to change the code to remove, add, resize, or even rearrange your web page copy.  

    Why you shouldn’t A/B test during Holidays

    A/B testing is not just about trying to maximize revenue – at its core, it’s about understanding how your visitors think and behave by analyzing the test results. 

    ALSO READ: How to analyze A/B testing results and statistical significance

    But the problem with running tests during the holidays is that the results might easily be skewed. This means that you can’t rely on holiday test results to make permanent changes to your website. 

    Remember, during holidays; you’re dealing with consumers who are driven by the cost and inventory – this means that their behavior is different from your usual customers during the year. 

    For instance, holiday shoppers tend to have a higher sense of urgency, lower bounce rates, and more engagement with particular items. They also tend to have more intent than regular browsing – meaning that there are more likely to purchase their first visit. 

    Most online retailers tend to increase their Ad spend during holidays to attract more customers. And this sometimes means bringing in an audience from new or less popular sources – this kind of audience is more likely to have a different mindset because they are not within your usual target audience. 

    During the holiday season, what might have worked might not be as efficient during the rest of the year. In other words, your winning test during the holiday might not be the winner when you test it on regular days. 

    ALSO READ: Holiday Marketing Strategies: 11 Tactics Every Small Business Should Leverage

    So, relying on the insights obtained during holiday testing is something you should carefully think about. Before you think of using the insights you gained from holiday tests, we recommend rerunning your holiday tests on regular days to validate your results. 

    Conclusion: Yes, holiday testing is worth it!

    Testing during the holiday is a good idea. You can still gain some valuable insights that you can use during high urgency periods. 

    All you have to do is put in place an A/B testing strategy and be mindful of the unique visitor behavior you will be dealing with. 

    Your testing strategy should also include conducting extensive quality assurance. You should also test high-impact elements – product page descriptions, value propositions, return policy, promoted products, etc. – and not any other elements. 

    Avoid testing variations that have to do with the functionality of your website, as they may cause some technical issues on your website. 

    ALSO READ: 10 simple ways to boost your holiday conversion rate

Simbar Dube

Simba Dube is the Growth Marketing Manager at Invesp. He is passionate about marketing strategy, digital marketing, content marketing, and customer experience optimization.

View All Posts By Simbar Dube

Join 25,000+ Marketing Professionals

If you enjoyed this post, please consider subscribing to the Invesp blog feed to have future articles delivered to your feed reader. or,receive weekly updates by email: