{"id":15396,"date":"2022-03-30T07:16:29","date_gmt":"2022-03-30T12:16:29","guid":{"rendered":"https:\/\/www.invespcro.com\/blog\/?p=15396"},"modified":"2022-03-30T07:16:29","modified_gmt":"2022-03-30T12:16:29","slug":"should-you-run-the-same-test-across-all-devices","status":"publish","type":"post","link":"https:\/\/www.invespcro.com\/blog\/should-you-run-the-same-test-across-all-devices\/","title":{"rendered":"Should You Run The Same Test Across All Devices?"},"content":{"rendered":"Reading Time: <\/span> 4<\/span> minutes<\/span><\/span>

Should you run the same test across all devices?\u00a0<\/span><\/p>\n

Or let me rephrase, should you be A\/B testing on mobile or desktop separately, or not?\u00a0<\/span><\/p>\n

Well, the correct answer is of course that you should run separate tests on mobile and desktop.\u00a0<\/span><\/p>\n

That is a no-brainer\u2026<\/span><\/p>\n

But when you answer that question, the way I did, you generate another smaller, tangential question such as:\u00a0<\/span><\/p>\n

Why should you A\/B test on mobile and desktop separately?\u00a0<\/span><\/p>\n

Now, there are dozens of reasons why you should test separately. In this article, I will give you five reasons:<\/span><\/p>\n

User Behavior Varies by Device\u00a0<\/span><\/h2>\n

It\u2019s no secret that mobile and desktop visitors exhibit different behaviors. Most people tend to browse using their mobiles and complete the purchase using their personal computers.\u00a0<\/span><\/p>\n

Understanding users\u2019 motivations for visiting your site can help you with insights that may be used as the foundation for how to optimize different experiences on your website.\u00a0<\/span><\/p>\n

Even if both mobile and desktop users use the search on your website, how they go about it is not similar.\u00a0<\/span><\/p>\n

In general, mobile device users don\u2019t want to spend time idling on your site. Their aim is to get things done as efficiently as possible. But that\u2019s not the case when it comes to desktop users \u2013 they are more open to having more elements on a website.\u00a0<\/span><\/p>\n

These nuances show that user behavior is something that is highly dependent upon the type of device being used. It\u2019s essential that you understand your audience\u2019s preferences then come up with an A\/B testing strategy for each device type.\u00a0<\/span><\/p>\n

What works for Mobile doesn\u2019t necessarily work for desktop\u00a0<\/span><\/h2>\n

This point sounds like a bleed-off from the previous section. But, here me out, it deserves a whole section of its own.\u00a0<\/span><\/p>\n

Some of the web elements that help increase conversions on desktop may actually affect conversions on mobile devices. In other words, if it works for desktop, it doesn\u2019t necessarily mean that it will work for mobile.\u00a0<\/span><\/p>\n

This is the reason why you see some sites \u2013 after being optimized for conversions \u2013 having different experiences on mobile and desktop. For instance, when you are on Facebook using your desktop you will find that you have too many choices than when you are on a mobile device.\u00a0<\/span><\/p>\n

Over the years, we have seen winning designs for a mobile test being very different from the winning design of the desktop test of the same page. This, therefore, means that you will have to serve two different experiences \u2013 on the same website \u2013 based on the device type that the visitor is using.\u00a0<\/span><\/p>\n

Having two different web designs \u2013 mobile and desktop \u2013 for the same webpage can lead to development problems. This is why some companies tend to avoid being in this situation by using a similar design. But that doesn\u2019t necessarily solve the problem, if anything, it ignores it.\u00a0<\/span><\/p>\n

Traffic Varies By Devices<\/span><\/h2>\n

Recent research indicates that mobile usage continues to increase (with no signs of slowing down) at a faster rate when compared to desktop. More users are using their mobile devices to surf the web, shop, and make purchasing decisions.<\/span><\/p>\n

Mobile and desktop traffic are never the same. Some websites have more traffic coming from their mobile devices and less traffic from desktop devices. But, for some websites, it\u2019s vice-versa.\u00a0<\/span><\/p>\n

So, when you run the same test, at the same time, on both mobile and desktop, how are you going to split the traffic evenly? Take a few seconds to think about it\u2026<\/span><\/p>\n

You may have enough sample size combined, but when you look at the data of your test, you will notice that one device type contributed more traffic than the other.\u00a0<\/span><\/p>\n

And this means that at the end of the day, whatever result you have, it won\u2019t be specific to a particular device. It will be on average \u2013 and as you may (or may not) know, you can\u2019t make decisions based on \u2018on average\u2019 results.\u00a0\u00a0<\/span><\/p>\n

In trying to come up with a solution for this, people end up extending the test duration so as to get enough sample size on each device. When this happens, you will be eating into time that you could be using to prepare for other tests.\u00a0<\/span><\/p>\n

Less Development and QA time\u00a0<\/span><\/h2>\n

When you run an A\/B test on desktop and mobile devices at the same time, this means more development and QA work for your team. And suppose the test is complicated, you will need more development and QA time. When this happens, you end up eating into time that could have been working on other tests.\u00a0<\/span><\/p>\n

But if you test separately, on desktop and mobile, the development won\u2019t be as complicated and the QA time spent on each test won\u2019t be that much.\u00a0<\/span><\/p>\n

Mobile traffic purchase behavior varies<\/span><\/h2>\n

Mobile users are now willing to spend more and more money. In Q1 of\u00a0 2019, Paypal\u2019s mobile payment volume increased by<\/span> 52%<\/span><\/a>.\u00a0<\/span><\/p>\n

Recent research also shows that in the last six months, <\/span>79%<\/span><\/a> of smartphone users have purchased something online using their mobile devices.\u00a0<\/span><\/p>\n

And according to <\/span>Google<\/span><\/a>, 20% of smartphone users make a purchase online on a daily basis.\u00a0<\/span><\/p>\n

There\u2019s no denying that the number of mobile users who are using their smartphones to purchase is still increasing even to this day.\u00a0<\/span><\/p>\n

But when you take a deep dive into the purchase behavior of mobile users, you will see that it varies. Even though Android users are the majority on most websites, research shows that <\/span>iPhone users tend to spend more<\/span><\/a> than any other smartphone user.\u00a0<\/span><\/p>\n

What we have seen over the years is that what works for iOS doesn\u2019t necessarily mean that it will also increase conversions on Android or Windows mobile devices.\u00a0<\/span><\/p>\n

Perhaps, the best way to solve this is to segment your users by their mobile operating system and target them separately or even better, do a post-test analysis and check how each operating system performed.\u00a0<\/span><\/p>\n

Another segmentation to think about is to test mobile devices using wifi and mobiles on 5G\/4G separately as that could show different buying behaviors.\u00a0<\/span><\/p>\n

Conclusion\u00a0<\/span><\/h2>\n

For reasons mentioned in this article, it\u2019s always recommended to test on mobile and desktop separately. Doing so will make your life easier and you will be able to run more and more tests without any development complications.\u00a0<\/span><\/p>\n

Additional Resources<\/p>\n

    \n
  1. A\/B Testing Guide: Everything You Need To Know About A\/B Testing<\/a><\/li>\n
  2. What is AB testing (split testing)?<\/a><\/li>\n
  3. What is Multivariate Testing?<\/a><\/li>\n<\/ol>\n","protected":false},"excerpt":{"rendered":"

    Reading Time: <\/span> 4<\/span> minutes<\/span><\/span>Should you run the same test across all devices?\u00a0 Or let me rephrase, should you be A\/B testing on mobile or desktop separately, or not?\u00a0 Well, the correct answer is of course that you should run separate tests on mobile and desktop.\u00a0 That is a no-brainer\u2026 But when you answer that question, the way I […]<\/p>\n","protected":false},"author":50,"featured_media":15397,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[116],"tags":[],"class_list":["post-15396","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ab-testing"],"_links":{"self":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts\/15396","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/users\/50"}],"replies":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/comments?post=15396"}],"version-history":[{"count":0,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts\/15396\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/media\/15397"}],"wp:attachment":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/media?parent=15396"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/categories?post=15396"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/tags?post=15396"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}