{"id":14915,"date":"2021-06-24T06:04:58","date_gmt":"2021-06-24T11:04:58","guid":{"rendered":"https:\/\/www.invespcro.com\/blog\/?p=14915"},"modified":"2021-06-24T06:04:58","modified_gmt":"2021-06-24T11:04:58","slug":"expert-answers-to-9-tough-cro-questions","status":"publish","type":"post","link":"https:\/\/www.invespcro.com\/blog\/expert-answers-to-9-tough-cro-questions\/","title":{"rendered":"Expert Answers to 9 Tough CRO Questions"},"content":{"rendered":"<span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\"> 18<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span><p><span style=\"font-weight: 400;\">Conversion rate optimization is always going to be complicated.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">It was complicated in its early days; it still is complicated today, and it will still be tricky tomorrow.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">This is why there\u2019s no one universal process to approach it. Different CRO experts use different methods when optimizing websites.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">But no matter how you approach conversion optimization, they will always be burning questions that will make you scratch your head before you answer them.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">I compiled 9 of those tough questions and reached out to five conversion optimization experts from different parts of the world. The point here is to understand different ways to look at the same conversion optimization problem.\u00a0<\/span><!--more--><\/p>\n<p><span style=\"font-weight: 400;\">Well, without further ado, let\u2019s meet the CRO experts and learn some new tricks, shall we? <\/span><\/p>\n<p>&nbsp;<\/p>\n<div class=\"speaker-list-item\">\n<p><img decoding=\"async\" class=\"alignnone size-full wp-image-14919 speaker-list-item-image\" src=\"https:\/\/www.invespcro.com\/blog\/images\/blog-images\/Tim-Stewart--e1623322901209.jpeg\" alt=\"\" width=\"200\" height=\"200\" data-wp-pid=\"14919\" \/><\/p>\n<div><a href=\"https:\/\/www.linkedin.com\/in\/timstewart\/\">Tim Stewart<\/a>, Digital Consultant at <a href=\"https:\/\/trsdigital.com\/\">trsdigital<\/a><\/div>\n<\/div>\n<div class=\"speaker-list-item\">\n<p><img decoding=\"async\" class=\"alignnone size-full wp-image-14918 speaker-list-item-image\" src=\"https:\/\/www.invespcro.com\/blog\/images\/blog-images\/Georgi-e1623322877488.jpeg\" alt=\"\" width=\"200\" height=\"200\" data-wp-pid=\"14918\" \/><\/p>\n<div><a href=\"https:\/\/www.linkedin.com\/in\/geoprofi\/\">Georgi Georgiev<\/a>, Founder of <a href=\"https:\/\/www.analytics-toolkit.com\/\">Analytics-Toolkit.com<\/a><\/div>\n<\/div>\n<div class=\"speaker-list-item\">\n<p><img decoding=\"async\" class=\"alignnone size-full wp-image-14711 speaker-list-item-image\" src=\"https:\/\/www.invespcro.com\/blog\/images\/blog-images\/Lorenzo.jpeg\" alt=\"\" width=\"200\" height=\"200\" data-wp-pid=\"14711\" \/><\/p>\n<div><a href=\"https:\/\/www.linkedin.com\/in\/lorenzocarreri\/\">Lorenzo Carreri<\/a>, CRO, User Research and UX Consultant<\/div>\n<\/div>\n<div class=\"speaker-list-item\">\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-14916 speaker-list-item-image\" src=\"https:\/\/www.invespcro.com\/blog\/images\/blog-images\/Andre-Morys--e1623322853514.jpeg\" alt=\"\" width=\"200\" height=\"200\" data-wp-pid=\"14916\" \/><\/p>\n<div><a href=\"https:\/\/www.linkedin.com\/in\/andremorys\/\">Andr\u00e9 Morys<\/a>, Founder &amp; CEO of <a href=\"https:\/\/www.konversionskraft.de\/\">konversionsKRAFT<\/a><\/div>\n<\/div>\n<div class=\"speaker-list-item\">\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-14917 speaker-list-item-image\" src=\"https:\/\/www.invespcro.com\/blog\/images\/blog-images\/Chad-Sanderson--e1623322831845.jpeg\" alt=\"\" width=\"200\" height=\"200\" data-wp-pid=\"14917\" \/><\/p>\n<div><a href=\"https:\/\/www.linkedin.com\/in\/chad-sanderson\/\">Chad Sanderson<\/a>, Head of Product, Data Platform at Convoy<\/div>\n<\/div>\n<div>\n<h2><strong>1. Are there any risks involved in an optimization project?<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">Every project comes with risks. That&#8217;s probably a known fact. But does this ring true in the world of conversion optimization? If it&#8217;s true, what kind of risks are there in a CRO project? And how can you avoid falling victim to them?\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Well, according to Georgi:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">Yes, similar to any other project. The difference is that most optimization projects rely on experiments to manage those risks, so usually, such projects involve much less risk than other types of marketing or product development work. This means that if your CROs do not know what they are doing, this will become evident during testing (assuming the tests are done properly), which is not necessarily the case with PPC or SEO. While experiments are possible in PPC, I don&#8217;t think they are as widespread as in CRO. Experiments in SEO are very difficult if not outright impossible and of much more limited value<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Lorenzo says that he always see a couple of risks in optimization projects:<\/span><b>\u00a0<\/b><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">The first one involves the people running an optimization program without proper experience. The second thing that could also be risky is optimizing for the wrong KPI. Most people always obsess about optimizing the conversion rate. But that&#8217;s just one side of the coin in terms of trying to achieve the company&#8217;s overall goal. Sometimes, there could be some opportunity to increase the AOV or retention rate, but if you are only obsessed with conversion rate, you might miss other important opportunities<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;\u00a0<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">It&#8217;s funny that CRO should help you avoid risks, and yet it comes with its own set of possible risks, says Andr\u00e9. He also goes on to mention three potential risks you should avoid when doing conversion optimization:\u00a0<\/span><\/p>\n<blockquote><p><i><span style=\"font-weight: 400;\">&#8220;CRO should be about risk mitigation, but it also has its own risks. If you do not know what to optimize and why you don&#8217;t need to A\/B test anything, you could save a lot of money by not doing A\/B testing and optimization.\u00a0<\/span><\/i><\/p>\n<p><i><span style=\"font-weight: 400;\">The first risk associated with CRO is testing weak hypotheses.\u00a0<\/span><\/i><\/p>\n<p><i><span style=\"font-weight: 400;\">The second risk is weak operationalization. This usually happens when you test things and deduce the wrong assumptions from the results. Sometimes you may have already tested something, not get positive results, and then decide to stop at that point. But, that doesn&#8217;t mean they might not be another way of testing it.\u00a0<\/span><\/i><\/p>\n<p><i><span style=\"font-weight: 400;\">The third risk is the wrong implementation. Sometimes your variations may work in another browser, and you think that&#8217;s alright, and you forget to test on other browsers and devices. That&#8217;s a huge risk that can cost you a lot of money.&#8221;<\/span><\/i><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Chad says there are dozens of risks in a CRO. He also warns against implementing a test without <a href=\"https:\/\/www.figpii.com\/blog\/quality-assurance-for-ab-testing-best-practices\/?utm_source=invespcro&amp;utm_medium=inline-link&amp;utm_campaign=backlink\">proper Quality Assurance<\/a>:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">There are many risks, and one I can think about is the implementation of the test. Rolling out an experiment without doing Quality Assurance. We once ran an experiment where we changed the priority order of a list of offers for our customers. When the experiment was rolled out, the swipe to go down the page wasn&#8217;t working. There was a bug. And because of that bug, we saw a pretty big negative impact<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;\u00a0<\/span><\/p><\/blockquote>\n<p><i><span style=\"font-weight: 400;\">&#8220;Nothing of value is without risks<\/span><\/i><span style=\"font-weight: 400;\">,&#8221; says Tim. He also goes on to say, &#8220;<\/span><i><span style=\"font-weight: 400;\">the biggest risk in experimentation is people not doing it in the way it is intended and trying to treat it like some kind of shortcut to money<\/span><\/i><span style=\"font-weight: 400;\">.&#8221; Tim also mentions different kinds of CRO risks you should look out for:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">Another risk is to try CRO, but not give it the correct thinking, and you continue to do the wrong thing. Then there&#8217;s the risk of making decisions based on too little data on the false assumptions that some data is better than no data and making the wrong decisions believing that they are the right ones.<\/span><\/i><\/p><\/blockquote>\n<h2><strong>2. What comes first, CRO, SEO, or PPC?<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">If you are reading this, you probably know the difference between CRO, SEO, and PPC. But do you know which one should come first? If CRO has to come first, why is that so? Well, Georgi says they should work hand-in-glove and not as foes:\u00a0<\/span><\/p>\n<blockquote><p><i><span style=\"font-weight: 400;\">&#8220;I&#8217;ve always seen them as complementary and not as mutually exclusive options.&#8221;<\/span><\/i><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Andr\u00e9<\/span> <span style=\"font-weight: 400;\">agrees with Georgi, as he says, &#8220;traffic multiplied with conversion rate equals revenue.&#8221; But he also adds that CRO problems are usually the plague of most businesses:<\/span><\/p>\n<blockquote><p><i><span style=\"font-weight: 400;\">&#8220;So, of course, if you have constraints in your traffic acquisition, it&#8217;s as bad as having problems with your conversion rate. The thing is, I strongly believe that SEO has a 10 years maturity advantage. The SEO industry is quite mature, the knowledge is quite mature, and it&#8217;s not that complex. I see much more companies having much more conversion rate challenges compared to traffic acquisition. But in general, it&#8217;s two independent variables.&#8221;<\/span><\/i><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">On the other hand, Chad says CRO should always be given the first priority because it enables you to unlock your customer pain points:\u00a0<\/span><\/p>\n<blockquote><p><i><span style=\"font-weight: 400;\">&#8220;CRO should always be first. But the problem is that most people think of good CRO as a particular set of techniques. And that&#8217;s not the whole idea of CRO. Experimentation and CRO is simply the thought process of thinking about the problems of our customers. We need to build engineering solutions to solve those problems. And then, we need to measure the results. So we have some idea whether it worked or not&#8230;that should be the core of every business. You can always keep your business strong \u2013 even without SEO or PPC \u2013 if you build more things and test to see if they work.\u00a0\u00a0<\/span><\/i><\/p>\n<p><i><span style=\"font-weight: 400;\">A great example of this is Airbnb. They cut their marketing spend by 25% during the pandemic, and they didn&#8217;t see a significant drop in sales. The reason that people use Airbnb is not because of the marketing. It&#8217;s because they&#8217;ve developed an excellent, fantastic product that people love to use, and it solves a problem. I&#8217;m not saying that marketing is terrible. SEO, PPC, or Facebook ads can be a good solution. Especially if customers have never heard of you.&#8221;\u00a0<\/span><\/i><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Lorenzo agrees with both Georgi and Andr\u00e9 in putting CRO, SEO, and PPC in the same box,<\/span><i><span style=\"font-weight: 400;\"> &#8220;they should all work together.&#8221; <\/span><\/i><span style=\"font-weight: 400;\">But he also says that if you are to choose which one you should start with first, go for CRO.\u00a0<\/span><\/p>\n<blockquote><p><i><span style=\"font-weight: 400;\">&#8220;But it&#8217;s better to start by doing CRO and figuring out \u2013 from that perspective \u2013 the customers&#8217; decision-making process. And based on those findings, then you can create an SEO or PPC strategy.&#8221;\u00a0<\/span><\/i><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Tim says that what comes first is determined by the kind of problem you have on your site. He also goes on to say that not all CRO issues should be given first priority over SEO or PPC:\u00a0<\/span><\/p>\n<blockquote><p><i><span style=\"font-weight: 400;\">&#8220;Sometimes, when clients approach me, I look at their data, and they don&#8217;t have a CRO problem. Or, if they do have a CRO problem, it might not be the problem they should be solving first, so I tell them to fix their SEO or PPC. Part of it is down to maturity or situation.\u00a0<\/span><\/i><\/p>\n<p><i><span style=\"font-weight: 400;\">Sometimes, I look at what type of CRO program fits their business. I mean, if their Search Engine Marketing is poor, then the conversion optimization effort can be spent on trying to understand why it is performing poorly. This can help us answer the following questions:\u00a0<\/span><\/i><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><em><span style=\"font-weight: 400;\">Are they communicating the wrong message to the wrong people?\u00a0<\/span><\/em><\/li>\n<\/ul>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><em><span style=\"font-weight: 400;\">Do they have the right message but to the wrong people?\u00a0<\/span><\/em><\/li>\n<\/ul>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><em><span style=\"font-weight: 400;\">Or maybe they have the wrong product-market-fit?.\u00a0<\/span><\/em><\/li>\n<\/ul>\n<p><i><span style=\"font-weight: 400;\">Ultimately, CRO, SEO, and PPC are all tied together. The idea is to work towards the ultimate end-goal of the business and not try to isolate.&#8221;<\/span><\/i><\/p><\/blockquote>\n<h2><strong>3. Should CRO be framed as a marketing initiative or a business strategy?<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">Before you read the experts&#8217; answers, what&#8217;s your take on this? Think about it for a second. Most companies tend to CRO as a marketing initiative. But is it the right way of going about it, or should they turn it into a business strategy?\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In this regard, Georgi says:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">In my mind, it [CRO] sits between marketing and product development, but I&#8217;ve not delved on the issue of taxonomy too much.<\/span><\/i><span style=\"font-weight: 400;\">&#8220;<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Lorenzo says CRO shouldn&#8217;t be considered as a marketing initiative or business strategy, &#8220;<\/span><i><span style=\"font-weight: 400;\">It should be pitched more as a process or mindset, and not as a business strategy or marketing initiative.&#8221; <\/span><\/i><span style=\"font-weight: 400;\">He also adds,<\/span><i><span style=\"font-weight: 400;\"> &#8220;this is because CRO allows you to learn a lot of valuable things that other departments \u2013 like product development and customer support \u2013 can take advantage of.<\/span><\/i><span style=\"font-weight: 400;\">&#8221;\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Andr\u00e9 agrees with Lorenzo, but he also adds that calling it CRO is wrong:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">100% I would always try to frame CRO as a mindset, give it a different name towards management level and frame it as a strategic cultural mindset in the company. I don&#8217;t even think CRO is the proper term to address its importance towards the management level. As soon as you&#8217;re talking CRO, you&#8217;re talking tactics, and management is not interested in tactics.\u00a0<\/span><\/i><\/p>\n<p><i><span style=\"font-weight: 400;\">CRO is basically a mindset of analyzing a challenge carefully, being very customer-centric, understanding what customers want, then deducing hypotheses and prioritizing them to find out what has the business impact<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;\u00a0<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Chad is wary about framing CRO as a marketing initiative. He believes that framing CRO as a marketing initiative would breed another problem of comparisons:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">If you frame it as a marketing initiative, then it is always going to be compared to other marketing efforts like email campaigns, advertising campaigns that have a clear line to revenue. When using Google ads, you will have a dashboard, and it will show you how many conversions have resulted from those Google ads. But with optimization, this isn&#8217;t always the case, and it&#8217;s not even the way you should use experimentation. Sometimes you just want to run experiments to learn things, and you&#8217;re not expecting any type of ROI.<\/span><\/i><span style=\"font-weight: 400;\">&#8220;<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Instead of framing CRO as a marketing initiative or business strategy, he suggests treating it as a department that is as important as other departments in a company:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">CRO should be framed as a business strategy. In fact, CRO should be treated the same way someone would treat a security team. You don&#8217;t invest in a security team because you expect it to make you money. You invest in a security team because it makes sense to be safe. And with CRO, it makes sense to always be learning and be making reliable measurements<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">According to Tim, the main reason why most people treat CRO as a marketing initiative is that &#8220;<\/span><i><span style=\"font-weight: 400;\">it has the quickest and most evident effect on KPIs that are traditionally used by marketers<\/span><\/i><span style=\"font-weight: 400;\">.&#8221; But also goes on to support the idea of treating conversion optimization as a business strategy:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">CRO works best when it is planned as a business-wide strategy that helps to improve the whole business. The whole business should have an experimentation mindset and determination to get better at what they do<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;\u00a0<\/span><\/p><\/blockquote>\n<h2><strong>4. What&#8217;s the key to an effective Conversion Strategy?<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">The importance of a conversion strategy cannot be overstated. Whether you have a small or enterprise business, a well-crafted conversion strategy will always benefit your company one way or the other. But is there anyone thing key element that you should always include in your conversion strategy:\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Georgi says everything involved in CRO is equally important; you can&#8217;t single out one thing: <\/span><\/p>\n<blockquote><p><em><span style=\"font-weight: 400;\">&#8220;Quality of the data, analysis, <\/span><a href=\"https:\/\/www.invespcro.com\/blog\/expert-advice-on-developing-a-hypothesis-for-marketing-experimentation\/\"><span style=\"font-weight: 400;\">hypothesis generation<\/span><\/a><span style=\"font-weight: 400;\">, prioritization, and statistical testing are all crucial. Get one wrong, and the rest will function much worse than optimal.&#8221;<\/span><\/em><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Andr\u00e9 also agrees with Georgi on this one, <\/span><i><span style=\"font-weight: 400;\">&#8220;the big challenge with conversion optimization is that it is not one thing. It&#8217;s a connection of disciplines, and it&#8217;s always a collective effort.&#8221; <\/span><\/i><span style=\"font-weight: 400;\">He also says that most business struggle because they make the mistake of focusing on one element:\u00a0\u00a0<\/span><\/p>\n<blockquote><p><i><span style=\"font-weight: 400;\">&#8220;Many companies fail because they see it as a data thing, and they completely forget about the customer. But if I am to condense it, I would single out two things, and that&#8217;s being customer-centric and data-driven.&#8221;<\/span><\/i><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Chad believes that the essential thing to do is to focus on customers:\u00a0<\/span><\/p>\n<blockquote><p><i><span style=\"font-weight: 400;\">&#8220;I think the key is to focus on problems, not solutions. Often, people come up with ideas like changing button colors and thinking that will increase their conversions. But that&#8217;s starting with a solution and assuming that there&#8217;s going to be an impact. Customers have so many problems. Reach out to them and ask about their pain points. Sometimes you might find that their pain point is not even related to your website. Maybe it&#8217;s the customer service that&#8217;s causing people not to buy anything. So, focusing on customer problems is the key to a conversion strategy.&#8221;\u00a0<\/span><\/i><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Tim says that the crucial thing to do in your conversion strategy is to align CRO and business goals. He also advises against copying your competitors or other more prominent brands in your industry:<\/span><b>\u00a0<\/b><\/p>\n<blockquote><p><i><span style=\"font-weight: 400;\">&#8220;There&#8217;s no one thing because there are a dozen things that make this up. But you have to align your CRO program with your business. People often make the mistake of copying bigger brands just because they do a lot of testing. They copy other brands and forget that they don&#8217;t have that ability; they don&#8217;t have the budget; they don&#8217;t have the traffic; they don&#8217;t have an audience that fits what they see big brands doing. And so they try and do a conversion strategy that doesn&#8217;t fit them. The secret of success in a conversion strategy is matching your ability against your needs. There has to be a balance between your goals and ability.&#8221;<\/span><\/i><\/p><\/blockquote>\n<h2><strong>5. How do you calculate the revenue impact of a CRO program?<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">If you ask any seasoned optimizers, they will tell you that calculating the revenue impact when running a CRO program is challenging. Most optimizers don&#8217;t really know if they should be doing it or not? But should you really calculate the revenue or impact? If yes, how do you do it?\u00a0<\/span><\/p>\n<p>According to Georgi:<\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;It&#8217;s<\/span><i><span style=\"font-weight: 400;\"> one part business accounting, one part meta-analysis of projections from the estimated effects from initial and confirmatory tests, and one part art. The last one refers to anticipated changes in the entire environment and how that would affect the future performance of the implemented changes, and it becomes more of a factor for longer-term estimates of the revenue impact<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;\u00a0<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Lorenzo says that &#8220;<\/span><i><span style=\"font-weight: 400;\">instead of trying to calculate the impact, you should be focusing more on understanding the learnings that the program is giving you. And learn how to use the process to make decisions based on a scientific methodology.<\/span><\/i><span style=\"font-weight: 400;\">&#8221;\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Andr\u00e9 says that there&#8217;s no way you can calculate the revenue and the impact of a CRO program, &#8220;that&#8217;s<\/span><i><span style=\"font-weight: 400;\"> a big myth with no one answer. If you reduce experimentation to direct business impact, it becomes hard to calculate. And that&#8217;s not even the real value of experimentation<\/span><\/i><span style=\"font-weight: 400;\">.&#8221; He also adds that:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">There are many other values \u2013 faster learnings, better insights, more agility, and customer-centricity \u2013 you can get from experimentation. But you have to always ask your clients for what they consider to be non-financial values of expectation. And you try to pile them up and make sure the value is perceived from expectation<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Avoiding the revenue impact is the best thing to do, says Chad. He goes on to say that big brands don&#8217;t even bother to do the calculations:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">The best way of thinking about it is to avoid calculating the revenue impact. Look at the top companies in the world that are doing experimentation, Google, Twitter, Amazon. You will see that they don&#8217;t calculate the revenue impact of an optimization program. They understand that doing conversion optimization is as essential as having a security team.\u00a0<\/span><\/i><\/p>\n<p><i><span style=\"font-weight: 400;\">So, the value of experimentation and conversion optimization lies in having a reliable measurement. Metrics will always be variable, customer behaviors will change due to certain seasons, and if you don&#8217;t have experimentation, you won&#8217;t know what might be causing changes on your website.<\/span><\/i><span style=\"font-weight: 400;\">&#8221;\u00a0<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Although Tim also believes that there&#8217;s no way you should calculate the revenue impact, he also thinks that you should define what a CRO success is to you and your clients before you even start the project:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">The truth of the matter is that there&#8217;s no correct answer or the right number. You can&#8217;t say, &#8220;I did this, and it resulted in this many dollars and cents.&#8221; That&#8217;s not possible. Can we estimate? Yes, but we will be basing on some assumptions, parameters, and data. If any of those parameters and data change or are incorrect, whatever you forecast as a potential impact is wrong or, arguable, right.<\/span><\/i><\/p>\n<p><i><span style=\"font-weight: 400;\">So, you need to have an agreement with the client you are working with. Agree on what you will define as the success of a CRO program. If you are just measuring it on revenue, then you need to make sure that everything that relates to making money for that business is also controlled by the optimizers.<\/span><\/i><span style=\"font-weight: 400;\">&#8221;\u00a0<\/span><\/p><\/blockquote>\n<h2><strong>6. Do performance-based CRO projects work?<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">A performance-based CRO project is a plan whereby payment is granted to an optimizer after achieving a specific outcome or results. Some CRO consultants and agencies prefer having such contracts. But do they really work?<\/span><\/p>\n<p><span style=\"font-weight: 400;\">This is what Georgi had to say:\u00a0<\/span><\/p>\n<blockquote><p><b>\u201c<\/b><i><span style=\"font-weight: 400;\">Not any more than performance-based contracts in any industry other than sales and simple manufacturing tasks where the performance relies entirely upon the person executing the tasks. A performance bonus may have benefits, but otherwise, I think it is a bad deal for both parties. Such a structure makes it easy to claim credit and receive rewards without merit and suffer without reason. The incentives are never properly aligned in such a case<\/span><\/i><span style=\"font-weight: 400;\">.\u201d<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Chad also doesn\u2019t like the idea of performance-based CRO contracts as he believes that this concept might inhibit optimizers from testing to learn:\u00a0\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">\u201c<\/span><i><span style=\"font-weight: 400;\">No. This kind of model limits people from going for the learning. It might sound like a good idea to the client, but that will prompt optimizers to always go for the lowest denominator over and over again and not try new things. It also doesn\u2019t encourage risk-taking. If you incentivize risk-taking and failure, your optimization team is going to try innovative ideas.<\/span><\/i><span style=\"font-weight: 400;\">\u201d\u00a0<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Such projects are risky, according to Andr\u00e9.<\/span> <span style=\"font-weight: 400;\">He also believes that performance-based projects can disadvantage everyone involved in a project:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">\u201c<\/span><i><span style=\"font-weight: 400;\">That is dangerous because you will be risking a healthy relationship with your clients. You can have higher maturity and understanding of the underlying statistics, but your clients may not have that. Clients always have constraints, and you may not understand their limitations. Basically, it becomes a disadvantage on both sides. These kinds of contracts require a lot of trust and confidence on both sides to make it work. If you consider performance-based contracts, you should think about how you can always achieve great results<\/span><\/i><span style=\"font-weight: 400;\">.\u201d<\/span><\/p><\/blockquote>\n<h2><strong>7. Can you do testing on a site with low traffic\/conversions?<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">Conversion optimization can be done on any website. It doesn&#8217;t matter if the traffic or conversions of that particular site are high or low. But when it comes to A\/B testing, it can be tricky. Can you do it on a low traffic\/conversion site, or you can&#8217;t?\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Georgi believes that you can also test on any site. But he also says that &#8220;<\/span><i><span style=\"font-weight: 400;\">if low traffic and conversions are also accompanied by low revenue, then the relative returns on such efforts are significantly less attractive compared to when testing is applied to a larger project<\/span><\/i><span style=\"font-weight: 400;\">.&#8221; He also adds:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">Obviously, the statistical design of the tests would be different in a project with low traffic to make testing make business sense at all. Even with an optimal design, though, a smaller project will benefit less than a larger one, all else being equal. I&#8217;ve given an example of how the math works out in part on <\/span><\/i><a href=\"https:\/\/blog.analytics-toolkit.com\/2019\/a-b-testing-with-a-small-sample-size\/\"><i><span style=\"font-weight: 400;\">Economics of testing: small sample size vs. large sample size<\/span><\/i><\/a><i><span style=\"font-weight: 400;\"> in which the bigger business gets a 40% better bang for their buck simply due to their larger size.&#8221;<\/span><\/i><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">According to Chad, &#8220;<\/span><i><span style=\"font-weight: 400;\">The bigger the sample size, the easier it is to run an experiment and get a result you can act on.<\/span><\/i><span style=\"font-weight: 400;\">&#8221; However, Chad also says that testing can be done on any website:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">If you don&#8217;t have a big sample size, it doesn&#8217;t mean you can&#8217;t do experimentation. Many people forget that experiments happen all the time using tiny samples in the academic world. Psychology and health studies usually have a low number of samples. But the problem is that they usually last a long time<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">On the other hand, Lorenzo says that instead of testing, there many different alternatives that you can do on a low traffic or conversions site:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">If a site has about 20 or 50 conversions, you can&#8217;t launch an A\/B test and use conversions as the primary KPI. But with such websites, the best thing you can do is choose a <\/span><\/i><a href=\"https:\/\/www.invespcro.com\/blog\/optimizing-conversion-funnels-where-should-you-start-first\/\"><i><span style=\"font-weight: 400;\">top-of-the-funnel<\/span><\/i><\/a><i><span style=\"font-weight: 400;\"> KPI (like Add to Cart or visits to the pricing page) and focus on increasing those KPIs.\u00a0<\/span><\/i><\/p>\n<p><i><span style=\"font-weight: 400;\">The other thing you can do is gather insights from research, make the changes on your site and compare the before-and-after metrics. This approach might not be as accurate as testing, but it is worth trying. But when you make these changes, make significant changes, not just button colors, you can even go beyond layout design changes<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;\u00a0<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Andr\u00e9 believes that &#8220;<\/span><i><span style=\"font-weight: 400;\">It depends on the amount of validity you want<\/span><\/i><span style=\"font-weight: 400;\">.&#8221; He also goes on to say that &#8220;<\/span><i><span style=\"font-weight: 400;\">when there are no conversions, then that means there&#8217;s no business, and that makes it difficult for optimizers. I mean, what&#8217;s there to optimize?<\/span><\/i><span style=\"font-weight: 400;\">&#8221; <\/span><\/p>\n<p><span style=\"font-weight: 400;\">Just like Lorenzo, Andr\u00e9 also shares some ideas of what to do instead of testing:\u00a0\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">So, if I&#8217;m an early-stage startup with no traffic or micro-conversions, I won&#8217;t do A\/B testing. I would seek for other ways like qualitative research, quantitative research with online panels, and remote testing to validate my ideas<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;<\/span><\/p><\/blockquote>\n<h2><strong>8. How long does it take to see results from CRO?<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">Many components are needed to launch a successful CRO program. Suppose you now have <\/span><a href=\"https:\/\/www.invespcro.com\/blog\/what-does-it-take-to-successfully-execute-a-cro-program\/\"><span style=\"font-weight: 400;\">what it takes to successfully execute a CRO program<\/span><\/a><span style=\"font-weight: 400;\">. But you are still wondering how long it will take to see the results. Will it take a month? Two months? Or maybe more than 3 months?\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Well, Georgi says that &#8220;<\/span><i><span style=\"font-weight: 400;\">if the prioritization is right and the current CRO efforts are the first for the project, then results should arrive almost immediately<\/span><\/i><span style=\"font-weight: 400;\">.&#8221; He also adds:\u00a0<\/span><\/p>\n<blockquote><p><b>&#8220;<\/b><i><span style=\"font-weight: 400;\">Capturing the <\/span><\/i><a href=\"https:\/\/www.invespcro.com\/blog\/cro-low-hanging-fruits-that-will-give-you-an-immediate-win\/\"><i><span style=\"font-weight: 400;\">low-hanging fruit<\/span><\/i><\/a><i><span style=\"font-weight: 400;\"> needs little more than the time to run the first batch of online experiments, which is usually from a week to a month for these types of tests. If a user experience has already been through dozens of rounds of improvements, then it would be much harder to get substantial results on top of that as the marginal value of each additional attempt becomes lower<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Chad says that it may take longer than three months before you begin to see the results. But he also adds that the benefits of CRO go beyond the numbers:\u00a0\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">With CRO, the results are not going to be immediately apparent in a quarter, and sometimes even in six months. One of the biggest value-adds of an optimization program isn&#8217;t even the results. It&#8217;s the culture that it creates. Conversion optimization establishes a culture of accountability, where people are responsible for the metrics they believe they can change. And it also creates a culture of science where people are thinking in terms of hypotheses and outcomes.\u00a0<\/span><\/i><\/p>\n<p><i><span style=\"font-weight: 400;\">When the entire business adopts this culture of science, it&#8217;s going to enable people to be moving in a much more measured way; a much more robust way; and holding each other&#8217;s standards of accountability<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;\u00a0<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">According to Lorenzo, the number and quality of experiments you launch will determine when you will see the results:\u00a0 <\/span><b>\u00a0<\/b><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">It depends on how you run your experimentation process. But it all comes down to the number of experiments you&#8217;re running and the quality of those experiments. By experiment quality, I mean wherein the funnel are you running those experiments. Are you running experiments on pages that are more likely to have a bigger impact in terms of revenue or elsewhere? Or on pages with low revenue impact?<\/span><\/i><span style=\"font-weight: 400;\">&#8220;<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Andr\u00e9 agrees with Chad that CRO is not about the results only. But he also says that results may start to show up after six months:\u00a0<\/span><\/p>\n<blockquote><p><i><span style=\"font-weight: 400;\">&#8220;If you take into account that a great CRO project is not only about delivering results but changing an organization, enabling people, making them understand the importance of being customer-centric and data-driven, I would say at least six months. If you ask me how long does it take to set up an experiment and get results, I would say a couple of weeks, maybe four weeks<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;\u00a0<\/span><\/p><\/blockquote>\n<h2><strong>9. What type of benefit should I expect from a CRO project? Should I expect an increase in revenue or conversions? What %?<\/strong><\/h2>\n<p><span style=\"font-weight: 400;\">People that are getting into conversion optimization for the first time like asking this question. And because of the name CRO, most people tend to assume that it&#8217;s all about conversions. But is CRO only limited to conversions? If that&#8217;s the case, what percentage of conversions increase can one expect from a CRO project?\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Lorenzo believes that you can&#8217;t put an exact percentage to expect from a CRO project. He also adds that the rate is determined by the maturity of your CRO program:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">Well, percentage-wise, it depends on how mature your optimization program is. So, suppose you are optimizing a website like Booking.com (one of the top experimentation organizations in the world). In that case, it&#8217;s tough to make huge impacts because their site has been optimized for years by smart optimizers. But if you optimize a site that nobody has ever done experimentation before, you will more likely see bigger benefits<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;\u00a0<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">According to Georgi, a CRO project with proper target metrics should result &#8220;<\/span><i><span style=\"font-weight: 400;\">in an improvement in a metric as close to the bottom line as is possible to measure<\/span><\/i><span style=\"font-weight: 400;\">.&#8221; He also goes on to say: <\/span><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">Ideally that would be profit, but as this is rarely possible, average revenue per user would be the next best thing. Conversions per user come next in the list<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Considering how different businesses are, it&#8217;s impossible to predict the type of benefit you should expect from a CRO project, says Chad. But he also goes on to echo the same sentiments as Lorenzo on how the optimization program maturity can also be a determinant factor:<\/span><b>\u00a0<\/b><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;It&#8217;s<\/span><i><span style=\"font-weight: 400;\"> impossible to predict because it differs from one business to another. If you have a very mature business and it&#8217;s been around for a long time, there might be certain metrics that will be challenging to change. For instance, Google has been around for about 20 years, and they have optimized their search page repeatedly to the point where they change it anymore. So, if I&#8217;m going to try and increase the number of searches on Google&#8217;s homepage, I&#8217;m probably not going to make an impact at all.<\/span><\/i><span style=\"font-weight: 400;\">&#8220;<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Andr\u00e9 says that the benefits can also be influenced by the timeframe of your CRO program. But he also has an illustrative story about this question:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">Depends on the timeframe. But it&#8217;s perilous to imply a significant business impact. We once helped a client get a conversion uplift of nearly 40%. When they came to us again, I told them they can&#8217;t expect 40% more conversions because many factors impact the experiment.\u00a0<\/span><\/i><\/p>\n<p><i><span style=\"font-weight: 400;\">So this is a question of expectation management. And this is why it&#8217;s important to have workshops with clients and ask them what they expect. Sometimes the financial return might not be visible, or it&#8217;s not direct that you can attribute the accumulated uplift of experiments to your bank account.\u00a0<\/span><\/i><\/p>\n<p><i><span style=\"font-weight: 400;\">But on the other hand, you&#8217;ll get many different values that are not measurable, and you have to be aware of that. So, yeah, expectation management is king.<\/span><\/i><span style=\"font-weight: 400;\">&#8221;\u00a0<\/span><\/p><\/blockquote>\n<p><span style=\"font-weight: 400;\">Tim believes that the way you structure your business and CRO goals will decide the kind of benefit you can expect from your project:\u00a0<\/span><\/p>\n<blockquote><p><span style=\"font-weight: 400;\">&#8220;<\/span><i><span style=\"font-weight: 400;\">It depends on what your business needs. So, should you expect an improvement in conversion, revenue, or both? If that&#8217;s what you&#8217;ve designed your business strategy to do, if that&#8217;s what you&#8217;ve commissioned your team or the agency you&#8217;ve hired, then yes, you should expect your revenue and conversions to improve<\/span><\/i><span style=\"font-weight: 400;\">.&#8221;\u00a0<\/span><\/p><\/blockquote>\n<h2><strong><span data-preserver-spaces=\"true\">Conclusion<\/span><\/strong><\/h2>\n<p><span data-preserver-spaces=\"true\">Well, there you have it. The toughest CRO questions answered by the top pros themselves. Special thanks to every expert that contributed! If you want to keep on receiving such expert tips in your inbox, you can\u00a0<\/span><a class=\"editor-rtfLink\" href=\"https:\/\/www.invespcro.com\/newsletter-khalid\/\" target=\"_blank\" rel=\"noopener noreferrer\"><span data-preserver-spaces=\"true\">join our community<\/span><\/a><span data-preserver-spaces=\"true\">\u00a0of over 30k marketers.\u00a0<\/span><\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p><span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\"> 18<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span>Conversion rate optimization is always going to be complicated.\u00a0 It was complicated in its early days; it still is complicated today, and it will still be tricky tomorrow.\u00a0 This is why there\u2019s no one universal process to approach it. Different CRO experts use different methods when optimizing websites.\u00a0 But no matter how you approach conversion [&hellip;]<\/p>\n","protected":false},"author":54,"featured_media":14992,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[36],"tags":[],"class_list":["post-14915","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cro"],"_links":{"self":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts\/14915","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/users\/54"}],"replies":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/comments?post=14915"}],"version-history":[{"count":0,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts\/14915\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/media\/14992"}],"wp:attachment":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/media?parent=14915"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/categories?post=14915"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/tags?post=14915"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}