{"id":96098,"date":"2023-02-08T05:11:31","date_gmt":"2023-02-08T10:11:31","guid":{"rendered":"https:\/\/www.invespcro.com\/blog\/?p=96098"},"modified":"2023-02-08T05:11:31","modified_gmt":"2023-02-08T10:11:31","slug":"how-to-measure-the-revenue-impact-of-experimentation","status":"publish","type":"post","link":"https:\/\/www.invespcro.com\/blog\/how-to-measure-the-revenue-impact-of-experimentation\/","title":{"rendered":"How to measure the revenue impact of experimentation\u00a0"},"content":{"rendered":"<span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\"> 9<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span><p><span data-preserver-spaces=\"true\">Evaluating the revenue impact of your experimentation program is crucial for making informed business decisions.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">But the truth is many businesses don&#8217;t know how to forecast revenue or do revenue attribution for their testing program.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">You can measure the success of your experimentation program by measuring its total revenue impact over time. This approach helps you identify the most successful experiments, which could lead to more investment in similar opportunities.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">You may also want to look at how much revenue multiple experiments contributed directly (versus other factors like cost savings). This can help you decide whether or not to invest more in a particular area where you&#8217;ve seen some success \u2013 or if it might be better spent elsewhere.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Measuring the revenue impact of your testing program is one of the most important things you can do to make data-driven decisions.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">After all &#8211; you wouldn&#8217;t want to launch a slow-selling product on Amazon or your own ecommerce site just because it was optimized for search rankings&#8230;right? Unfortunately, this is all too common among sellers.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">But don&#8217;t worry! Here&#8217;s what you should know about measuring the revenue impact of experimentation:\u00a0<\/span><\/p>\n<h2><span data-preserver-spaces=\"true\">Why is Measuring the Revenue Impact of Experimentation Important\u00a0<\/span><\/h2>\n<p><span data-preserver-spaces=\"true\">The revenue impact of experimentation is an important metric to track. And it&#8217;s not a secondary metric either; it&#8217;s one of the primary determining factors. It can help you understand how much money you&#8217;re making from experiments and what&#8217;s working.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">If you don&#8217;t measure the revenue impact of experimentation, you won&#8217;t know if your efforts are paying off. Without this information, you&#8217;ll be flying blind when it comes to improving your business.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">For example, if a marketing campaign increased sales but decreased profits, then a measurement of just sales would show that it was a successful initiative. However, if we also measured earnings before and after the campaign took place (and reduced them), we would know that this campaign wasn&#8217;t worthwhile.<\/span><\/p>\n<p><strong><span data-preserver-spaces=\"true\">The revenue impact of experimentation helps businesses:<\/span><\/strong><\/p>\n<ul>\n<li><span data-preserver-spaces=\"true\">Understand which experiments are working.<\/span><\/li>\n<li><span data-preserver-spaces=\"true\">Determine which users are converting on their first visit or purchase.<\/span><\/li>\n<li><span data-preserver-spaces=\"true\">Identify which pages or features are driving the most revenue.<\/span><\/li>\n<\/ul>\n<p><span data-preserver-spaces=\"true\">The bottom line is: If you&#8217;re not tracking experiment revenue, then you won&#8217;t know if they&#8217;re worth continuing.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">This is why measuring the revenue impact of experimentation is so important. If you know how much revenue an experiment generates, then it can be easier to justify investing more time and resources into it.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Of course, measuring experiment revenue isn&#8217;t always easy \u2014 especially when multiple tests are running. But by taking some simple steps and using a few tools at your disposal, you&#8217;ll be able to track experiment performance quickly and easily.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">The good news is that you don&#8217;t have to be an expert in statistics or data analytics to measure the revenue impact of your experiments. You just need to know what data to collect and how to interpret it.<\/span><\/p>\n<h2><span data-preserver-spaces=\"true\">Ways to measure the revenue impact of experimentation<\/span><\/h2>\n<h3><strong>1. Set a conversion goal for your experiment.<\/strong><\/h3>\n<p><span data-preserver-spaces=\"true\">Experimenting with your website is a great way to improve conversion rates. But how do you differentiate a winning experiment from a losing one?\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">The answer lies in setting a conversion goal for each experiment. This is an easy but often overlooked step.<\/span><\/p>\n<p><strong><span data-preserver-spaces=\"true\">What is a conversion goal?<\/span><\/strong><\/p>\n<p><span data-preserver-spaces=\"true\">A conversion goal is an action that leads to a desired result. In marketing terms, this might be a sale or lead generated from an online ad campaign. In e-commerce, it might be a purchase made on your site.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">For example, let&#8217;s say you&#8217;re running an ad campaign for your new product line and want to increase sales of the product by 10% over the next month. You could set up an A\/B test to test two different landing pages and optimize for whichever performs better when it comes to achieving this goal (more sales and revenue uplift).<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Or, if you&#8217;re testing a landing page design, you might set a conversion goal of 5% more sales from the new version over the old one.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">For example, if you make $1 million per month and have 1 billion impressions per month across all pages on your website, then 5% more conversions would give you $50k in additional revenue per month \u2014 which is significant!<\/span><\/p>\n<p><strong><span data-preserver-spaces=\"true\">Pro Tip:<\/span><\/strong><span data-preserver-spaces=\"true\">\u00a0Once you have your outline, create goals using a tool like\u00a0<\/span><a class=\"editor-rtfLink\" href=\"https:\/\/taplytics.com\/\" target=\"_blank\" rel=\"noopener\"><span data-preserver-spaces=\"true\">Taplytics<\/span><\/a><span data-preserver-spaces=\"true\">. It will let you take complete control of your releases. You can also instantly test and roll out new features with its advanced feature management and experimentation features.<\/span><\/p>\n<h3><strong>2. Measure direct revenue<\/strong><\/h3>\n<p><span data-preserver-spaces=\"true\">Direct revenue includes revenue from sales, fees, and charges directly associated with a product or service. Indirect revenue includes all other sources of revenue that are not directly associated with any specific product or service.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">The reason direct revenue is so important when measuring the impact of experimentation is that it gives you a sense of how much money your business would have made if you had not experimented. In other words, it lets you know how much incremental value you created by testing one thing versus another.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">It also gives you a baseline for comparison, which is important because it allows you to determine whether or not your experiment worked.<\/span><\/p>\n<p><em><span data-preserver-spaces=\"true\">For example, let&#8217;s say that you run an online store selling T-shirts.\u00a0<\/span><\/em><\/p>\n<p><span data-preserver-spaces=\"true\">You run an experiment that changes the price of your most popular design from $15 per shirt to $17 per shirt. You track the number of shirts sold each week before and after the experiment, along with how much money each sale generated.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">At the end of two months, you notice that sales have increased by 10% \u2013 but how do you know that your experiment caused this increase? You can compare it against other factors like seasonality (maybe summer is busier than winter) or other external factors (maybe there was an ad campaign in that period).<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">If you don&#8217;t account for those external factors, then it&#8217;s impossible to know whether your experiment worked or not!<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">You can further break it down to revenue per visitor. RPV can be calculated by dividing the total revenue you gained during a fixed time period by the total number of visitors during the same period.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">For\u00a0<\/span><a class=\"editor-rtfLink\" href=\"https:\/\/www.optimizely.com\/optimization-glossary\/revenue-per-visitor\/#:~:text=What%2520is%2520revenue%2520per%2520visitor,value%2520of%2520each%2520additional%2520visitor.\" target=\"_blank\" rel=\"noopener\"><span data-preserver-spaces=\"true\">instance<\/span><\/a><span data-preserver-spaces=\"true\">, if you have earned $10,000 in revenue in the month of December and your site gets 2,000 visitors, your revenue per visitor would be $10,000\/2,000 or $5 per visitor.<\/span><\/p>\n<h3><strong>3. Measure the conversion rate.<\/strong><\/h3>\n<p><span data-preserver-spaces=\"true\">Conversion rate is a measure of how often visitors to your site take the action you want them to. For example, if your site&#8217;s goal is to sell something, your conversion rate would be the percentage of visitors who buy something.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Or for example, if your website has 100 visitors and 10 of them make a purchase, you have a 10 percent conversion rate.<\/span><\/p>\n<p><strong><span data-preserver-spaces=\"true\">Conversion rates are important for two reasons:<\/span><\/strong><\/p>\n<p><span data-preserver-spaces=\"true\">They help you understand the effectiveness of your marketing efforts, enabling you to create a more effective sales funnel.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">They tell you whether or not you should be running an experiment on a particular website page or element (such as a call-to-action button). If a change increases your conversion rate but doesn&#8217;t affect revenue, then it&#8217;s not worth making that change.<\/span><\/p>\n<p><strong><span data-preserver-spaces=\"true\">There are many ways to measure your conversion rate:<\/span><\/strong><\/p>\n<p><span data-preserver-spaces=\"true\">For example, if you want to <a href=\"https:\/\/www.invespcro.com\/blog\/click-tracking\/\">measure<\/a> how users interact with a mobile app or website, use Google Analytics (GA) event tracking to track when users perform an action that matters to your business (such as completing a purchase).<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">You can also measure the revenue impact of experimentation by\u00a0<\/span><a class=\"editor-rtfLink\" href=\"https:\/\/marketingsyrup.com\/set-up-event-goals-in-google-analytics\/\" target=\"_blank\" rel=\"noopener\"><span data-preserver-spaces=\"true\">setting up goals in GA<\/span><\/a><span data-preserver-spaces=\"true\">\u00a0and tracking them as they occur across experiments.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">You can also divide the number of conversions by the number of visitors and multiply that number by 100 to get a percentage.<\/span><\/p>\n<div class=\"blog_img\"><img fetchpriority=\"high\" decoding=\"async\" class=\"alignnone wp-image-96099 size-full\" src=\"https:\/\/www.invespcro.com\/blog\/images\/blog-images\/Conversion-rate-calculation.png\" alt=\"\" width=\"512\" height=\"299\" data-wp-pid=\"96099\" \/><\/div>\n<p><span data-preserver-spaces=\"true\">You can use tools like AdWords, Heatmaps, and\u00a0<\/span><a class=\"editor-rtfLink\" href=\"https:\/\/www.invespcro.com\/blog\/using-session-replay-videos-to-identify-conversion-problems-on-a-website\/\" target=\"_blank\" rel=\"noopener\"><span data-preserver-spaces=\"true\">Session replay tools<\/span><\/a><span data-preserver-spaces=\"true\">\u00a0to track conversion rates.<\/span><\/p>\n<h3><strong>4. Determine customer lifetime value.\u00a0<\/strong><\/h3>\n<p><span data-preserver-spaces=\"true\">When you&#8217;re experimenting, you need to measure the impact on the bottom line. That&#8217;s why the customer lifetime value (CLV) metric is so important.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">CLV represents the revenue that a customer will generate over their lifetime. It&#8217;s an estimate of how much a company can expect to make from a customer over time.\u00a0<\/span><\/p>\n<p><strong><span data-preserver-spaces=\"true\">The formula for calculating CLV is simple:<\/span><\/strong><\/p>\n<p><span data-preserver-spaces=\"true\">Customer lifetime value = average order value x average number of purchases per year x average customer lifespan in years.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">You can further use cohort analysis to attribute the impact the implemented changes had on your CLV and retention.<\/span><\/p>\n<div class=\"blog_img\"><img decoding=\"async\" class=\"alignnone wp-image-96100 size-full\" src=\"https:\/\/www.invespcro.com\/blog\/images\/blog-images\/Customer-Lifetime-value.png\" alt=\"\" width=\"512\" height=\"209\" data-wp-pid=\"96100\" \/><\/div>\n<p><span data-preserver-spaces=\"true\">(<\/span><a class=\"editor-rtfLink\" href=\"https:\/\/cxl.com\/blog\/using-cohort-analysis-for-conversion-optimization\/\" target=\"_blank\" rel=\"noopener\"><span data-preserver-spaces=\"true\">Source<\/span><\/a><span data-preserver-spaces=\"true\">)<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">For the uninitiated, cohort analysis is a statistical method used to study groups of people who share common traits. In the case of SaaS companies, a cohort is a group of customers who signed up at a similar point in time and have been tracked over time.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">You&#8217;ll also want to take a look at cohorts at different points in time to see how they&#8217;re performing relative to each other.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">For example, if you want to see how different cohorts are performing after six months, you might break down your customers into three groups: those who signed up six months ago, those who signed up one year ago, and those who signed up two years ago.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">The goal is that each group should have similar characteristics &#8212; say all from the same country or industry &#8212; so that any differences in performance can be attributed specifically to the difference in time since they joined the service rather than anything else about them.<\/span><\/p>\n<p><strong><span data-preserver-spaces=\"true\">Here&#8217;s how it works in practice:\u00a0<\/span><\/strong><\/p>\n<p><span data-preserver-spaces=\"true\">Cohort analysis compares two groups of users based on when they first used your product or service. The first cohort was acquired before you implemented your experiment, while the second cohort was acquired after you implemented your experiment.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">For example:<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">The first cohort might be all users who signed up for a new account between January and June, while the second cohort would be all users who signed up between July and December.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">You can then compare both cohorts&#8217; retention rates over time to see if there was an impact from your experiment.<\/span><\/p>\n<h3><strong>5. Leverage Net Promoter Score (NPS)<\/strong><\/h3>\n<p><span data-preserver-spaces=\"true\">Net promoter score (NPS) provides valuable insights into user behaviors by determining how likely customers are to recommend your company to others.\u00a0<\/span><\/p>\n<p><strong><span data-preserver-spaces=\"true\">It&#8217;s based on the principle that customers respond to one of three reasons and a response scale:<\/span><\/strong><\/p>\n<ul>\n<li><strong><span data-preserver-spaces=\"true\">Promoters (score of 9 or 10):<\/span><\/strong><span data-preserver-spaces=\"true\">\u00a0It means that customers are extremely likely to recommend your business to others. These people are your best advocates, and they are the ones who will promote you on social media and other channels.\u00a0<\/span><\/li>\n<\/ul>\n<ul>\n<li><strong><span data-preserver-spaces=\"true\">Passives (score 7-8):<\/span><\/strong><span data-preserver-spaces=\"true\">\u00a0those who don&#8217;t hate your product but don&#8217;t love it either. They will stay with you if there are no other options available to them.<\/span><\/li>\n<\/ul>\n<ul>\n<li><strong><span data-preserver-spaces=\"true\">Detractors (0-6):\u00a0<\/span><\/strong><span data-preserver-spaces=\"true\">It means they&#8217;re unlikely to recommend it \u2014 or even actively discourage others from buying or using it.<\/span><\/li>\n<\/ul>\n<div class=\"blog_img\"><img decoding=\"async\" class=\"alignnone wp-image-96101 size-full\" src=\"https:\/\/www.invespcro.com\/blog\/images\/blog-images\/Net-Promoter-Score-2.png\" alt=\"\" width=\"512\" height=\"179\" data-wp-pid=\"96101\" \/><\/div>\n<p><span data-preserver-spaces=\"true\">(<\/span><a class=\"editor-rtfLink\" href=\"https:\/\/www.hotjar.com\/blog\/nps-case-study\/\" target=\"_blank\" rel=\"noopener\"><span data-preserver-spaces=\"true\">Source<\/span><\/a><span data-preserver-spaces=\"true\">)<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">In other words, NPS is a great way to gauge how satisfied customers are with your product or service. It can also reveal how well you&#8217;re doing at improving their satisfaction over time and whether you&#8217;re delivering on your promises.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">The more promoters you have, the better your NPS will be. But here&#8217;s where it gets interesting: If you can figure out how much more money promoters generate than detractors, then you can use this information as a proxy for how much revenue your business might generate from experimentation.<\/span><\/p>\n<p><strong><span data-preserver-spaces=\"true\">The NPS can help you measure the revenue impact of experimentation in several ways:<\/span><\/strong><\/p>\n<ul>\n<li><span data-preserver-spaces=\"true\">To quantify the value and direct impact of new features or products you&#8217;re going to launch. If you have an existing product and want to know if a new feature will make it more attractive, you can use NPS to quantify the potential revenue increase or decrease from adding or removing features.<\/span><\/li>\n<\/ul>\n<ul>\n<li><span data-preserver-spaces=\"true\">To measure churn rates before and after experiments. You can compare churn rates for customers who were exposed to a certain experiment with those who weren&#8217;t exposed, then multiply that difference by the revenue impact of each experimenter group to get an estimate of its effect on your bottom line.<\/span><\/li>\n<li><\/li>\n<li><span data-preserver-spaces=\"true\">To measure user behavior, like customer satisfaction with new features or products already launched into production. You can compare customer satisfaction for people who were exposed to a certain feature versus those who weren&#8217;t exposed.\u00a0<\/span><\/li>\n<\/ul>\n<h2><span data-preserver-spaces=\"true\">The Bottom Line!<\/span><\/h2>\n<p><span data-preserver-spaces=\"true\">For marketers, experiments are a key component to success. Without them, developing new and innovative ideas for customers would be impossible, but measuring the effects of your decisions is also essential.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">Revenue gain is one of the key metrics for any business because it directly affects the bottom line. You can&#8217;t measure success without knowing where the money comes from.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">With that said, there is no single success metric (both primary and secondary metrics) for measuring revenue impact. It depends on what stage your company is in and what type of product or service you offer.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">For example, a large ecommerce company may use CPA as one of its success metrics because they have so much traffic and data available to them. On the other hand, a small startup might measure revenue as its primary metric because they have no other options available at this stage of its business development.<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">The type of product or service you sell will also play a role in determining which metrics you should focus on when measuring revenue impact. For example, if you sell software as a service (SaaS), it&#8217;s important to track customer churn rates because these customers generate recurring revenue over time.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">When you run multiple tests or are constantly running experiments, the last thing you want to do is lose track of the revenue effects and mess up your numbers.\u00a0<\/span><\/p>\n<p><span data-preserver-spaces=\"true\">With this article, we hope you can have an easier time measuring and identifying the revenue impact of your experimentation process.<\/span><\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p><span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\">Reading Time: <\/span> <span class=\"rt-time\"> 9<\/span> <span class=\"rt-label rt-postfix\">minutes<\/span><\/span>Evaluating the revenue impact of your experimentation program is crucial for making informed business decisions. But the truth is many businesses don&#8217;t know how to forecast revenue or do revenue attribution for their testing program. You can measure the success of your experimentation program by measuring its total revenue impact over time. This approach helps [&hellip;]<\/p>\n","protected":false},"author":50,"featured_media":96475,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[36],"tags":[],"class_list":["post-96098","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cro"],"_links":{"self":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts\/96098","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/users\/50"}],"replies":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/comments?post=96098"}],"version-history":[{"count":0,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/posts\/96098\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/media\/96475"}],"wp:attachment":[{"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/media?parent=96098"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/categories?post=96098"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.invespcro.com\/blog\/wp-json\/wp\/v2\/tags?post=96098"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}