Imagine a chessboard where each move reveals an opportunity or threat. Without studying your opponent’s strategy, you are likely to get outmaneuvered. The same applies in business—competitive analysis is your playbook for success.

By assessing competitors’ strengths, weaknesses, and strategies, you can uncover opportunities, refine your plans, and stay ahead of industry trends.

This article breaks down the essentials of competitive analysis with real-world examples and actionable steps to help you outsmart the competition.

What Exactly is Competitive Analysis?

Competitive analysis involves analyzing your competitors’ successes and losses to identify their strengths, weaknesses, strategies, and market positioning. This helps you identify opportunities, mitigate risks, and make informed decisions to gain a competitive edge. 

Think of it as doing homework before playing a high-stakes game—you can’t win if you don’t know your opponent’s moves.

Why is Competitive Analysis so Important?

Imagine running a race without knowing who else is competing, their speed, or their strategies. That’s what it’s like for a business without competitive analysis.

In the early 2010s, Nokia dominated the mobile phone market. Yet, they failed to recognize the shift toward smartphones driven by Apple and Android. Without a solid competitive analysis of these emerging players, Nokia stuck to outdated strategies and missed the touch-screen revolution. The result? A dramatic fall from being a market leader to an industry afterthought.

Here’s a quick overview of why competitive analysis is so integral to a business:

• Understand Market Gaps: Identify untapped opportunities your competitors are missing.
• Refine Your Strategy: Adjust your pricing, marketing, or customer experience based on competitors’ strengths and weaknesses.
• Stay Relevant: Predict industry trends and customer preferences before others do.

How to Conduct a Competitive Analysis

Define goals and objectives

Defining your goals and objectives is the first and most critical step in conducting a competitive analysis. Think of it as creating a map before you start a journey—you need to know where you’re going and why you’re going there.

Without clear goals, your analysis risks becoming a pile of data without direction or purpose.

Here are some types of goals for competitive analysis:

• Understand market position: Where do you stand compared to competitors? What gaps or opportunities exist in your industry?
• Improve customer experience: What do competitors do better? How can you adapt their strengths to meet your customers’ needs?
• Refine marketing strategy: What messaging resonates with your audience? How can you differentiate your branding?
• Identify innovation opportunities: What new features or trends are gaining traction? How can your business stay ahead?

Now that you know the goals you can set for your company, how will you define your goals? Here’s a step-by-step approach to help you determine your goals effectively: 
Start with your business objectives. Every competitive analysis should tie back to your company’s larger goals. For example, if your company wants to expand into a new market, your analysis should focus on local competitors and customer behavior. For example, when Starbucks expanded into China, it aimed to understand local tea culture and how competitors catered to it. The competitive analysis helped them integrate tea-based beverages into their menu, establishing a strong presence in a tea-dominated market.

• Identify your target audience. Define whose needs you want to meet. Your goals might focus on understanding why competitors attract a specific customer segment. In fact, a study by Epsilon revealed that 80% of customers are more likely to engage with brands that offer a personalized experience. And knowing your competitors’ audience can help you deliver personalization better.

• Ask key questions. Frame questions in a way that answers shape actionable goals.
  • “What do customers love about my competitors?”
  • “How do competitors use pricing, promotions, or products to win?”

• Use data-driven insights. Back your goals with data. You can use tools like Google Trends, SEMrush, and social media analytics to get insights into what works for competitors.

Pro tip: Use the SMART framework for goal-setting (Specific, Measurable, Achievable, Relevant, Time-bound). For example, “Increase market share by 10% within six months by targeting Gen Z customers through Instagram ads.”

Gather competitor data, including strengths and weaknesses

When conducting a competitive analysis, the next important step is to gather accurate, actionable data about your competitors. This data helps you identify their strengths, weaknesses, and ways you can differentiate yourself from them.

Here’s how to do it effectively:

• Start by defining what data you want to collect. Start by identifying the key aspects of your competitors you want to analyze. These typically include:
  • Product or service offerings: Features, quality, and pricing
  • Market positioning: Where they stand in the industry
  • Marketing and branding: Channels, tone, and customer reach
  • Customer experience: Support, reviews, and engagement

• Use reliable tools for data collection. To gather valuable and actionable competitor data, you’ll need the right tools to save time and provide deeper insights. These tools analyze everything from website traffic to customer engagement, clearly showing how your competitors operate and where they excel or fall short.

• SEMrush or Ahrefs: Analyze competitors’ SEO performance, keywords, and backlinks.
• SimilarWeb: Understand their website traffic, audience demographics, and referral sources.
• Sprout Social or Brandwatch: Monitor their social media strategies and audience engagement.
• Glassdoor or Indeed: Learn about internal strengths and weaknesses through employee reviews.

• Gather qualitative data. In addition to using tools, you need qualitative methods to gain deeper insights into what competitors are doing and how customers feel about them. These methods focus on understanding the why behind the data and provide actionable details that numbers alone can’t offer. Here’s how you can do it:
  • Competitor websites and reports: Look for annual reports, press releases, and blog content. Use these reports to determine their pricing details, product features, and unique selling points. Their blog content, whitepapers, and press releases will help you understand their brand’s messaging and priorities.
  • Customer reviews and forums: Use Yelp or Reddit to uncover customer sentiments about competitors.
  • Mystery shopping: It involves experiencing the products or services firsthand.

• Apply a framework to analyze strengths and weaknesses. To break down data, use a framework like SWOT (Strengths, Weaknesses, Opportunities, Threats) framework to break down data.
  • Strengths: Look for their competitive edges, like superior technology or loyal customer bases.
  • Weaknesses: Identify gaps such as high prices or limited product diversity.

Companies that fail to identify their weaknesses and those of their competitors often lose the market to a new contender. One of the most prominent examples is when Kodak failed to recognize its weakness—reluctance to embrace digital photography—and lost its competitive edge to companies like Sony and Canon.

• Combine insights from multiple sources. Cross-reference findings from tools and methods to build a holistic picture. For example:
  • If SEMrush shows high traffic for a competitor’s blog, use social media tools to check how they promote it.
  • If customer reviews indicate poor service, check Glassdoor for potential internal issues.

Pro tip: Competitors evolve constantly, so don’t forget to update your data regularly to stay informed—for instance, track product launches, partnerships, or rebranding efforts. You can also set up Google Alerts or subscribe to competitor newsletters to keep yourself updated.

Evaluate key metrics

The next step is to evaluate key metrics—which involves analyzing measurable data points that reveal your competitors’ strengths, weaknesses, and market performance.

Tangible metrics will show you where your business stands compared to others and what to improve or focus on to succeed in the market.

Start by understanding the key metrics to evaluate:

To get a complete picture, focus on these critical areas:

• Market share and reach: How much of the market do competitors control?
• Customer sentiment and engagement: What do customers say about their products and services?
• Product or service quality: Are their offerings superior or lagging?
• Pricing strategies: Are they underpricing, premium pricing, or bundling effectively?
• Operational efficiency: How quickly do they deliver, and what’s their supply chain like?

For example, Tesla disrupted the automobile market by evaluating its competitors’ reliance on internal combustion engines. Focusing on electric vehicles and assessing gaps in innovation, the company managed to carve out a dominant niche.

Leverage metrics to measure marketing effectiveness:

Use marketing metrics to see how competitors connect with their audience.

Here’s a list of metrics and elements to consider:

• Website traffic and SEO performance: Use tools like Google Analytics and SEMrush to reveal traffic volume, top-performing keywords, and backlinks.
• Social media engagement: Leverage tools like Sprout Social to determine likes, shares, comments, and audience demographics.
• Content marketing impact: Look at the frequency of blogs and videos and how they resonate with the audience.

Evaluate financial metrics:

Assessing competitors’ financial performance is essential to see whether their efforts are paying off. Financial metrics will give you a clear picture of their stability and growth.

Key metrics include:

• Revenue growth: Indicates market demand and business health.
• Profit margins: Reflects operational efficiency.
• Investment trends: Look at R&D spending or acquisitions to gauge innovation.

Track industry-specific metrics:

Depending on your industry, metrics may vary. Examples include:

• Churn rate: It’s especially critical in SaaS to measure how many customers cancel their subscriptions.
• Foot traffic: Important for retail and hospitality businesses.
• App downloads and ratings: Crucial for tech and mobile-first businesses.

For instance, Netflix tracks user churn to understand when and why subscribers leave. It uses this insight to adjust pricing, add new features, and prioritize trending content, keeping churn at competitive levels.

Finally, use collected data to compare competitors’ metrics against your own performance.

You can do this by identifying areas where you excel, and double down on those strengths. Secondly, spot gaps where competitors are ahead, and develop strategies to close them.

Create actionable strategies

Once you’ve gathered data and evaluated key metrics, the next step in a competitive analysis is to use your uncovered insights than turn them into actionable strategies.

This ensures your findings drive tangible changes to improve your position in the market and get an edge over the competition.

Your first step is to align findings with your business objectives. Ask yourself:

• What gaps can we exploit in the market?
• Where can we differentiate our offerings?
• How can we strengthen our weaknesses against competitors?

Once you have an answer to these questions, it’s time to develop strategies to get a competitive edge.

Your strategies can fall into two categories:

Offensive Strategies: Playing to Your Strengths

Offensive strategies are about staying ahead of your competitors by using your strengths or targeting areas they’re missing. They involve focusing on:

• Building on your strengths: Use what you do best to create an advantage.
• Filling market gaps: Look for unmet customer needs or underserved groups and step in.
• Innovating boldly: Introduce unique products or features competitors don’t offer.

For example, Airbnb expanded globally by addressing gaps in affordable and unique accommodations. Instead of competing directly with hotels, it targeted travelers seeking home-like stays or unconventional options, like treehouses or houseboats.

Defensive Strategies: Protecting What’s Yours

Defensive strategies are about holding onto what you already have—your customers, market share, and position. The goal is to address any weaknesses and prevent competitors from taking over your space.

It involves focusing on:

• Improving customer experience: Look at where competitors excel (e.g., faster delivery, better support) and close the gap.
• Matching competitive pricing and promotions: Keep an eye on competitor pricing—if they lower prices or run promotions, respond quickly. Offer added value like loyalty rewards or free shipping to stay ahead.
• Strengthening brand loyalty: Reward loyal customers with exclusive perks, early access, or personalized offers. Create emotional connections through strong branding and consistent messaging.
• Enhancing product and service quality: Focus on fixing weaknesses competitors could exploit. Improve durability, ease of use, or features in your offerings

One of the biggest examples of defensive strategy in the competitive market is between Walmart and Amazon.

When Walmart noticed Amazon’s dominance in convenience and delivery, it overhauled its e-commerce platform, launched same-day delivery services, and introduced grocery pickup options. 

The result? Walmart’s shopping app is now more popular in the US than even Amazon’s app. Addressing these operational gaps helped Walmart maintain its position as a major retail force.

Your Turn to Gain an Edge Using Competitive Analysis!

Competitive analysis is a powerful tool to stay ahead in business. By studying your competitors, you can spot opportunities, fix weaknesses, and improve your strategies. This helps you stay relevant, offer better experiences, and outsmart the competition.

Use the steps in this article to guide your analysis. By understanding what your competitors are doing, you can make smarter decisions and keep moving forward. Competitive analysis isn’t just about keeping up—it’s about leading the way. Stay proactive, and let this strategy help you win.

Competitive Analysis FAQs

The post What is a Competitive Analysis (With Examples) [2025] appeared first on Invesp.

Before jumping into conversion optimization, you need to identify your site visitor by recognizing, for example, concerns, motivations, uncertainties, and trigger words.

One of the first tasks we do as a company, which has been helping marketers optimize their content and their conversion rates, is to delve into understanding the market as best we can prior to initializing the recommendation process.

Most companies are aware of the importance of knowing customers, but few comprehend all that this recognition entails, especially online. The majority are still keeping their heads in the sand and overlooking the high costs of stepping away from clients.

The Global Customer Service Barometer, after interviewing more than 1,000 adult customers on how they view their brands, brings disappointing results 38% of the participants believe that businesses are not customer-oriented and fail to provide enough attention.

Image source: AmericanExpress

As data shows, many people are frustrated by how businesses are still using the same outdated methods to interact with their customers and not paying enough attention to them to satisfy their needs.

We find that companies face one of two challenges when it comes to integrating their marketing data into their selling process:

• Smaller companies do not have the know-how or the means to conduct research to identify and segment their market effectively. Some of them might not even appreciate the value market research brings to their business. As a consequence, they only make assumptions about their clients, which are often wrong, and risk missing crucial segments that can generate large.

• On the other end of the spectrum, many mid to large businesses perform their due diligence researching the market but do not know ways of applying the results. They are unable to integrate that data into actionable insights that can be implemented in their selling process, marketing campaigns, or on their websites.

Regardless of the size of the company, the overwhelming question is: how do you use the marketing data to make sure website copy, design, navigation, and overall site structure will engage most visitors, if not all, and move a high percentage of them through the conversion funnel?

When the average e-commerce store converts only 3-5% of its visitors into customers, you must pause and wonder what is stopping the other 95% from moving forward with the purchase process.

Conversion optimization starts with gathering detailed market information. The more detailed the information, the more precision you will find in the optimization process. You then use that data to shape your website’s interactions with visitors.

The most efficient way to address market concerns is through the creation of buyer personas.

But what are personas? And how can they help with conversion optimization?

Buyer personas are models, examples, and archetypes that humanize and individualize a precise target market. They are hypothetical individuals who represent target consumers.

Image source: Amy Cotton

Through persona creation, you should be able to relate to customers on an individualized level to bring personalization, engagement, and enthusiasm into your website by knowing what appeals to them and what turns them off.

And because buyer personas are at the helm of conversion optimization, they should guide every aspect of your website, including user interface and screen design, process flow, and web copy development.

Too many websites cater to the masses and not to a single person. As a result, the copy is dull, the design is boring, and visitors are left frustrated.

Creating your own buyer personas will help you relate to the different types of personalities of your customers and help you address their needs accordingly.

Market Segmentation Vs. Persona Creation

When we first present the concept of buyer personas to our clients, they frequently confuse persona development with market segmentation. These are not the same. However, persona creation relies heavily on market segmentation.

In order to create buyer personas, you need to understand the market segmentation and the various customer profiles you compiled with your team. Segmentation is the division of the market or population into subgroups with similar motivations.

Keep also in mind that there is a huge difference between marketing to a business (B2B) and marketing to a customer (B2C) although both of them are selling to a person.

The widely used bases for segmenting business-to-customer markets usually include geography, demography, personality (behavior), and psychographic factors. As for the bases used for segmenting business-to-business (B2B) markets, they are different; they include the location, the company type, and the behavioral characteristics.

This article focuses more on the techniques used in B2B markets during the buyer personas creation process for such businesses.

The use of personas is intensively growing, as more than 93% of companies that exceed revenues and lead goals are using personas by segmenting their database accordingly to personalize experiences for their visitors.

A recent study on the most important elements of effective B2B content has demonstrated that audience relevance ranks the highest at 90%.

Image source: Marketing Chart

In another survey, with 640 global senior B2B marketing executives and business leaders, half of them in the internet and software industries, 69% of participants agree that understanding buyers’ needs or pain points is the most effective go-to-market strategy for a product launch.

These studies show it’s now high time for marketers and business owners to put their feet in the customers’ shoes and start building up buyer personas that reflect the real needs and demands of their customers.

Every company, online or offline, must comprehend the market they are working in as well as the best techniques to better cater to their ideal customers.

You can go in-depth and analyze the various demographics and similarities within the market to create segmentations and narrow the target market. This information must be quite detailed to create the ideal target customer.

How Are B2B Marketers Building Buyer Personas?

Cintell surveyed 137 B2B marketers to find out how they build up their buyer personas and what kind of data they use.

Image source: Marketing Charts

The survey segmented these marketers into three different groups, as shown in the graph above. The first group, the one doing well by exceeding revenues and lead goals, was conducting qualitative interviews for both customers and non-customers, including the executive team and the salespeople. The data obtained were more likely segmented by demographic factors than other fields mentioned above, and their focus was oriented towards addressing the fears and motivations of their customers because these are more applicable to buyers as individuals.

Consider a site that sells discount furniture, catering to three different market segments:

• Middle-income families, middle-aged, 70 – 100k household income
• College students, 10-25k income
• Building company

The three markets differ in income, personality traits, and needs. Their interests and motivations behind the purchase are different. One group is purchasing for their families, the other for themselves, and one for potential clients or home staging. The site’s goal is to appeal to all markets equally. Buyer personas help you achieve this goal by guaranteeing you capture each group’s interest.

In this scenario, we would introduce Jennifer, a 45-year-old homemaker. She likes to update her home furniture every year. She enjoys decorating her home and always needs to update because of her rowdy kids. She is price-conscious but looking for something sturdy with good quality. She just wants to ensure that her satisfaction is guaranteed because she needs quality items. Return policies are apparently imperative to her.

Compare these traits to Evan’s, a 37-year-old builder who must find great, unique pieces to stage the homes his company builds. He wants to find pieces that are neutral enough so that prospects can imagine themselves in the home. He also tries to find items that will pique prospects’ interest so he can sell them with the home. He is not as concerned about price but does have a budget.

Finally, the website targets Sally, a 19-year-old college student who is rooming with two other friends. She needs cheap, nice furniture items that are a bit trendier. She is trying to make a quick decision because move-in is within the next couple of weeks. She needs white-glove delivery and someone to build the pieces. She is looking for practical solutions for their tiny space and considering items with tons of storage.

Selling to these individuals is a lot easier than to a general market segment because you can put yourself in Sally, Evan, and Jennifer’s shoes. You can anticipate their concerns and personalize the experience with them.

Buyer personas pave the way for:

1. Understanding what makes different segments tick, what appeals to them, and what turns them off.
2. Relating to the various personality types that will come, address their needs accordingly.
3. Recognizing that all users come at different buying stages. Evoking empathy within site developers, marketers, and sales teams.
4. Knowing competition well, starting from which market segments are likely to identify with a competitor more than with you, and why?

The Four Temperaments

Many philosophers and psychologists have played with the concept of temperaments at length, beginning with Aristotle in 325 BC, passing by Carl Jung in the 1920s, to a modern-day version with David Keirsey in the 1950s. The concept boils down to the fact that humans carry particular sets of characteristics. For a marketer, understanding what makes each of the temperaments tick can help move the needle in conversions.

While you create each persona, you can also assign a temperament (or combinations of them). That way, when designing, you may have Jennifer in mind, but now you also have in mind that she is a caring person, triggered and motivated by social proof, which can be something to address on the website.

These are the four temperaments to consider.

Logical

A logical persona is meticulous, methodical, and detail-oriented.

When a visitor of a logical nature enters your site, she searches for every single detail of your product and services. She also expects your site to offer answers in a logical, systematic, and easy-to-locate fashion.

Someone who is logical is going to research your product or service longer, reading and looking for information about it more than others. The purchasing decision for logical personas comes as a result of researching, shopping around, and getting to know more of your and your competitors’ offers.

How do you address logical personas?

When targeting a logical persona, your site should:

• Have easy-to-locate information.
• Contain answers to all questions about your company (date of establishment, range of products and services offered, reputation).
• Refer visitors to the methodology, about us page, or the blog.
• Detailed product pages with detailed descriptions.
• Systematic explanation of your services.

Caring

The caring persona is impacted by people.

A caring visitor to your site looks for reviews and testimonials. She searches for the opinions of others, as well as the number of customers who have used your product or hired your service. People with this temperament are trusting. Their buying decision is impacted by what others feel and say about your company.

The purchasing decision for caring personas takes some time. They are not disciplined in considering different alternatives, which makes selling them a little bit hard.

How do we address caring personas?

When targeting a caring persona, your site should:

• Provide testimonials.
• Present news quotes and announce awards received.
• Show numbers of clients.
• Display reviews on product pages.
• Showcase social media mentions.

Impulsive

The impulsive persona buys fast and regrets fast.

The credit-card-in-hand type of visitor may purchase on impulse but often regrets the decision. Attempt to understand how to get them to click right away, with the right elements visible. However, try to avoid the regret phase by giving them the information they will need through the conversion process.

Impulsive personas are determined to make a purchase. They need to find a reason to buy from you as soon as they enter your site, or they will leave to make the purchase on another site.

Impulsive buyers are an e-commerce company’s dream client.

How do you address impulsive personas?

When targeting an impulse persona, your site should:

• Present benefits in the form of 3 or 4 extremely convincing bullets.
• Include a big “call to action” in a bright color, immediately following the benefits.
• Make sure the check-out process is smooth and fast.

Aggressive

The aggressive persona is very competitive in nature.

This persona type likes to be outstanding and to acquire the finest products and services. Convincing them that you offer premium, first-rated products or services will help you win them over. These personality types strive to be the best by flushing out the competition.

If an aggressive persona visitor sees a reason that your product or service will place them ahead of the crowd, then you can be sure that they will be loyal customers.

How do you address aggressive buyer personas?

When targeting an aggressive persona, your site should:

• Offer great incentives, such as the best price and the newest product.
• Give a good reason why your product or service is the best and what it will do for them.

Do you need to address all 4 persona temperaments on your web pages?

That is a good question!

These are some of the things you should keep in mind when creating a web page that caters to the four persona temperaments:

1. Depending on what you are selling, you can determine which temperaments you will probably address. If it is something rather expensive, whatever personality type you are dealing with, the likelihood of more methodical instincts will be present. That does not dismiss the fact that many spontaneous individuals still want to buy cars and diamond rings. However, you can expect them to ask “more” questions because it is a greater buying decision and investment. High-value transactions (contracts worth thousands of dollars) will be meticulously reviewed in a very methodical format. For smaller decisions, you are more likely to have all 4 temperaments.

2. The ratio of temperaments within a population is:

• 45% logical
• 35%impulsive
• 15% caring
• 5% aggressive

3. Within a single page, yes, you absolutely need to have the language that can address these 4 temperaments.

In the video below, part of one of our webinars, Ayat Shukairy, and Khalid Saleh talk about the different persona temperaments and explain psychological understandings of personas in a historical progression.

The post Using Buyer Personas: The Cornerstone Ingredient for Better Conversions appeared first on Invesp.

Humans are flawed creatures. We conform to a number of biases outlined by the world’s greatest psychologist. Wikipedia has listed 180 biases that impact our everyday decisions.

This in turn makes our jobs as marketers and optimizers quite tricky. According to social scientists Amos Tversky and Daniel Kahneman, you can’t quite overcome your biases. You’ll always fall into them. Tversky died in 1996 and Kahneman went on to win the 2002 Nobel Prize in Economics for the work the both of the men did together. His work was summarized in the highly acclaimed and 2011 best seller, Thinking, Fast and Slow.Wait so marketers and conversion rate optimizers– they’re biased creatures too!? Reminder: we are as human as the consumer.

I relay the story of my own bias during a qualitative research session with my team where I was completely convinced that the target customer behaved a certain way.

And that’s just taking into consideration the marketer and optimizer who most likely is somewhat conscious of these biases humans hold (if they know what they’re doing). On the consumer side it’s worse. This concept that we cannot change our predisposed biases, ultimately means, we can’t alter consumer behavior.

The bias that Kahneman and Tversky listed as the most damaging: the confirmation bias. Think about the current political divisiveness in America. The confirmation bias allows for this to fester because it’s the effect that leads us to look for evidence confirming what we already think or suspect. The right believes the left is bonkers and is driving the country to shambles, and the left believes the right is extreme and out to destroy the fabric of America. Neither, even when confronted with opposing data to what they believe, are willing to accept anything but what they believe.

This idea scares me about what many marketers and optimizers may be doing. They may not be conscious of this bias and they simply continue to investigate in within a scope of data that confirms what they believe refusing any alternative evidence.

I read an article a year ago in the New Yorker on how facts simply do not change our mind. This is present in our every day life if we consider the deep partisanship of America as mentioned above. But it’s even present in the racial divide in terms of how races view each other and themselves.

The confirmation bias actually impacts me personally very often. I speak internationally at conferences such as Inbound, SMX series, Business of Software and many more. But as a hijab adorning (the scarf on my head) Muslim woman, I’m wearing a label that many people have a confirmation bias against. Whether I like it or not, as much as I talk and reveal my expertise, there are too many people that cannot “change” their minds about me because of the bias they have against Muslims.

Kahneman himself is pessimistically convinced is it almost impossible change our biases citing the Muller-Lyer Illusion. It is basically two lines, both the same length, but at the end of each line the arrows are pointed in different directions; one inward, and one outward.

In the real world, it’s not so easy to reach system 2. And when dealing with daily quick decisions by consumers, the challenge is greater. Most of our decisions are made with system 1 (about 90% of them!).

There are many stories and experiments done testing the effect of psychological biases. What we have found conducting CRO for many years that indeed, it’s not easy to change people’s thinking, although not impossible.  

This is supported by social psychologist Richard E. Nisbett, a social psychologist at the University of Michigan. He believes, as a result of his research, that there is a great possibility in training people to overcome pitfalls of various biases.

The reality I found while researching this article was that the biases touch each and every part of the sales cycle and consumer journey. And this confirms something I’ve talked about in regards to CRO: it’s not about testing your website. If you have a company that is doing template tweaking for you, it’s not going to give you that huge impact you’re seeking. Ultimately every decision at any point in the sales cycle may already have preconceived biases or builds up biases. If the marketer or optimizer is tackling just one small part of the experience, they are missing the boat.

Let’s peel it a part – understand some of the most prevalent biases that impact the consumer, your customer, and you, and then present different methodologies and approaches to countering the particular bias:

The Confirmation Bias

I’ve talked about this specific bias at length above. 

“Confirmation bias, also called confirmatory bias or my side bias,[Note 1] is the tendency to search for, interpret, favor, and recall information in a way that confirms one’s preexisting beliefs or hypotheses.[1] It is a type of cognitive bias and a systematic error of inductive reasoning.”         Wikipedia

How does it impact consumers?

According to Jeff Toister, confirmation bias can: 

• “A customer’s first impression can anchor how they feel about your business.
• Online reviews can convince customers that you’re awesome (or not).
• How quickly and how well you handle problems can cement a reputation.
• Making personal connections with customers can strengthen their positive bias.
• One prickly employee can convince customers you suck.”

Well, for one, we find this present in consumers when they are exposed to negative reputation or reviews about a company. It’s very difficult to “change” their mind’s to believe that the reputation is sound and the negative comments are isolated incidents.

Think about McDonald’s and their attempt to redeem their reputation by advertising better quality, and fresher food options.

What we’ve found is businesses battling confirmation bias need to be transparent and honest. This is paramount to repair the damage confirmation bias causes. Like the infamous KFC full page advertisement admitting their erroneous error of running out of chicken, they tackled the issue head on.

Social media is also a great tool to connect with customers, hear their voice and address the issues head on.

Another way of countering this bias is by feeding into already held beliefs. So if you’re promoting a product of sorts, try to fit it within preconceptions and expectations that your consumer may already have. For example, a client of ours promoting their book’s compatibility with e-learners or e-books like the kindle, use that convenience element that the kindle’s have to promote their own books.

Of course if someone has a positive confirmation bias of your organization you have it made. For example, this happened to me, when Zappos first became popular with their “free shipping both ways” policy. I was infatuated with the ease and convenience of it, that although I had personally experienced inconvenience with their service and found their prices outrageous, I still shopped there. It happened to me when “Ideeli” was a thing. I loved the concept of getting designer labels for a fraction of the price, and ignored some of the negatives I experienced with the service.

I am your typical consumer. Eventually I recognized what was happening and stopped shopping at these places. But the point is, confirmation bias can actually be positive, and what’s key is to keep those already happy customers happier. That rarely happens.

Duration Neglect

“Duration neglect is the psychological observation that people’s judgments of the unpleasantness of painful experiences depend very little on the duration of those experiences.”            Wikipedia

I always tell my clients, our work does not happen in a vacuum. I’m sorry, if your visitors turned customers have a beautiful, seamless purchase experience, but the end result is negative, they will remember the negative part. And that’s why it’s important to us to be a part of the entire experience, because it all impacts the end result of what we are trying to achieve.

In consulting this is a constant struggle. We have clients that are super happy with our service, and one misstep, wrong email, negative performing test, and it all goes to hell.

While this bias is present, it’s actually not difficult to counter. A friend of mine started an e-commerce operation. In all of the packages she included a special letter and beautifully scented tissue paper. She found eventually that this wasn’t sustainable, and removed it from her process as she grew. What she forgot was how that final unpacking is what clings in people’s minds about the product. And that beautiful scent and experience is what is paramount to leave lingering in people’s minds.

So don’t just care about making the sale. Care about the long-lasting relationship, the repeat customer, because that confirms the type of lingering effect your changes had on the consumer (positive or negative).

Loss Aversion

“In cognitive psychology and decision theory, loss aversion refers to people’s tendency to prefer avoiding losses to acquiring equivalent gains: it is better to not lose $5 than to find $5”    Wikipedia

Visitor’s always want to feel like they’re winning. If they feel like they’ve lost or were somehow ripped off, this will certainly leave a bad taste in their mouths.

We has a client that built his entire model on hiding fees from customers. He would make money off of the hidden fees. Long term, this plan did not pay off for him. People felt they lost a few more dollars when they thought they were going to be getting a discount, so the psychological effects were great. Ultimately they were unable to maintain long-term, repeat business in a space that easily could have achieved that for them.

I have this when I shop at any of the online flower shop. The deal looks great but then I see the attached fees for shipping and handling and I become supremely discouraged. What starts out as a $50-60 dollar purchase, reaches 100 easily when everything is said and done. Every single time.

Other Biases that work to our advantage

There are a number of other biases that we need to be aware of. I’m sure we will have webinars and posts that delve deeper on this topic (shedding to light other very important biases).

But the interesting thing is there are a number of biases that actually help marketers and optimizers as well. If we are able to anticipate the negative biases, and tap into the ones that work in our favor, we can really understand what our customers want at a deeper level.

Let’s see what some of the biases are and how to tap into them:

Anchoring – with anchoring this is prevalent in SaaS when listing the different packages – marketers tend to list 3 or 4 with varying benefits, and they anchor the packages with the most expensive package, but offer a “best value” at the most desirable package at the price point that makes the most sense to the company.

Bandwagon Effect – the tendency of people following the group. This is helpful if celebrity or influencers use a product or service, many will “jump on the bandwagon” and get the same product or service.

Framing and Priming – priming specific desirable packages with better value by using terms that help visitors see it. We conducted a test with a gift basket retailer by placing badges on the category page to draw attention and distinguish some baskets over others.

In-group favoritism – Apple does this wonderfully. This is the idea of creating a tribe around the product, and once you’ve committed to the tribe, you likely have the business forever. In Apple, Samsung, and other competitors, they continuously have more products to up-sell their customers. That becomes challenging for retailers that don’t have the same type of product that constantly needs uploading .

Salience – if your product or service has a compelling characteristic, consumers tend to focus on that one thing over others. This makes the job of marketers a bit tricky to ensure that certain characteristics become more memorable for the visitor.

Zero-Risk Bias – no surprise, humans hate risk. That’s why when you offer 100% satisfaction guarantees and free trials, people jump on the chance. If there’s 0 risk, they are more likely to try it.

The Decoy Effect – By presenting visitors with options that offer more value as compared to a more expensive or less expensive option, it taps into the decoy effect bias. Visitors will see it as getting a great deal and go with the medium option.

The Humor Effect – People remember humor, so by offering something funny or running a humorous campaign, you can stick better in people’s minds.

Mere Exposure Effect – Retargeting campaigns work for a reason. By getting your product in a potential customer’s face repeatedly, on many different platforms you’ll be in mind, especially when they need your product or service.

Sunk-cost fallacy – I’ve already invested in it, I may as well see it through. When President Donald Trump decided to send more troops to Afghanistan because: “Our nation must seek an honorable and enduring outcome worthy of the tremendous sacrifices that have been made, especially the sacrifices of lives.” More lives and more investment in a lost war because we’ve already started is a clear case of sunk-cost thinking.

Ultimately it tells us to stick with a bad investment, a product, a service: because of the money we have already lost on it. This happened recently when I decided to watch the movie Crazy Rich Asians. I know I’m one of the few people out there that didn’t enjoy it, but I really disliked it and just went because of the bandwagon effect on me (everyone was buzzing about it). From the onset I didn’t like it but I gave it a chance. 40 minutes in I knew I hated, but thought, I already sat through half of it, I really don’t want to walk out now, maybe it’ll get better. It didn’t. And I was so angry I lost all that time watching a movie I really hated. But I fell into the sunk cost fallacy.

Ok how can this help marketers? Actually, stay away from this if possible. Don’t make people start with your product and service and just stick with it because they already started.

Conclusion

But remember, tricks and simple nudges are no longer enough. Marketing campaigns need to be studied, tested and really push the limits to give visitor’s an experience they won’t forget.

The post The Reason CRO is FLAWed appeared first on Invesp.

Most A/B testing platforms do not provide the option to choose between one-tailed vs. two-tailed analysis. Moreover, until 2015, several popular testing platforms relied on one-tailed in statistical analysis.

The same applies to many other disciplines where having to choose between a one-tailed vs. two-tailed is rarely encountered since the majority of statistical software only offers two-tailed analysis as the single option.

There is a common belief that two-tailed tests are generally “better” in some way, leading to more accurate and more reliable results.

This is an apparent misconception.

Surprisingly, these misconceptions originate from wrong guidelines present in many statistical publications. Let’s take a look at the following from Wikipedia:

“A two-tailed test is appropriate if the estimated value may be more than or less than the reference value, for example, whether a test taker may score above or below the historical average. A one-tailed test is appropriate if the estimated value may depart from the reference value in only one direction, for example, whether a machine produces more than one-percent defective products.”

This sentence is misleading since one is often interested in discovering one direction difference (did the test generate an uplift) even when both directions are possible.

When conducting an AB test, we are typically interested in finding out if a variation generated a generated a statistically significant increase in conversions. Thus, for these cases, a one-sided test is appropriate since we would like to find a difference in one particular direction. According to Kylie Vallee:

“If you’re running a test and only using a one-tailed test, you will only see significance if your new variant outperforms the default. There are 2 outcomes: the new variants win or we cannot distinguish it from the default.”

A two-tailed test should be used when we would like to discover a statistically significant difference in any direction. In this case, we examine if the variation generated a statistically significant increase or a decrease in conversions.

This article covers several issues around the choice between one-tailed vs. two-tailed tests that you should be aware of if you are conducting AB testing.

However, the decision on which approach to choose should always be based on the aim of the experiment and the question of interest, not the technical differences between the two tests. These technical differences are merely a consequence of their statistical construction.

Let’s start with the real-life examples.

Example 1: Using statistical methods with an e-commerce store

Let’s say an e-commerce store is looking to modify product pages to increase the website conversions rate.

The conversion rate for the product page, in that case, is defined as the number of orders divided by the number of visitors to product pages. The marketing team is considering three new designs for the product page. Their aim is to choose the best design out of the four available ones: the existing design and three new ones.

We will show below how to use statistical methods to make such a decision, using the conversion rate as a criterion.

Example 2 – Statistical methods with a SaaS website

A SaaS website is looking to increase its number of subscribers by modifying its pricing page.

The conversion rate for the pricing page is defined as the number of subscriptions divided by the number of visitors to the pricing page.

The marketing team is considering three new designs for the pricing page. The team would like to decide if any of the new designs are better than the existing one, i. e. it leads to more subscribers.

Also, in this case, statistical methods allow for making such a decision using the conversion rate as a criterion of comparison.

Step 1: Formulating One Tailed /Two Tailed Hypothesis

‎To address these examples through A/B testing, we will first formulate your hypothesis of interest.

For both examples, the hypothesis is that the conversion rate for one of the three new page designs ([latex]\rho _{N}[/latex]) is equal to the conversion rate for the current design ([latex]\rho _{C}[/latex]). This called a null hypothesis and is denoted by H0. Therefore:

H0: [latex]\rho _{N}=\rho _{C}[/latex]

Then you formulate your alternative hypothesis H1. The alternative hypothesis can be one of the three possible forms:

H1:[latex]\rho _{N}[/latex] not equal [latex]\rho _{C}[/latex]  (less or greater)

or

H1: [latex]\rho _{N}[/latex] greater than[latex]\rho _{C}[/latex]  (right-sided) – one of the variations will cause an uplift in conversions

‎ or

‎H1:[latex]\rho _{N}[/latex]  less than [latex]\rho _{C}[/latex] (left-sided) – one of the variations will cause a drop in conversions

In the first case, you do not assume that the new design will generate an increase or decrease in conversions. The first type of test is called a two-tailed test.

In the last two cases, you assume the impact of the new design. This is referred to as a one-sided test (left-or right-sided)

In a pure statistical world, you would typically conduct three different AB tests since you have three variations. Each test will run the original against one of the variations. Also, for each test, we will use the same procedure of hypothesis formulation. It is also essential to decide IN ADVANCE what form of the alternative hypothesis you would like to test.

In other words, this choice should NOT be a data-driven but a consequence of a question of interest. You will ask yourself:

• Do you care to see if a variation generates a significant change, or
• Do you care to see if a variation generates a significant uplift, or
• Do you care to see if a variation generates a significant drop

Your decision to use a one-tailed test to prove a statistically significant difference in a particular direction, but not in the other direction.

Typically, you would run a single AB test with three variations against the control. However, you should be aware that doing so could cause a “multiple testing” problem which is a common problem in statistics. It reflects the increasing the probability of discovering false positive significant result as more variations are introduced to a test. There are several solutions to this problem which I will discuss in a later article ( correction for multiple testing or using other statistical methods).

Step 2: Conducting the A/B Test

‎After formulating your hypothesis, you are ready to conduct the AB test. As the testing software collects data on each of the variations, it starts to do some computational work to determine the performance of the control and the different variations.

The A/B testing software computes the z-score (test statistics) which is a mathematical formula based on the observed values of conversion rates for the variations ([latex]r_{N}[/latex]), the conversion rate for the control ([latex]r_{C}[/latex]), the number of visitors to the variation ([latex]n_{N}[/latex]) and the number of visitors to the control([latex]n_{C}[/latex]).

If we assume the same (true) variances for both rates, the Z-score has the form:

[latex]Z=\frac{r_{N}-r_{C}}{\sqrt{r(1-r)(\frac{1}{n_{N}}+\frac{1}{n_{C}})}}[/latex]

r is a pooled proportion (conversion rate) calculated as:

[latex]r=\frac{r_{N}+r_{C}}{n_{N}+n_{C}}[/latex]

In a more general case, when the variances for the two rates are different we cannot use the pooled r estimator and the Z score is calculated using the following formula:

[latex]Z=\frac{r_{N}-r_{C}}{\sqrt{(\frac{r_{N}(1-r_{N})}{n_{N_{}}}+\frac{r_{C}(1-r_{C})}{n_{C}})}}[/latex]

Notice the difference!

When we created our test hypothesis, we talked about [latex]\rho _{N}[/latex] and [latex]\rho _{C}[/latex]. When we conducted the test, we are using [latex]r_{C}[/latex] and [latex]r_{N}[/latex]

[latex]\rho _{N}[/latex]  and [latex]\rho _{C}[/latex] are hypothetical conversion rates for the whole populations of users. They are the assumed conversion rates for all visitors who will come to the page.

[latex]r_{C}[/latex] and [latex]r_{N}[/latex] are the actual observed conversion rates while are conducting the AB test which runs for a sample from the population.

Why is this important?

This is the biggest challenge of AB testing! You run the test for a specific period and observe the conversion rates for the control and the variations. You are collecting the behavior of a sample of your users and using that sample to judge how the whole population will react to these designs.

Here is a highly sanitized example to drive this point home:

Let’s say that you have a page that gets 10,000 visitors/day. You run an AB test for a period of 14 days. So during that period, your page gets 140,000 visitors. Before launching the test, your boss tells you that you will use the winning design from the AB test for the next 365 days. In this case, the population is 10,000 * 365 = 3,650,000 visitors.

When you conclude the AB test, you will use the data from the reaction of 140,000 visitors to judge the response of 3,650,000 visitors. So, you are using data from 3.8% sample to judge the performance of the population.

Step 3: Decision Making

Critical region

The critical region is a set of outcomes of a statistical test for which the null hypothesis is to be rejected.

Statistical testing using critical region -step by step:

Critical region in two-tailed A/B tests

Let’s consider a numerical example for the above step-by-step analysis.

Let us assume that we observe two conversion rates:

• The control conversion rate 70% (100 visitors, 70 conversions)
• Variation conversion rate 80% (100 visitors, 80 conversions)

For the sake of simplicity, I used a small number of visitors and conversions. I also assumed that both the control and the variation will get the same number of visitors. I am also assuming equal variances for both rates in the population.

Formulate null and alternative hypotheses

Let’s assume that we want to test if the new design will result in an increase or a drop in conversion so we will use a two-tailed AB test.

Calculate Z score

First we calculate pooled conversion rate:

[latex]r=\frac{80+70}{100+100}=\frac{150}{200}=0.75[/latex]

[latex]Z=\frac{0.8-0.7}{\sqrt{0.75(1-0.75)(\frac{1}{100}+\frac{1}{100})}}[/latex]=
[latex]\frac{0.1}{\sqrt{0.1875\cdot
0.02}}=\frac{0.1}{\sqrt{0.00375}}=\frac{0.1}{0.0612}=1.633[/latex]

Find critical values

After calculating of the z-score, we check if this value fails into so-called critical region.

The critical regions are the x-axis part of the shaded areas on the pictures below. They look differently for different types of alternative hypotheses:

The critical region (red part on the x-axis) is placed on both sides for two-sided test and on one side for the one-sided test.

Since we are interested in conducting the two-sided test, so the critical region looks like on the third picture. It consists of the two intervals : (-∞;-critical value] and [critical value;+∞).

The negative critical value (- critical value) is the point on the x-axis where the red region finishes on the left side and when it begins on the right side. [latex]z_{1-\frac{\alpha }{2}}[/latex]

We take the critical value from the special statistical tables.  The critical value is a value determined by the significance level (see the next section) and is denoted for the two-sided test as.

For our example, it will be equal to 1.96. So the critical region in our example has the form:

(-∞;-1.96] and [1.96;+∞)

Make a decision

Since our computed Z score value equals 1.63 it does not fail in the right part of the critical region. Therefore we do not reject the null hypothesis.

Therefore, we cannot state that one of the designs will increase conversion rates more than the other. All we can state is that we do not have enough evidence to conclude the relevant difference between the control and the variation.

The picture shows the distributions of the Z-score. This illustrates all possible values of the statistical test formula that could be obtained if you could repeat your experiment very many times. As can be noticed they are typically distributed (around the zero mean), so the density curves are bell-shaped. They are referred to as normal or Gaussian distribution. It is one of the most popular statistical distributions. According to Cliffs notes website:

“The decision of whether to use a one‐ or a two‐tailed test is important because a test statistic that falls in the region of rejection in a one‐tailed test may not do so in a two‐tailed test, even though both tests use the same probability level.”

Normal distribution widely used since it describes most of the mechanisms in nature. Like any statistical distribution, it illustrates the proportions of some continuous feature in a population. For example, if you measure the age of some defined group like the age of users visiting the website then, based on the mathematical theory, it will have the normal distribution, provided that we have a large sample. Statisticians say that “everything is normal sooner or later.”

The critical thing about this distribution is that is centered around the mean of the observed feature (mean age in the example ), and it can be more or less spread around it depending if the variability in the observed group is less (less spread) or larger (widespread).

Critical region in the one-tailed A/B test

What would be our conclusion if we would like to perform the right-sided test, so testing that the new design will convert more visitors compared to the control? Then the critical region would look like on the first picture.

The value of the Z score remains the same.

So it is still 1.63.

Find critical value

The critical value [latex]z_{1-\alpha }[/latex] is now 1.64 so the critical region is of the form [1.64,+∞]

Make a decision

Since the test statistic does not fail in this interval, we cannot reject the null hypothesis about the equality of the two designs. Thus, we cannot conclude that the new design is better than the current one. However, we see that we are now closer to the critical region than before. According to Surbhi S.:

“One-tailed test represents that the estimated test parameter is greater or less than the critical value. When the sample tested falls in the region of rejection, i.e. either left or right side, as the case may be, it leads to the acceptance of alternative hypothesis rather than the null hypothesis.”

The results would be much different if we assume non-equal variances for both groups!

Z score will be then 1.644 (I am skilling the calculations but the formulas are above!).

In this case, the Z-score falls into the critical region for the right-sided test. Therefore, we can conclude that the new design does increase conversion rates.

As you can see the assumptions are crucial here and the right-sided test has more chance to detect the difference.

Do we need to do it the Z-score calculation by hand?

In practice, the AB testing software or statistical software does all the calculations for us.

As a result, it provides so-called p-values as a final result of a test. P-value is the probability of rejecting the null hypothesis when H0 is true. If the P-value is less than the significance level (usually 0.05), than we say the result is significant. If it is greater or equal to the significance level, then we state that the difference is not significant.

Significance level

The significance level of a test (sometimes referred to as Alpha “α”) is the shaded area.

The α false null hypothesis (concluding that the new design is different from the existing one) you make this right decision with the probability of 1-α (see the smiley face in the first row on the graph below).

You have to decide how big the significance level α can be for your test. Typically we use 0.05. ‎When you reject the null hypothesis, statisticians say that the test is significant.

The probability that you were able to detect the difference between the control and the variation, if it exists, is called power and is denoted by 1-β. (See the table above).

• If you fail to reject the null hypothesis, you made the right decision if H0 was true.
• if you fail to reject the null hypothesis, you made the wrong decision if H0 was false. This is typically referred to as type II error (β).

In the last case, statisticians say that your study was underpowered meaning that you didn’t have enough evidence to detect the true difference. The smaller the difference between the true rates the more data you need to detect it (more traffic).  It is like looking through a small magnifying glass. You need a bigger one if you are looking at a tiny object.

Confidence intervals for one-tailed vs. two-tailed tests

We typically like to construct a “confidence interval” to determine the true difference between the two rates.

A confidence interval is an interval that contains all possible values of that difference that you would obtain if you could repeat your experiment very many times, with the probability.

For example, for 100 repetitions of the same experiment, the computed difference between the rates would fail into 95%CI in 95 cases.

Calculating the confidence interval for the two-tailed test

First, we need to calculate the limits of the interval [LL,UL].

[latex]LL=r_{N}-r_{C}+z_{1-\frac{\alpha
}{2}}\sqrt{\frac{r_{N}(1-r_{N})}{n_{N}}+\frac{r_{C}(1-r_{C})}{n_{C}}}[/latex]

[latex]UL=r_{N}-r_{C}-z_{1-\frac{\alpha
}{2}}\sqrt{\frac{r_{N}(1-r_{N})}{n_{N}}+\frac{r_{C}(1-r_{C})}{n_{C}}}[/latex]

For the two-sided test those limits can be calculated as follows:

[latex]UL=r_{N}-r_{C}+z_{1-\frac{\alpha }{2}}\cdot
\sqrt{\frac{r_{N}(1-r_{N})}{n_{N}}+\frac{r_{C}(1-r_{C})}{n_{C}}}=0.8-0.7+1.96\sqrt{\frac{0.8(1-0.8)}{100}+\frac{0.7(1-0.7)}{100}}=0.1+1.96\sqrt{\frac{0.16}{100}+\frac{0.21}{100}}=0.1+1.96\sqrt{\frac{0.37}{100}}=0.21[/latex]

[latex]LL=r_{N}-r_{C}-z_{1-\frac{\alpha }{2}}\cdot[/latex]

[latex]\sqrt{\frac{r_{N}(1-r_{N})}{n_{N}}+\frac{r_{C}(1-r_{C})}{n_{C}}}=0.8-0.7-1.96\sqrt{\frac{0.8(1-0.8)}{100}+\frac{0.7(1-0.7)}{100}}=0.1-1.96\sqrt{\frac{0.16}{100}+\frac{0.21}{100}}=0.1-1.96\sqrt{\frac{0.37}{100}}=-0.019[/latex]

This is under of assumption of different variances for both rates. In case of a pooled rate estimator the formula under the squared root changes like for the Z test statistics. So for our numerical example from paragraph we obtain:

Therefore the 95% CI for the two-sided test has the form: [-0.019;0.219].

It means that the true values of the difference between the two rates vary from -0.019 to 0.219.

Notice that the interval contains zero, which means that it possible that there is no difference between the means at all. Therefore, we state that the result is not significant.

Calculating the confidence interval for the one-tailed test

We can also compute the confidence interval for a one-sided test.

Confidence interval, in this case, has the form of (-∞;UL] for right-sided test and [LL;+∞) for the left-sided test. For the right-sided test the formula for UL is as follows:

[latex]LL=r_{N}-r_{C}+z_{1-\alpha
}\sqrt{\frac{r_{N}(1-r_{N})}{n_{N}}+\frac{r_{C}(1-r_{C})}{n_{C}}}[/latex]

And for the left-sided test the formula for UL is of the analogical form:

[latex]UL=r_{N}-r_{C}-z_{1-\alpha
}\sqrt{\frac{r_{N}(1-r_{N})}{n_{N}}+\frac{r_{C}(1-r_{C})}{n_{C}}}[/latex]

For the two-sided tests, we also check if zero falls into the interval. This is equivalent to statistical testing.

In practice, you use statistical software to compute the confidence interval. If it has the option of only two-sided interval then take a double of alpha and only half of the interval.

The common misunderstanding of two-sided tests

Some marketers think that they can test if a new design will increase conversions or cause a drop (or vice versa) in a one-sided test. So mathematically, that the H0 in two-sided tests is of the form:
H0: [latex]\rho _{N}>\rho _{C}[/latex]

Vs

H1: [latex]\rho _{N}<\rho _{C}[/latex]

Or
H0: [latex]\rho _{N}<\rho _{C}[/latex]

Vs

H1: [latex]\rho _{N}>\rho _{C}[/latex]

That is not correct. The null hypothesis (H0) ALWAYS has an equality (in two or one-sided tests)!
H0: [latex]\rho _{N}=\rho _{C}[/latex]

Only the alternative hypotheses are with inequality signs in one-sided tests.

So, you do not test if a new design causes an increase vs. causes a drop in conversions in a one-tailed test.

You test if:

the new design conversion rate is equal to the control conversion rate

versus

the new design conversion rate is higher than the control.

Therefore, when you do not reject the null hypothesis in a one-tailed test, you do not conclude anything about the new design. You do not have evidence to state that it is different from the existing one.

Worst-case scenarios

Image Source: KingKong

When you reject the null hypothesis in the two-sided test, you do not know if the new design is better or worse.

All you can conclude is that it is not equivalent to an existing one. You can always determine the direction of the association by looking at your point estimates of [latex]\rho _{C}[/latex] and[latex]\rho _{N}[/latex], which are the observed conversion rates [latex]r_{C}[/latex] and [latex]r_{N}[/latex]. If  [latex]r_{C}<
r_{N}[/latex] you may use right-sided test and if ‎ [latex]r_{C}>r_{N}[/latex] you may try the left-sided one.

Cons for one-tailed tests

Let’s assume that in reality [latex]\rho _{C}>\rho _{N}[/latex] so the new design is worse than the current one. Then if you use the wrong direction of a one-tailed test (right- instead of left-sided), you will be not able to reject the null hypothesis that both designs are equally good.

If you would like to compute 95 percent confidence interval (95%CI) for the true difference between the two rates, note that in case of 1-sided test it is infinite from one side.

So for right-sided test 95%CI looks like [number, +∞) and (-∞, number] for the left-sided test.

Cons for two-tailed tests

When the conversion rate of the control and the conversion rate of the variation are not equal, but the difference between them is very small, you will not be able to detect this scenario since you “spend” all the power on both sides instead of one.

Let’s summarize the advantages and disadvantages of both methods:

A/B test power

Not rejecting the null hypothesis does not mean that you accept it.

Again, let me state that: You never accept the null hypothesis!

If your test does not produce a statistically significant result, that only means you did not have enough evidence to reject the null hypothesis. So, it might mean that your AB test was underpowered or that the true difference between the conversion rates is very small and if you would like to detect if you need more power.

It is always a good idea to calculate the power after conducting the comparison to see if the power is not too small for a given sample size. It is even better to calculate the required sample size when planning your experiment to achieve a minimum of 80% power in your test.

When you calculate the power before planning your experiment you need to specify the minimum relevant difference that you would like to be detected. The computed sample size will be adjusted to achieve that.

The formula for the sample size calculation for the two rates comparison experiment is as follows:

For not equal variances of the two rates:

[latex]n_{C}=R\cdot n_{N}[/latex] , [latex]n_{N}[/latex] = [latex]\frac{\rho _{N}(1-\rho _{N})}{R}+\rho
_{C}(1-\rho _{C})\frac{(z_{1-\beta }+z_{1-\frac{\alpha }{2}})2^{}}{(\rho
_{N}-\rho _{C})2^{}}[/latex]

Where R is the ratio of users in a control of a new design group (equals 1 if the same number of users is assigned for both designs)

ρ is the pooled true rate (for which r was the estimator)

ß is the desired power of the study (usually 0.8 or 0.9)

[latex]\rho _{N}-\rho _{C}[/latex] is the minimal difference between the rates we would like to detect

[latex]\alpha[/latex] is the chosen significance level

It is a good idea to look how the minimal sample size for the study changes depending on the minimal difference you would like to detect in a study (effect size). The sample size increases as the effect size decreases in a nonlinear way, which is illustrated in the graph below:

Let’s calculate the minimum sample size for our numerical example.

We assume R=1 so we want the same number of users to be assigned for the control and the variation.

Let us further assume that we want to detect 0.1 difference between the two conversion rates, the one we just observed in our sample size and we assume the same values of the true rates like the ones we observed.

Suppose we would like to have the power ß equal 0.9 and significance level [latex]\alpha[/latex] equal to 0.05 for the two-sided test.

Therefore we obtain (omitting the exact calculations):

[latex]n_{C}[/latex] = [latex]R\cdot n_{N}[/latex] , [latex]n_{N}[/latex]=[latex]\frac{\rho _{N}(1-\rho _{N})}{R}+\rho
_{C}(1-\rho _{C})\frac{(z_{1-\beta }+z_{1-\frac{\alpha }{2}})2^{}}{(\rho
_{N}-\rho _{C})2^{}}[/latex] equal to 388.

But if we change the expected rates to 0.7 and 0.85 we need only 118 users per group. That is only 18 more than we had for our samples!

Usually, the statistical software or the online calculators allow us to compute the sample size as well as the power.

Let us look at the power now.

Suppose that the alternative hypothesis is true and the difference between the rates is [latex]\theta[/latex]. Then the formula for the power for the two-sided test is as follows:

[latex]1-\beta =\phi (z-z_{1-\frac{\alpha }{2}})+\phi (-z-z_{1-\frac{\alpha
}{2}})[/latex] , where [latex]z=\frac{\rho _{N}-\rho _{C}}{\sqrt{\frac{\rho
_{N}(1-\rho _{N})}{n_{N}}+\frac{\rho _{C}(1-\rho _{C})}{n_{C}}}}[/latex]

and is the cumulative normal distribution function.

For our example, we had only 0.38 power to detect 0.1 difference between both samples equal 100 per each group. But if we change the expected rates to 0.7 and 0.9 the power increases to 0.95!

How to increase A/B test power?

Gif source: Giphy 

One way of increasing a power is using a one-sided test.

Another is to increase the sample size which in case of A/B testing means increasing a traffic.

Remember that any, even very small difference between the rates could be detected if the traffic is large enough. The question is if this very small difference like 1% between the two rates is still relevant for you. In other words, can you state that the new design is better than the current one based on such a small detected difference? Kylie Vallee explains:

“Testing is supposed to make it easier for marketers to understand the impact of a certain change without the need for IT intervention, but when the difference between one-tailed and two-tailed tests goes ignored, both the marketer’s time and the IT resources are at risk of being wasted. “

Bad and good habits when it comes to one-tailed vs. two-tailed A/B tests

• Use a two-tailed test when you do not have a clue which one is a better approach for your specific situation.
• Do not conduct one after the other. If you are not able to reject the null hypothesis in 2-sided test (with less power) you will not be able to reject it with a one-tailed test (more power).
• Do not conduct 1-sided test after 2-sided. If you were able to reject the null in a two-tailed test you will be able to reject it in a one-tailed too (with the right side of the association chosen).
• Conduct a one-tailed test if your 2-tailed test was not enough powered.
• If you do not have a strong evidence that the new design is better than the current one use two-sided test to account for both possible: harmful and beneficial effects of a new design.
• Most software uses 2-sided tests but it is very easy to use them to conduct the one-sided test. In order to do it, it is enough to multiply your current significance level for the 2-sided test by [latex]2\alpha[/latex]. So, use[latex]\alpha[/latex]instead of.

The post One-tailed vs. Two-tailed Tests – Everything You Possibly Need to Know One-tailed vs. Two-tailed A/B Testing appeared first on Invesp.

Visits, page views, bounce rates, cart abandonment, etc. are all great metrics, but their utility wanes when it comes to their ability to provide any actionable intelligence about advanced efficiency metrics (attribution, channel ROI, etc.) and effectiveness metrics (think strategic business outcomes like market share, category growth, etc.).

Advanced metrics inevitably require combining data from various sources and businesses to make informed decisions about the best approach to implementing analytics rigor that is in line with business priorities, resource availabilities and commercial constraints.

In this post, I discuss the importance of tagging within the context of digital data mining and outline 3 different implementation approaches. I will contrast the limitations of various tagging approaches when dealing with advanced data analysis scenarios and how they impact decision making.

Digital data collection-The technology options

The common vocabulary used for enriching clickstream data is tagging. For the purpose of a wider discussion around digital insights though, we swap the word tagging with Digital data collection. A capability that can be implemented using 3 distinct approaches.

Option 1– Using Plain JavaScript

Tech-savvy analysts would already be aware of how JavaScript tags are typically used in conjunction with pixel-based tracking techniques (Google Analytics, Coremetrics, Adobe Analytics etc.) to capture data about important events relevant for specific business context.

Image source: Tekraze

This technique is the predominant method for collecting digital behavior data, yet it has its limitations in terms of how effectively and efficiently can the data be collected and the ease with which it can be assimilated to provide advanced digital intelligence.

Consider for example a simple use case of a small SaaS business that is just starting out with its analytics journey. Basic metrics such as page views, unique sessions, newsletter signups, CRM leads, collateral downloads etc. would likely be enough as a starting point to building an optimization agenda. The company is likely using simple lead capture forms with inbuilt support for plugging into tools such as Google Analytics. In such cases, simple JavaScript snippets are written directly within the CMS (e.g. within WordPress themes or landing page builder platform) would likely be enough to send any additional data into the analytics tracking tool.

Small team sizes, in-frequent website code updates, single installation web properties, and an online-only acquisition channel are some of the factors that make it perfectly fine to simply use vanilla JavaScript coding to enrich basic clickstream data already captured by the web analytics tool.

In terms of technology options, tools such as Google Analytics or Mixapanel can be easily set up to use JavaScript-based coding approach.

Option 2 – Using Tag Management Technology

Tag managers provide an alternative to the direct JavaScript tags-based data collection approach but come with their own limitations around getting insights for metrics that were not originally planned when the data layer was put in place or where merging with offline datasets is required.  

Consider the scenario of a media publisher whose business model relies heavily upon how well it can monetize its digital properties. Some specific considerations around business context include the following:

1. The publisher operates multiple sites for varying audience categories, but all of them just follow the same underlying analytics architecture.
2. The Company is looking to roll out an analytics solution that will help better quantify digital engagement so it can use the stats to demand better pricing for its digital inventory.
3. The Company wants to segment its users based on various behavioral signals that the visitors exhibit as they browse through the company’s media properties.  A Technology advertiser looking to target its content specifically to an engaged and in-market C-level audience would likely pay a higher price for ad exposure if the company can show the advertiser’s ads specifically to cookies that meet this criterion.

How can this be implemented technically?

Behavioral signals used to assess engagement are generated as the user interacts with a website.

Is it still advisable to capture these using plain JavaScript?

Perhaps not. Consider the following arguments to understand why;

1. The company has multiple web properties with different HTML markups. Building code that reads directly from HTML DOM can be highly problematic because the changes will have to be implemented separately for every DOM variation. For example, we need to tag all visitors who come from a specific campaign and interact with an article in a certain manner (spend > x secs on a page, reach section y of the page, etc.). Site A identifies all such articles of interest as HTML id ‘A’ while Site B uses different identifiers. How do we write JavaScript code once and make it work across HTML DOM variations?
2. Every digital property has its own release cycle, which means that putting a code into production cannot happen simultaneously if we inject JavaScript code directly into web pages. This will likely present significant reporting challenges when conveying campaign performance figures to Clients.
3. The specific signals that define ‘intent’ may be a matter of experiment and involve trying out different JavaScript codes before adding a cookie to a segment. For example, the first iteration might assume engagement to be consisting of:

a. Visitor spending > x seconds on an article

b. No exit from the landing page

c. At least one follow-up interaction (submit a query, phone call, online webinar registration etc.)

However, the company might want to try out different such conditions and then run A/B experiments to identify the most relevant definition of engagement. All this requires a significant amount of testing with JavaScript coding variations and having a dependency on code release cycles can be a significant drag in the marketing team’s ability to provide answers to sales teams about exactly what constitutes engagement.

4. The operational issues around maintaining JavaScript code that must change every time the HTML markup is changed can be simply unmanageable and cost prohibitive. Consider the process nightmares involved in having to communicate code changes to a central team or worst, having to take formal approvals before making HTML DOM changes!

As you can see, directly injecting JavaScript code into HTML pages may no longer be a viable option in this specific scenario.

A better approach would use a Tag Manager solution that can provide the following benefits:

4. 4. 1. Tag Managers use an intermediate data layer that can decouple server-side logic from HTML markup. For example, instead of having separate code on site A that counts the number of downloads of a form with div id ‘A’ and then another code on site ‘B’ that identifies the same form with div id ‘B’, we could write and deploy a single snippet that fires for every value in an intermediate JavaScript variable ‘C’. This variable would be populated at the time of page load and would require no changes in the client-side JavaScript code.
      2. As long as an intermediate canonical data format is used (a.k.a. the data layer), individual website owners would have the flexibility in using whatever HTML markup that they have been using historically and not run the risk of break existing functionality.
      3. One of the biggest USPs of the Tag Management approach would be that all changes to JavaScript tags can be implemented centrally as opposed to on individual properties. This would allow not just the centralization of Technical Architecture but also significantly reduce the time taken to roll out tag changes.

But of course, these benefits come with additional costs and implementation complexities and every business must make its own unique assessment around which approach might suit better given a set of constraints about business requirements, skills availability, and time/budget constraints.

In terms of technology options though, tools such as Google Tag Manager, Qubit, Ensighten can all be considered as viable platform choices.

Option 3- Using Server-side tagging on raw clickstream data

This is the most flexible but the most resource and cost intensive approach to data collection and involves creating meta-data on the fly instead of using JavaScript-based tags.

The first two scenarios outlined above focused entirely on tracking behavior online.

But what if analysis needs require working with a combination of online/offline datasets?

To illustrate the tracking complexity in this situation better, consider a large B2B Software Company that captures leads online but where the actual conversion happens after a number of interactions with the website, webinar presentations, offline events, and sales team calls.

How does this company profile and segment its prospects and customers? Can we still feasibly send interaction data from multiple channels into the analytics tool via the Tag Management solution?

Perhaps not. Consider some reasons as to why not;

4. 4. 1. Every interaction would have a different customer/prospect identifier. Even if you could somehow send in offline data into the analytics tool, you would still need to bring data out to create a single profile that combines the data from different events into a record with the common identifier. Tag Manager and certainly basic web analytics tools such as Google Analytics are simply not designed for this use case.
      2. Most large companies have enterprise-level data warehouses and advanced business intelligence tools that are purpose-built for aggregating and analyzing complex data sets created from varied sources. Going off the established Enterprise Architecture would almost always be resisted by most IT teams.

A better approach in this scenario would be to deploy a server-side tagging architecture which allows not just attaching arbitrary metadata to visitor level clickstream records but also facilitates complex ETL operations through integration with leading data integration tools.

Such an approach could include following general steps:

4. 4. 1. In addition to sending interaction data into the web analytics tool, we would record the cookie ID in CRM as soon as a lead is generated online.
      2. The backend code would ensure that the cookie ID gets associated with the unique CRM record identified.
      3. The cookie ID could then be used to regularly pull all web behavioral data into an intermediate data store.
      4. Finally, the behavioral data from intermediate data store could be merged with data collected from other offline interactions such as event attendance, roadshow participations, webinar registrations on third-party sites etc. to create composite datasets for powering advanced customer intelligence.

By following the 4 steps above, you will have a unified profile of every visitor and which combines data from online(web/mobile) and offline interactions would allow the brand to develop advanced personas that can then be deployed in personalized marketing.

Tools such as iJento, Tealium, and Adobe Insights provide varying degrees of capabilities when it comes to server-side tagging.

Summary

Small businesses may never have an immediate need for implementing advanced digital measurement infrastructures but would like to at least be aware of the options. Mid-sized companies may have clarity about the specific analysis needs but lack the knowledge (or budget!) to get there. Large companies may have both the need and the knowledge but might lack the roadmap that incrementally onboard analytics rigor in line with business priorities and resource constraints.  

Regardless of the context though, analytics and conversion specialists should have a perspective of the various approaches (and the pros and cons of each) that can be leveraged to put in place robust digital insights capabilities.

The post 3 Approaches to Harnessing Advanced Digital Insights With Data appeared first on Invesp.

She conducted experiments early in her tenure that were groundbreaking. She set up a tasting table at a grocery store, offering visitors a taste of an assortment of 24 different jams. 60% of the shoppers stopped by and took a taste. But then she set up a table with just 6 flavors. A lesser amount of 40% of shoppers stopped by and took a taste.

But the kicker was that 30% of those who stopped the 6-choices table ended up making a purchase, vs. a measly 3% of those who stopped by at the table with the larger assortment.

This study “raised the hypothesis that the presence of choice might be appealing as a theory,” Professor Iyengar said last year, “but in reality, people might find more and more choice actually to be debilitating.”

Over the years, versions of the jam study have been conducted using all sorts of subjects, like chocolate and speed dating.

What’s the point of this? While companies try to push a number of different products at customers and site visitors, however, research proves simple is better. When humans are confronted with too many choices, they will be unhappy with their selection or unable to make a choice. According to Hoa Loranger:

“Humans have limited capacities for processing information and often choose the path of least effort, even though an alternative path would result in better outcomes. Users often take shortcuts and may appear lazy, but their actions are ways of protecting themselves from information overload and fatigue.”

Think about the last time you went to a restaurant, if the menu is too big (like Cheesecake factory), you take forever to make a decision, and you may just revert to something you’ve already tasted before. Where is the fun in that?

The abundance of choice is often “more limiting” to the consumer than having a controlled number of choices, which makes the decision and precision in the choice easier.

According to Nora Popova:

“Feeling overwhelmed by number of potential options often leads to inertia.”

We’ve all experienced this online. Think of the last time you shopped at an e-commerce store and were presented with an insane number of options. For instance, if you’re looking to buy a pair of shoes and even know in your head what kind of shoes you want to buy,

Even if you have a ballpark idea of what you want, you will likely find hundreds of options on a specialized shoe website.

You may even veer off what you are searching for in the first place – get frustrated and leave.

Is Abundance of choice ever a good thing?

Well, in case you haven’t noticed already, humans are very interesting creatures, and there is really a right answer when it comes to how people think.

As the renowned American psychologist Barry Schwartz says in his paper ‘Tyranny of Choice’

“As the number of choices we face increase, the psychological benefits we derive start to level off… Some of the negative effects of choice… begin to appear and rather than level off, they accelerate.”

What’s clear is when humans are confronted with many choices, their expectations rise, which likely results in dissatisfaction.

Schwartz points out some results of too many choices:

1. People end up selecting something good enough instead of the best option
2. Expectations are much higher, which results in greater disappointments
3. You pay more attention to what others are doing with more choices, rather than deciding what’s best for you

How many times have you made a decision about something when presented with many options, and then looked back regretting and maybe thinking you could have done better? It happens all the time. Like the last time I booked a hotel on Orbitz I thought, but what if I chose that other hotel?

Janelle De Weerd offers:

“Make the decision less painful: Use simple design, give shoppers time to decide, provide choice shaping information.”

When you have too many options to choose from, you will make choice and then start to feel that you could have done better. You might feel poorly about the choice you made because you will always think you could have made a better choice because there were so many options available. In the end, it will be like a domino effect. Triggered by the abundance of choice leading to high expectations finally followed by dissatisfaction.

Barry Schwartz also says something similar in his interesting Ted Talk. He talks about buying a pair of jeans. He states that when back in the day when so many options of jeans were not available and almost everyone used to buy the same kind of pair, they used to walk out relatively happy. Though the jeans may not have fit well, they were not dissatisfied because not too many options were available.

But now, with hundreds of kinds of jeans made available to us, we almost never are happy. Because we know we could have done better. We literally had hundreds of jeans to choose from.

Simplicity Over Abundance in Online World

We ran a qualitative study for customers visiting a client site selling home goods and stationery items. One of the questions we asked visitors who intended to leave (exit intent) what was the source of their dissatisfaction with the site that ultimately prompted them to leave?  The said the product selection was too limited.

After reading all of these studies by professors Schwartz and Iyengar, this perplexed us. Visitors then want to see a great breadth of a product but get overwhelmed and likely dissatisfied with too few options.

So what’s the solution?

Consider a site like aliexpress.com for women’s dresses, the results come back with 464,868 choices. Talk about a breadth of product. But this can cause grave dissatisfaction because of the abundance of choice is magnified with endless options. In a physical world, the space available to showcase any products is limited. But virtual world is infinite, and the number of products that can be displayed is endless.

I consider my experience shopping at a store with a limited product like Ann Taylor, vs. going to a store like Marshalls with too much product. My experience at Ann Taylor is always more positive.

And the jam study findings reaffirm my feeling and this state that our human brains, both off and online, can’t deal with too much choice. The excess choice ultimately leads to anxiety, and the brain begins to focus not on what it has, but instead on what it doesn’t have. The greater the sacrifice, the more dissatisfaction is felt.

Let us also see how the abundance of choice works in SaaS industry.

The industry standard is to have between 3 and 4 packages on the pricing grid. Many SaaS pricing pages have that number and tend to highlight one of the plans as “most popular”.

What becomes clear is that it’s not the number of plans that matters [abundance of choice], but the amount of information that a person has to consider.

If we see the pricing package for both the websites, one offers 3 plans and the other 4 plans. BidSketch purposefully presents its users with only 1 factor to consider, number of users. In contrast, Surveygizmo used to give its users 15 factors. According to information overload theory, choosing a plan at Surveygizmo will be more overwhelming than choosing a plan at BidSketch because of the number of options present.

Less is more

Benjamin Scheibehenne, a research scientist at the University of Basel in Switzerland, said it might be too simple to conclude that too many choices are bad, just as it is wrong to assume that more choices are always better. It can depend on what information we’re being given as we make those choices, the type of expertise we have to rely on and how much importance we ascribe to each choice.

Mr. Scheibehenne recently co-wrote an analysis, to be published in October in The Journal of Consumer Research, examining dozens of studies about choices. One problem, he said, is separating the concept of choice overload from information overload.

Gif source: Tenor

In other words, he said, how much are people affected by the number of choices and “how much from the lack of information or any prior understanding of the options?”

As per the 401[k] pension fund study, if more fund options are offered, fewer people participate. And the highest participation rates are among those employees who are automatically enrolled in their company’s 401(k)’s unless they actively choose not to.

This is a case where offering a default option of opting in, rather than opting out doesn’t take away choice but guides us to make better ones.

Sometimes, it’s a good thing

Having discussed the perils of the abundance of choice, sometimes too much of choice can be a good thing too.

We have discussed hundreds of pairs of jeans to too many dresses on the website to choose from, but have we ever actually walked out of a website or an actual store without buying anything because we had too much of choice?

Sometimes we actually walk into a place because it offers a wide choice. Like various kinds of cheeses at a deli. For cheese lovers, they likely go because they are offering a huge choice of cheeses.

Egor Driagin offers:

“Help customers to identify the dominant option in the choice set:

This could mean making it easier for users to eliminate “inferior” options in the choice set (review ratings, bestseller tags, and filters).

This could mean designing products for distinct segments so that users would not need to make complicated trade-offs between products.”

Simplicity or abundance of choice doesn’t revolve only around the number of products or services to choose from, but also comes into picture while choosing the specifications of the product or a particular service.

For example, the iPhone remains to be an extremely popular product despite the fact that there are many mobile phones out in the market which are cheaper, have bigger screens and a lot more features. That is because Apple has simplified the decision-making process by providing the users with exactly what they need, without too many frills. Of course, there are some users who like the complexity of the technology, but a lot more people have proven to like the simplicity and easy to use technology. And that’s the reason Apple dominates the market.

When you have an abundance of choice, there are ways to make the decision easier for your visitors:

1. Filtration: Providing visitors a way to find precisely what they need allows for a better experience, without losing breadth of choice.
2. Anchoring: is when an initial piece of information is used to make subsequent decisions (reg vs. sale price). Incorporating anchoring in the choices visitors have, helps them make decisions.
3. Frame and account: is how our decisions are shaped based on language 5% fat vs. 95% fat free. By suggesting different variants to the language on the variety of products you can help visitors make a better decision.
4. Prospect theory: is about how we evaluate options and make decisions. Visitors will estimate in a biased and unlikely way the perceived likelihood of all the options. The way you order the products considering intent can have a tremendous impact.
5. Loss aversion: losses have more than twice the psychological impacts of equivalent gains. This is one of the biases that can come from prospect theory. Visitors will tend to give more weight to small probabilities to guard against losses.
6. Endowment effect the feeling of ownership you have once you’ve been given something. Promotions and incentives can help move visitors through to the next stage.

The Balancing Act

Too much of choice is a bad thing, but it’s not bad all the time in some cases.

Image source: Superheroes

Increasingly the studies are proving that more options always lead to fewer actions and conversely fewer options lead to more actions being taken.

But it has to be about finding the middle path. We can take away a lot of knowledge from the above-mentioned studies to understand how this concept works the best for conversion rate optimization.

• You can eliminate too much of choice on the webpage by providing just one clear call to action button, which calls for just one desired action from the user.
• Making the outreach personal and relevant, and by not sending emails with too much information.
• By reducing the number of products or services that are displayed per page.
• By using some of the psychological tactics to help decipher through the abundant choices.

According to Barry Schwartz:

“Does choice overload always occur? Of course not. Does it affect all people, in all domains of decision making? Of course not. Does it matter how options are organized and arrayed? By all means, yes. Does adding options improve decision making by making salient features of alternatives that might otherwise be ignored? Sometimes, yes. But sometimes it has a perverse effect, by making salient features of options that ought to be ignored.”

While talking about simplicity over abundance of choice and while explaining why simplicity is better, let me briefly tell you about decision fatigue, and why it affects the human psychology in terms of making choices.

Image source: Intech

Decision fatigue is when we start taking shortcuts as we exhaust our mental energy. Research suggests that humans have a limit on how many active, deliberate decisions they can make in a certain time period. The more we exhaust that reserve, the more likely we are to look for shortcuts.

There was a study conducted to prove this point. Researchers analyzed more than 1,100 decisions made by court judges. They found that as the day went by (and judges made more decisions), judges were becoming more likely to take shortcuts when deciding which prisoners to release before their official sentence was over.

Result?

Prisoners who appeared at the start of the day received parole about 65% of the time. Those who appeared late in the day received parole less than 10% of the time. Put simply, as judges got tired they used the least-risky option and allowed only a small number of prisoners out.

Conclusion

Ultimately If people have a strong preference for a particular product, the choice won’t overwhelm them. They will simply choose their favorite option.

Human beings have limited energies for processing information and more often than not, choose the path of minimum effort, even though an alternative path would result in better results. Users often take shortcuts and may appear disinterested, but their actions are ways of protecting themselves from information overload and fatigue.

All decisions people make require mental effort. As technological advancements proliferate, so does the amount of work required to make good choices. Every decision, large or small, costs us time and effort.

And in the end, as Mr. Scheibehenne said:

“It is not clear that more choice gives you more freedom. It could decrease our freedom if we spend so much time trying to make choices.”

The post The Paradox of Choice in Web Design: Simplicity vs. Abundance of Choice appeared first on Invesp.

• There are hundreds of case studies and examples of A/B testing. While A/B testing is important, it’s just a small fraction of the overall CRO process.
• AB testing isn’t foolproof and like anything in statistics, results can be inaccurate. But the more you know about what makes a test valid, and basic statistical concepts, the more likely it is that you will not face errors.
•  Validity threats are risks associated with certain uncontrollable or ‘little-known’ factors that can lead to inaccuracy in results and render inaccurate A/B test outputs and they’re categorized as type 1 and type 2 errors.
• A null hypothesis is an assumption stating that there is absolutely no relation between two datasets. Hypothesis testing is done to either prove or disprove if an assumption is correct or wrong.
• In statistics, type 1 error is said to occur when a true null hypothesis is rejected, which is also called a ‘false positive’ occurrence. Results might indicate that Variation B is better than Variation A as B is giving you more conversions, but there might be a type 1 error causing this conclusion.
• In statistics, type 2 errors or a false negative occur when a false null hypothesis is retained or accepted. Or, in other words, when a test is inconclusive when in reality it is conclusive. 
• Flicker effect: this occurs when original content flashes for a brief time before the variation gets loaded onto the visitors’ screens. This leads to visitors getting confused about content, and can result in conversions dropping.
• History effect: this happens when an extraneous variable is introduced while a test is running, and leads to a skewing of results. It happens because an AB test is unlike a lab test and does not run in isolation. Therefore, AB tests are prone to be affected by external variables and real-world factors.
• Instrumentation effect: these are errors related to your testing tool and code implementations. It happens when the tool you’re using is faulty or the implemented the wrong code.
• Selection effect: this bias or error occurs because the sample is not representative of your entire audience. One of the reasons why selection error happens is because of sample bias. Marketers conducting experiments get attached to the hypothesis that they have constructed. Everyone wants their hypothesis to win. So, it is easy to select a certain sample for testing and eliminate factors or variants that might result in their hypothesis being incorrect.
• Novelty effect: error or changes in test results that are an outcome of introducing something unusual or new that the visitor is not used to. The novelty effect happens because something new is fed to visitors.
• Statistical regression: this happens when you end a test too early. This leads to data being evened out over the time period. Most people end the test when a 90% significance level is reached, without reaching the required sample size. You cannot be sure of the AB test results only by reaching 90% significance. You must be able to reach the required sample size as well.
• Simpson’s paradox: this happens because of changing the traffic splits for variants while the test is going on. It occurs when a trend that was being observed in different sets of data disappears by combining those groups.

Here’s A Longer And More Detailed Version Of The Article.

There are hundreds of case studies and examples on AB testing, explaining what makes it highly useful for conversion optimization. While AB testing is important, it is a mere component of the overall CRO process. What’s critical to keep in mind is not dive into AB testing right away until you know everything about interpreting, analyzing, and understanding test results.

AB testing isn’t foolproof and like anything in statistics, results can simply be wrong. But the more you know about what makes a test valid, and basic statistical concepts, the more likely it is that you will not face errors. This is where validity threats because of an important topic of discussion. If left unidentified or unaccounted for, they can lead you to take the wrong decision.

What are validity threats, you ask? In simple words, validity threats are risks associated with certain uncontrollable or ‘little-known’ factors that can lead to inaccuracy in results and render inaccurate AB test outputs. Broadly speaking, validity threats can be categorized as type 1 and type 2 errors. But before we define these errors, let’s understand what a null hypothesis is.

Null hypothesis: An assumption stating that there is absolutely no relation between two datasets. Hypothesis testing is done, to either prove that is an assumption is wrong or correct.

Gif source: Giphy

Type 1 errors: In statistics, type 1 error is said to occur when a true null hypothesis is rejected, which is also called ‘false positive’ occurrence. Results might indicate that Variation B is better than Variation A as B is giving you more conversions, but there might be a type 1 error causing this conclusion. Such errors are said to occur when a test is declared as conclusive although it is inconclusive. In every test, there is some amount of probability of false positives or incorrect conclusions.

Hypothetical case study explaining type 1 errors:

You have a SaaS product and you believe that changing the CTA ‘free trial’ from fixed to floating will get you higher free trial subscriptions. Variation A will have fixed CTA and Variation B a floating free trial CTA.

You launch the test and check results within 48 hours. Results point out that Variation B is giving 2% higher conversions with 90% confidence. You declare Variation B as the winner. A week passes by and you see that conversions are starting to show a downward trend. What went wrong? Probably you checked the results too early or maybe you did not set the right confidence interval.

Type 2 errors:  In statistics, type 2 errors or a false negative occurs when a false null hypothesis is retained or accepted.Or, in other words, when a test as inconclusive when in reality it is conclusive. As opposed to type 1 error, type 2 error occurs when evidence shows that Variation A is either performing better or just like Variation B.

Hypothetical case study explaining type 2 errors:

For example, your hypothesis is that introducing the option ‘Pay by PayPal’ is likely to improve purchases.

Version A (Control): Does not have PayPal payment option

Version B: Includes the option to ‘Pay by Paypal’ on checkout

Test results show that Version A wins and the option to ‘Pay by PayPal’ did not have any effect on final conversions. However, in reality, this might have happened because maybe your sample size was not appropriate. If you had increased the sample size, you might have eliminated this type 2 error or false negative.

Types of Validity Threats and How to Minimize Them

Having understood the two common categorizations of validity threats, let’s study in detail the common threats to the validity of AB tests. Also, let’s get to know how to minimize them.

1. Flicker Effect

Gif source: Kameleoon

What is it: Occurs when original content flashes for a brief time before the variation gets loaded onto the visitors’ screens. This leads to visitors getting confused about content, and can result in conversions dropping. Any disturbance on the website and slow loading can put visitors off. If visitors see two different content being shown to them during the website loads, it is likely to make them suspicious and withdraw.

Why it happens: Flickering might happen because of slow website loading speed, the code being added incorrectly to the webpage being tested, because of code being asynchronous, or because too many scripts are being loaded before the test script. There could be some other reasons as well, other than what we’ve mentioned, which lead to flicker effect.

Example: For example, on your variations being testing, you have applied the testing code to the bottom of the test variation rather than to the header. This can cause flickering as the browser will execute the code only in the end and not as soon as the visitor lands on the variation. And, as a result, the visitor will first see the original content on his screen.

How to minimize it: Optimizing your website speed will help reduce flicker. You should also be careful while implementing the code. Another thing to ensure is that the testing script is removed from the tag manager or that it is set to a synchronous code.

2. History Effect

What is it: History effect happens when an extraneous variable is introduced while a test is running, and leads to skewing of results.

Why it happens: It happens because an AB test is unlike a lab test and does not run in isolation. Therefore, AB tests are prone to be affected by external variables and real-world factors.

Example: You are running a marketing campaign utilizing a landing page from your website and you are running an AB test on that landing page. You might see a spike in the traffic on your landing page due to the marketing campaign that you are running on it. This might lead to increased sign-ups on that landing page as well. Now, it might lead you to conclude that the original landing page is better while in reality Variation B might have won in case you wouldn’t be running the marketing campaign at the time. In this case, ‘the marketing campaign’ is the extraneous variable that has caused your AB test to give skewed results.

How to minimize it: The best way to tackle the history effect is to take into account any external factors that can skew results. Apart from this, it is important to let everyone in the organization know if an AB test is being run. Knowing that an AB test is being conducted will ensure that no one on the team introduces any external factors/variables during the tenure of the test, to the pages being tested. Making use of Google Analytics alongside your AB testing tool will also help you track any changes in traffic that have happened not because of the test but because of an external variable. This will save you from deploying the wrong variation and incur losses.

Case Study: This post on validity threats, by Marketing Experiments, talks about a case study where they wanted to determine which ad headline would fetch the highest click-through-rate for their subscription-based website. During the test, an external ‘real world’ event occurred that led to a significant and transient change in the traffic coming to the website. And, this resulted in the skewing of results as well.  

3. Instrumentation Effect

What is it: These errors are related to your testing tool and code implementations.

Why it happens: When the tool you are using is faulty. Or, in case you have deployed the incorrect code. It can also happen if the code is deployed incorrectly or is not compatible with browser types. Although, deploying incorrect code is not like a faulty test engine – each of those are problems and each need to be diagnosed differently.

Example/Examples: One out of the 4 variations you are testing is not running properly on Chrome. This means the chances of recording conversions on that variation are slim. If the code was compatible with Chrome, that Variation would have given a different result.

How to minimize it: A/A testing is one way that can help you determine if your tool is faulty or hasn’t been deployed correctly, or is inefficient. If this is the case then your AA test will conclude a winner even between two identical variations. The problem, however, with AA testing is that it is time-consuming. It is best to perform an AA test only if your website gets loads of traffic.  

Another way to tackle the instrumentation effect is to double check that your experimentation has been set up the right way. There is no code error, no mismatch between code and browser compatibility, and that your data is being passed correctly onto the CRM. Being vigilant and watchful for errors can help you minimize instrumentation effect in the best manner.

Raphael Paulin-Daigle recommends:

“Before launching ANY tests, you should always do rigorous Quality Assurance (QA) checks such as performing cross-browser and cross-device testing on your new variations, and trying out your variations under multiple different user scenarios.”

You must also ensure that the testing engine is compatible with Google Analytics in order to see the testing data in the GA and have that as a source of comparison.

In case, all of your measures to minimize instrumentation errors fails, it can be indicated that the testing engine is faulty.

Case Study: You can read this case study which talks about how Copyhackers ran a split test and found out there were major loopholes in their testing engine.

4. Selection Effect

What is it: The bias or error that occurs because the sample is not representative of your entire audience.

Why it happens: There are many reasons why one could end up picking a sample that does not completely or accurately represents the entire audience set. One of the reasons why selection error happens is because of sample bias. Marketers conducting experiments get attached to the hypothesis that they have constructed. Everyone wants their hypothesis to win. So, it is easy to select a certain sample for testing and eliminate factors or variants that might result in their hypothesis to be incorrect. While calculations can tell you the appropriate sample size, they are not helpful in deciding who should comprise the sample type. The idea is to keep the sample completely representative of your entire audience, free of bias.

Example: You are running ads on your website for premium hotels and your hypothesis is that number of bookings will increase by running this campaign. However, the conversions on Variation B that has the ad campaign running goes down instead of going up. Maybe your main traffic type comprises middle-income group and you haven’t considered this group in your sample. This is likely to skew your test results.

How to minimize it: Regularly study your analytics reports and keep digging deeper into the source of traffic.  Make sure that your sample that is truly representative and is free of sample pollution. Not taking into account the different types of traffic that visit and interacts with your website and comprises your sample, will cause regression to the mean. Include both new as well as returning traffic in your sample, and consider both weekday vs. weekend traffic in it.

Optimize smart summarizes the point:

“Each traffic source brings its own type of visitors, and you can’t assume that paid traffic from a few ads and one channel mirrors the behaviors, context, mindset, and needs of the totality of your usual traffic.”

Case Study: “In that case launching a winning variation may not result in any real uplift in sales/conversion rate. The launch of winning variation may, in fact, lower your conversion rate. When you’re analyzing the test results, make sure to segment by sources in order to see the real data that lies behind averages.” via SplitBase

5. Novelty Effect

Gif source: Giphy

What is it: Error or changes in test results that are an outcome of introducing something unusual or new that the visitor is not used to.

Why it happens: Novelty effect happens because of something new being fed to visitors. It occurs because of innate human behavior to prefer something new over old, such as alterations made to a landing page that visitors are not used to seeing on your website.

Example: Let’s flip back to online. You introduce slider images for your apparels section, in Variation B. In Variation A, you have one image for the apparel. Your hypothesis is that the version with slider images will fetch you more conversions. Even though your hypothesis wins, it might be that because of the new change is attention-grabbing, and results in the novelty effect kicking in, the conversion has seen a temporary spike.

How to minimize it: Your old set of the audience might behave differently just because they have been exposed to something new. Conversions, in this case, are simply likely to spike not because one version is better than the other, but because there is something different that the audience is getting to see. The best way to eliminate this bias is to try driving new traffic to your website while introducing something new and AB testing it.

Case Study: Let’s look at  FC Dallas stadium case study by Clive Thompson

“In 2005, FC Dallas — the city’s pro soccer team — moved into a new, state-of-the-art $80 million stadia. Over the next two years, games drew 66% more fans, with an average of about 15,145 attending each game. Over the next few years, though, as the novelty of the stadium diminished, some of those new fans began drifting away, and average attendance slid to 12,440.”

6. Statistical Regression

What is it: Regression towards mean, also called statistical regression happens when you end a test too early. This leads to data being evened out over the time period.

According to Ted Vrountas:

“If you’re making business decisions based on your A/B tests just because they reached statistical significance, stop now. You need to reach statistical significance before you can make any inferences based on your results, but that’s not all you need. You also have to run a valid test.”

Why it happens: Most people end the test when a 90% significance level is reached, without reaching the required sample size. You cannot be sure of the AB test results only by reaching 90% significance. You must be able to reach the required sample size as well. Otherwise, the results might actually just be imaginary. If, for example, the required sample size for your test is 50 and you end the test at the sample size 20 because you’ve reached 90% significance, your test results are skewed. There are a number of AB testing sample size calculators that can help you find out the required sample size for your AB test.

Example: You create a new landing page for your SaaS product and the all of your first 15 visitors convert on the new variation. This means that there is a 100% conversion rate. Does that mean that the new landing page is far better than the older one? No. First 15 visitors do not denote the full sample size. Your sample size calculator tells you that your minimum sample size is 50. You cannot conclude variation as winner having reached a sample size of 15.

How to minimize it: Do not stop your AB test when you reach statistical significance. You need to collect as much data as possible, which in turn will lead to higher accuracy in your test results. Reaching the required sample size is the key to eliminating statistical regression errors.

You must also calculate the sample size prior to running the test so that you can ensure meeting that number of conversions before the conclusion.

7. Simpson’s Paradox

Gif source: Giphy

What is it: Errors that happen because of changing the traffic splits for variants while the test is going on.

Why it happens: It occurs when a trend that was being observed in different sets of data disappears by combining those groups. This happens because when calculating test results, weighted averages are taken into account. Simpson’s paradox can also occur when an alteration to traffic distribution for a variation is done manually. For example, you see that Variation B is a winning variant, so you change traffic distribution to that variant. Another reason that it happens is that members of the population leave or join when a test is going on.

Example: To understand the point on weighted averages and Simpson’s paradox, let’s compare conversions and test results for control and variation, in the University of California example that Josh Baker explains:

“In 1973, the University of California at Berkeley was sued for showing bias in admissions for women to their graduate school. Men had a much better chance to be admitted than women according to the statistics given. “

The subgroups combined as below, showing men more likely to be admitted:

“But according to individual department numbers, it showed that there was a small but statistically significant bias that favored the women in actually having a higher chance of being admitted.”

The above chart is a good example of Simpson’s paradox and depicts how women results are performing better than men results when divided by departments.

How to minimize it: Rather than opting for what total numbers tell you, we should be ensured before start to testing, the groups are similar as far as possible.

Gordon S. Linoff explains:

“Simpson’s Paradox arises when we are taking weighted averages of evidence from different groups. Different weightings can produce very different, even counter-intuitive results. The results become much less paradoxical when we see the actual counts rather than just the percentages.”

Georgi Georgiev recommends:

“We should threat each source/page couple as a separate test variation and perform some additional testing  until we reach the desired statistically significant result for each pair (currently we do not have significant results pair-wise).”

Let’s look at another example.

Example: To understand the point on weighted averages and Simpson’s paradox, let’s compare conversions and test results for control and variation, in the following hypothetical example.

The above chart is the perfect exemplification of Simpson’s paradox and depicts how Variation A is performing better than Variation B when divided by traffic sources.

How to minimize it: Rather than opting for what aggregates tell you, dig a little deeper into a segment-wise performance of your variations. Maybe, you would like to retail Variation B for traffic source 3 as it is performing equally well as Variation A is for the same traffic source. Maybe, deploying Variation A for traffic source 2 is a good idea. Such insights can help you improve your decision making and infer better out of AB testing.

Conclusion

AB tests aren’t free of bias and there are a number of factors that can skew the results that you obtain from AB testing. However, if you are aware of validity threats – type 1 and type 2, which we have discussed in this post, you can stay vigilant, take into account the scope of error, and wisely interpret test results. We’d love to know if you ever ran an AB test and encountered a validity threat. Share with us your experience and learnings in comments below. Feedback is welcome!

The post Validity Threats to Your AB Test and How to Minimize Them appeared first on Invesp.

Disclaimer: This section is a TL;DR of the main article and it’s for you if you’re not interested in reading the whole article. On the other hand, if you want to read the full blog, just scroll down and you’ll see the introduction.

• A conversion optimization project is typically divided into four different phases that intertwine:
• As you go through the process of identifying conversion issues on your website (scrutinize), you can easily identify anywhere from 150 to 250 possible test scenarios.
• The scrutinize phase of our SHIP process contains several activities designed to identify possible conversion blockers on a platform:
• As you conduct your analysis, there are four different types of issues you will identify; 1, research opportunities, 2. Issues to investigate further, 3. Instrumentation and 4. Issues to fix right away.

Things To Look Out For During A Conversion Rate Audit.

• Website bugs: arrears where the website is broken and not working as expected.
• Website usability issues: issues where website usability is not ideal.
• Conversion blockers: issues related to getting customers to convert

Understanding your initial AB test plans: the research opportunities list

• It is not enough to identify research opportunities list for a website. You will need to go a step further and determine our initial test hypothesis for each item in the list.
• For us at Invesp, we have the initial hypothesis and the concrete hypothesis.
• The initial hypothesis is the first stab our team takes on how we should address a potential problem on the website/webpage.
• The concrete hypothesis is developed from further research (more on this in the article)

How To Prioritize Your A/B Test Ideas

• CRO practitioners have used several frameworks to classify testing ideas; 
• PIE by Widerfunnel: evaluates items based on three different factors: potential, importance, and effort.
• The Hotwire framework: Pauline Marol, the Lead Product Manager for optimization at Hotwire introduced this model back in 2015.
•  The PXL framework by CXL: it uses 7 criteria to score items on your research opportunities list (more on this in the article).
• At Invesp, we have our own prioritization framework that contains 18 criterias for your conversion research.

Here’s A Longer And More Detailed Version Of The Article

A conversion optimization project is typically divided into four different phases that intertwine:

• Scrutinize stage: You conduct various activities to determine possible testing areas on a platform or campaign
• Hypothesis: you create a hypothesis for the different testing ideas identified during the scrutinize phase
• Implement: you validate your hypothesis through the process of AB or usability testing
• Propagate: you look for insights, share them within the organization, across marketing channels and verticals

As you go through the process of identifying conversion issues on your website (scrutinize), you can easily identify anywhere from 150 to 250 possible test scenarios. Shana Rusonis states:

“Prioritization is the million-dollar question in optimization programs. It gets at the heart of our most frequently asked questions: “What should I test?”

The next logical question to ask is how do you prioritize the testing ideas?

Before we get into the framework we use to prioritize the results of the scrutinize phase; it will be valuable to provide insights on how we structure some of the processes to prioritize our final list.

What do you look for when optimizing for conversion?

The scrutinize phase contains several activates designed to identify possible conversion blockers on a platform:

1. Identifying visitor experience/personas
2. Conducting expert reviews
3. Conducting quantitative research
4. Conducting qualitative research
5. Conducting competitive analysis
6. Conducting usability testing

As you conduct your analysis, you will identify four different types of issues:

1. Research opportunity: These are items you want to implement on your platform using split testing or usability testing.
2. Investigate further: You identified a problem, but you are not sure why visitors behave a certain way. You will need to conduct further research to understand visitor behavior.
3. Instrumentation: missing analytics tracking
4. Fix Right away (stop the bleeding report): bugs identified on the website that must be fixed right away.

You should immediately implement Items that you classified as “Instrumentation” and “fix right away” issues.

Your “investigate further” list requires further research and will typically turn into research opportunities and fix right away items.

Your prioritization should be focused on the research opportunities list. But you are not ready yet to do that.

Bugs, usability issues, and conversion blockers

I will take a quick detour to talk about different items we uncover as we conduct a conversion rate audit for a website.

1. Website bugs: areas where the website is broken and not functioning as expected. You should classify the bugs you identified in terms of priority (P1: blockers to P4: small items). Bugs are typically put into the “fix it right away” bucket.
2. Website usability issues: issues where website usability is not ideal. Usability issues are typically classified as either “fix it right away” or “research opportunities” buckets.
3. Conversion blockers: issues that relate to persuading visitors to convert. Conversion issues are typically classified as “research opportunities” bucket.

Many conversion consultants focus their effort on identify bugs and usability issues on a website and rarely pay close attention to conversion issues. Without getting into a philosophical debate, I firmly believe that is a vast difference between creating a user-friendly website and a highly converting site.

Usability issues:

1. Fixing usability issue on a website creates a user-friendly website.
2. Usability issues focus on making the site easier to use
3. Usability issues focus on top of mind issues for website users

Conversion issues:

1. Fixing conversion issues on a website creates a user-friendly website (Similar to fixing usability issues)
2. Conversion issues focus on making the site more persuasive
3. Conversion issues focus on psychological items that persuade visitors to convert

Every usability issue is a conversion issue but not the other way around.

Beyond problems – adding new website features

CROs focus a lot on issues that stop visitors from converting on a website. However, your research during the scrutinize phase should expand beyond that to include possible new features that you can implement on the platform to further enhance conversions.

Understanding your initial AB test plans: the research opportunities list

It is not enough to identify research opportunities list for a website. You will need to go a step further and determine our initial test hypothesis for each item in the list. The initial test hypothesis identifies how we think the particular issue could be tested/fixed.

A hypothesis is a predictive statement about a possible change on the page and its impact on your conversion rate.

Now, notice how I used the term “initial hypothesis.”

An initial hypothesis is the first stab our team takes on how we should address a potential problem on the website/webpage.

Example of initial hypothesis:

“Adding social proof will enhance the visitor trust in the website and increase conversions”

We use the initial hypothesis to help us assess each line item on our research opportunities list. When we decide to implement a particular test, we further develop our initial hypothesis to a

“concrete hypothesis.” Let’s take the initial hypothesis above and see the concrete hypothesis for it:

Based on qualitative data collected from online polling, we observed that website users do not trust the brand and are unaware of how many users are using it. Adding social proof on the homepage will increase visitors trust and improve conversion rates by 10%.

A concrete hypothesis

• states how we identified the issue
• indicates the problem identified on the page
• states the potential impact of making the fix (we set an actual numerical goal for the test)

Now, that you have your research opportunities list and each item has an initial hypothesis attached to it, we are ready to go through the exercise of prioritization items on our research opportunities list.

Prioritizing your AB testing ideas (the research opportunities list)

CRO practitioners have used several frameworks to classify testing ideas. I will cover some of the common frameworks and suggest a new model use.

Karl E. Wiegers explains:

 “When setting priorities, you need to balance the business benefit that each function provides against its cost and any implications it has for the product’s architectural foundation future evolution.”

Pie by Widerfunnel

Pie framework evaluates items based on three different factors: potential, importance, and effort.

1. Potential: Not all of your testing ideas will present the same potential in terms of increasing conversion rates. A good example of this is testing different color of a CTA on a page (no real impact to little effects) compared to changing page layout to emphasize value proposition.

While the potential is significant, it is very subjective. What I might think will have a significant impact on bottom line, someone else might think it will have no effect.

2. Importance: How important is the page you are optimizing?

The Widerfunnel team defines Importance as follows:

“How valuable is the traffic to the pages? Your most important pages are the ones with the highest volume and the costliest traffic. You may have identified pages that perform terribly, but if they don’t have a significant volume of expensive traffic, they aren’t testing priorities.”

According to Chris Goward:

“How do I figure out where to test first?”It’s a smart question to ask. You can’t test all pages at once. With limited time and resources to commit and, most importantly, limited traffic to allocate to each test, test prioritization is an important part of your conversion optimization plan.”

3. Ease: How easy is it to implement the test? Widerfunnel defines ease as:

There are two elements to consider when it comes to ease:

• Technical implementation
• Organizational or political barriers”

Both of these elements impact how easy it is to implement a testing idea, albeit, they are somewhat subjective.

Hotwire framework

Pauline Marol, the Lead Product Manager for optimization at Hotwire introduced this model back in 2015.

Here are the elements in this framework:

• Main Metric
• Location
• Fold
• Targeting
• New Information
• Benchmarking
• Conversion Veins
• Strategic Topic
• Mobile
• Opaque

What I like most about this model is the specificity it provides the potential problem and potential fix. It also brings the idea of how strategic a test to the overall business objective for a business. According to Pauline Marol: 

“Prioritization is a critical part when building an efficient, impactful testing program. Creating a framework to make smarter choices and thinking deeply about the key factors will make your program successful in the long term.”

PXL framework from CXL

The PXL framework uses the following criteria to score items on your research opportunities list:

• Is the change above the fold? Changes above the fold are noticed by more people, thus increasing the likelihood of the test having an impact
• Is the change noticeable in under 5 seconds? How fast does a user notice the change?
• Does it add or remove anything? The idea is that “removing distractions” or “adding key information: will have more impact
• Does the test run on high traffic pages? Changes to “high traffic page(s)” will have a larger impact on the bottom line.
• Designed to increase user motivation: does the introduction of your solution increase visitor motivation? Highly motivated visitors are more likely to convert
• How was the problem identified?
  • User testing
  • Quantitative data
  • Qualitative data (polls, surveys, etc.).
  • Mouse tracking heat maps or eye tracking
• Ease of implementation

The PXL framework is a step in the right direction.  Peep listed three main advantages to using it:

1. It makes any “potential” or “impact” rating more objective
2. It helps to foster a data-informed culture
3. It makes “ease of implementation” rating more objective

The central theme is the focus on trying to establish a more accurate model to score different line items on your list.

I believe that there is room for improvement for the following reasons:

1. All of the prioritization frameworks use a mix of points to score the identified issue and the potential fix. But a single problem could have multiple possible solutions to it. And each potential solution should be assessed with a different score.
2. A potential problem could be identified in various ways which give it a higher score. But does identifying a problem in 4 ways make that problem a more significant issue compared to a problem identified in 3 different ways? At some point, adding more ways to how a problem is detected should not give the problem more weight.
3. There is not enough focus on how a potential solution could impact visitors view of the website. PXL does an excellent job of using motivation, but there are other criteria that we should consider when scoring a possible solution.
4. The models do not distinguish enough between the different types of pages (top of the funnel, mid of the of the funnel and bottom of the funnel
5. Finally, the models use “high traffic pages” as a way to score a problem. Since analytics gives a specific number to identify what is a high traffic page, we should use that.

Invesp Prioritization framework

Let me state this again; a particular problem can have multiple initial hypotheses of how we can fix it. Some of our team members like to call these as testing ideas. This is important because each problem and each initial hypothesis are called out as separate line items in our prioritization framework.

You can address a single research opportunity in different ways that require different time to implement and will have a different impact on the bottom line.

Next is the criteria we use to evaluate each research opportunity:

1. What is the potential Impact?
2. Is the problem above the fold?
3. Was the problem discovered through expert review?
4. Was the problem identified through qualitative research?
5. Was the problem identified through user testing?
6. Was the problem identified through analytics?
7. Was the problem identified through heatmaps/Video recording?
8. Known top competitor feature?
9. Does the test idea enhance primary conversion goal?
10. Is implementing the change easy (politically, LOE)?
11. Adding or removing an element?
12. Does the test idea increase visitor trust?
13. Does the test idea reduce FUDs?
14. Does the test idea incentivize visitors to act right away?
15. Does the test idea increase engagement?
16. What type of change?
17. What type of page?
18. The percentage of page views

Not all of these items carry the same weight.

Let’s discuss some of these elements:

1. Potential Impact: we do think that this is a subjective measure. However, we did not delete it. The value of the impact is high (score of 4), mid-impact (score of 2) and low impact (score of 1).
2. Is the problem above the fold: where did you uncover the problem? The higher the location of the problem on a page, the more likely visitors will notice it.
3. How did you discover the problem: most CROs will conduct a mix of activities to uncover issues on a site. Each of these activities should be called out. These are
4. Expert review (some might refer to it as heuristic analysis),
5. Analytics data
6. Heatmap/session
7. Qualitative data
8. User testing

Most conversion blockers are uncovered through a combination of the above elements. So, we decided to take a weighted average as follows:

If an item discovered through one method                     score of 3

If an item identified through two methods                      score of 6

If an item identified through three methods                    score of 9

If an item identified through four or five methods        score of 12

4. Known competitive feature: there are instances where you want to test an item because you observed that one of your competitors have implemented. By no means should you understand this as a way to copy your competitors, but you should not ignore them either.
5. Does the test idea enhance primary conversion goal: what is the primary conversion goal for the page? Is your testing idea going to increase that conversion goal?
6. Is implementing the change easy (politically, LOE)?
7. Adding or removing an element: Your test can do several things at the same time:

Add or remove an element: scored at 3

Replace an element: scored at 2

Change the location of an element: scored at 1

Emphasize an element: scored at 1

8. Evaluation of visitor-centric factors that would impact how a visitor perceives a page:

• Does the test idea increase visitor trust?
• Does the test idea reduce FUDs?
• Does the test idea incentivize visitors to act right away?
• Does the test idea increase engagement?

Two or more elements gives you a score of 6. One gives you a score of 3.

9. Type of change: This focuses on the testing idea at hand. Are you:

• Focused on an element level (changing a single element) scored at 1
• Focused on page level (changing layout or several elements on the page) scored at 2
• Focused on visitor flow around the website scored at 3

10. Type of page: How is the page classified in your funnel?

• Top of funnel page scored at 1
• Mid-funnel page scored at 2
• Bottom of the funnel page scored at 5
• Global Change scored at 5

11. % of page views: By looking at the % of unique page views the page gets compared to the overall website page views, you can select one of the following values:

Is there a perfect prioritization framework?

I do not think there is a perfect framework out there. We continue to tweak this framework as we work on different projects. On a recent project, our team identified close to 200 research opportunities on a project. We ran it through a previous version of the framework and looked at the results. Some of the top items confused us. It did not make sense that we should start with them. We went back and reworked some of the formulas. I am sure that someone can take the same framework and tweak it further. According to Chris Goward:

“There are no standard rules for which pages are best to prioritize. Your website lives in a unique target market, including factors like your competition, seasonality, and internal cultural environment; all of these affects how your site is used and should be optimized. The priority rating you give each of your potential test pages will depend on this unique business environment.

It is a step in the right direction. We tried to build on some of the already existing frameworks out there and add more elements that impact prioritization on our different projects.

You can download the excel for the prioritization framework from here.

As always, if you have suggestions, please do let’s know!

Frequently Asked Questions

What is Conversion Rate Optimization (CRO)?

• Answer: Conversion Rate Optimization, or CRO, is the practice of increasing the percentage of website visitors who take a desired action, such as making a purchase or filling out a form. It involves analyzing user behavior and implementing changes to improve the user experience and boost conversions.

What is a Conversion Funnel?

• Answer: A conversion funnel is a series of steps that a website visitor goes through before completing a desired action. It usually starts with attracting the visitor to the site and ends with them taking an action like making a purchase.

What is A/B Testing?

• Answer: A/B testing is a method where you compare two versions of a webpage or app against each other to determine which one performs better in terms of conversions.

How Do I Prioritize My A/B Testing Ideas?

• Answer: Prioritizing A/B testing ideas involves evaluating the potential impact, ease of implementation, and relevance of each test. Frameworks like PIE (Potential, Importance, Ease) or the Invesp Prioritization Framework can help in this process.

What are FUDs in CRO?

• Answer: FUDs stand for Fears, Uncertainties, and Doubts. These are psychological barriers that can prevent a visitor from converting. Identifying and addressing FUDs can significantly improve your conversion rate.

What Does ‘Above the Fold’ Mean?

• Answer: ‘Above the fold’ refers to the portion of a webpage that is visible without scrolling. Elements placed here are more likely to be seen by visitors and therefore have a higher impact on conversions.

How Long Does It Take to See Results from CRO?

• Answer: The time it takes to see results from CRO can vary widely depending on the changes made and the metrics you’re tracking. However, a good rule of thumb is to allow at least a few weeks for meaningful patterns to emerge.

The post Creating A Conversion Roadmap: How to Prioritize Conversion Problems on Your Website appeared first on Invesp.

Elizabeth Wellington explains;

“Customers give information that reveals their attitudes (how they think they should behave) rather than behaviors (how they actually make decisions)”

But conversions and the path that leads to those conversions is highly driven by the customer. The image of a funnel is what comes to mind when many think of conversions, however, because the customer is irrational and highly distracted, it looks more like this:

There are ways of conducting qualitative research to gather customer data and understand more about what they are thinking, what is motivating them.

How To Determine When To Do A Qualitative Research

• Little is known or present understanding is inadequate
• Making sense of complex situations or social processes
• Learn from participants about their experiences (beliefs, motivations, opinions)
• Construct a hypothesis/theory from data
• Understand phenomena deeply and in detail

You’re essentially putting an organized framework on the messiness of life:

“Chaos often breeds life, when order breeds habit.”  Henry Adams

But one must be wearied with some of the results because surveys and polls aren’t always accurate.  Results of those qualitative studies aren’t always accurate. There is a science in the way that you ask and how you ask it (the method to use).

Susan Farrell explains;

“Write neutral questions that don’t imply particular answers or give away your expectations.” Keep it short. Every extra question reduces your response rate, decreases validity, and makes all your results suspect.”

Qualitative research participants often answer something, but there subconscious and actions do something entirely different. Many studies have been conducted on how faulty relying on qualitative data can be. Participants tend to say the first thing that comes to mind or what they think they want. But you dig deeper to realize that there’s a lot more than you must be uncovered, and what they think they want or why they think they choose is a product or service isn’t the true motivation.

So although it’s important to assume the majority of qualitative research participants aren’t maliciously telling untruths, it happens often enough to make you really think about why it’s happening and what you can do about it.

It’s also very possible that they simply don’t remember an answer. If you are asking a question about whether or not someone will buy from you again in the future (or subscribe to your service), you are likely to get a positive response that doesn’t mirror the true future.

These all point to one overwhelmingly clear pattern: a clearly poorly designed qualitative research method. Some questions shouldn’t be asked in that specific format, or the way the question is asked almost forces a specific kind of response, leading to a greater change of untruths. Growing up in the 1990’s and early 2000s, I had to fill plenty of forms and always got asked the question of my ethnicity. I could never figure out where I belonged exactly – in what group. It felt limiting and likely lead me to select what felt right but wasn’t actually right.

So let’s break it down – why do participants lie and what can you do about it?

1. Participants care about appearances: When you ask for demographical details, income details, employment details, people hate appearing like they’re worse off than others.
2. Participants want to stay socially viable: If there is something socially unacceptable, people will tend to tell untruths about the reality of their situation. Think about a topic like voting – it’s socially undesirable to be a citizen that simply does not vote or participate.
3. Participants are sensitive too: Social and personal questions, including illegal behaviors, may lead participants to tell untruths about realities they deal with.
4. Participants aren’t malicious and want to “help:” If they think that’s the answer the research wants to hear, they will give the answer they believe the research “needs” in order to please or help them.
5. Participants are human which means greedy: If there is something in it for them, or they think there is, they will shape the answer to meet their needs.

Ok, this is a little disheartening, but qualitative research is still quite invaluable. The key is how to use it, using what method, how to conduct it and analyze it. But first…

The basics: what is qualitative research?

Image Source: qualitativeresearch

Elizabeth Wellington describes it well:

“In his book “How Customers Think: Essential Insights into the Mind of the Market,” Zaltman characterizes this part of the mind as a “wonderful, if messy, stew of memories, emotions, thoughts, and other cognitive processes we’re not aware of or that we can’t articulate.” As customers, we make decisions from that convoluted (and unexamined) place. Without the amygdalae — the instinctual part of our brain that generates emotions — we wouldn’t be able to make any decisions at all.”

It’s a not a single approach, but rather a very broad systematic collection and organization and interpretation of textual information.

There is no one way to conduct qualitative research. It is definitely complicated. The wrong question be it open ended questions or close-ended questions  can lead to useless data. It definitely requires much organization and categorization. And finally, qualitative data requires deep interpretation and many methods of doing that, one of which is textual analysis. A well-formed open ended question is the key to good research and often leads to more research and more questions. When you consider what a good research question is, consider that it should include the following qualities:

• It is inductive, exploratory.
• It’s impartial in the statement, and there are no underlying hints given
• Careful focusing on a single idea, don’t overdo it by asking too much

How To Conduct Qualitative Research

Image Source: qualitativeresearchprocess

1.    Set a goal

Never begin a qualitative research activity without having oriented yourself with what it is you are trying to learn and achieve. Otherwise, you will not be able to formulate the right question, nor come up with the correct mechanism and/or setting to conduct your research.

If you know your goal, and you are familiar with the different methods of conducting qualitative research and the application of those methods; which will be covered later in this blog, you will have a better outcome with the research. Ultimately whatever method of research you select based on your goal, will help identify what processes must be followed for the best results.

Finally, part of your goals should include who the participants that you are targeting are. Throwing a wide net to all visitors of a website is less meaningful then selecting a segment depending on the question(s) being asked.

2.    Consider the Outcomes

There are few possible products or outcomes that can come out of qualitative research. The outcomes are highly dependent on the type of qualitative research you conduct as well. Some of the outcomes you can expect include:

• Recurrent themes and hypotheses
• Survey instrument measures
• Taxonomies
• Conceptual models (theories)

“Not everything that can be counted counts; not everything that counts can be counted.”  Albert Einstein

One thing to understand about qualitative research is it’s almost never about the numbers, but rather the overall meaning. You’re essentially capturing a specific aspect of social or psychological life.

According to Priscilla Esser and Ditte Mortensen:

“When you’re sharing results from qualitative user research efforts, you’re most likely focusing on creating an understanding for the lives people lead, the tasks that they need to fulfill, and the interactions they must effect so as to achieve what they need or want to do.”

So, what does that mean for you? Well for one, there are some important considerations to keep in mind, which will better help you interpret the data:

Don’t expect a single answer.  You will always get a multitude of answers that you must decipher.  Looking at the data, don’t look at it from a quantitative analysis perspective because you’re looking for quantitative data, because it’s not like that. You’re merely telling a story. Your story may not be entirely true; however, it MUST be: plausible, and coherent, and if you want some truth behind it and a stronger case: quantitative data to back it up.

3.    Know the context to better understand the responses

A key area to think about when it comes to qualitative data is that it’s relative.

“Truth can be compelling without being absolute.” Remember ultimately that many truths are relative, especially considering qualitative data.

So, you need to understand and think about context when you conduct and analyze the data coming from qualitative research.

Context includes:

Source of traffic – understand the journey that brought them to your site will help you understand why they answer what they answer. A great place to start is by analyzing the sources of traffic that brought these visitors to your site in the first place. That way you can understand their journey that much better.

Bias – There’s no way around it, qualitative data is tainted with bias. While quantitative data enjoys an absolute truth because numbers are numbers, qualitative data is quite the opposite. Trying to gauge human bias in your sample group is important for context.

Subjectivity – we always say as a golden rule to optimization is that you want to get from subjectivity to objectivity and data-driven changes. However, it’s important to recognize that qualitative data is quite subjective during the analysis process. That’s why you want to try and pain some of that qual data with analytics or other qual data to provide objectivity to a particular issue you uncovered.

4.    Eliminate Researcher Bias

You’re biased. Everyone is. As a researcher, however, this bias can stand in the way of asking the right questions and seeing other perspectives. So, our recommendation is to have some sort of a questionnaire or checklist before going into the research to ensure that you practice good reflexivity:

• Consider 5 factors how participants will view you (presumptions)? Because you are the company asking, how will that affect the answers they give?
• Then consider the 5 factors that impact the way you analyze data (as a researcher). Consider what assumptions you, as the researcher, have about life, people, website or business, and how those factors may shape your reading of the data.

That brings us to two important terms that everyone who conducts qualitative research should know: Ontology and Epistemology.

Ontology refers to what is nature of the research vs. the reality. Ultimately, the researcher’s philosophical assumptions about the nature of reality.

Epistemology refers to what is possible to know and how we can generate meaningful knowledge. So, the researcher’s assumptions on the best ways to inquire to find out the truth. Being aware of these two terms helps you understand that you too have a bias. Knowing it and understanding what will help combat it can give allow for a better chance to ensure your research will not be polluted.

Knowing these two terms will help you construct “3 fundamental questions to your research:

The ontological question: What is the form and nature of reality?

The epistemological question: What is the basic belief about knowledge (what can be known)?

The methodological question: How can the research go about finding what she believes can be known?” (Guba, Lincoln 1994)

5.    Beware of subjectivity

Ultimately, researchers often fall into the mistake of subjecting histories, assumptions, and values into research the research they conduct.  This is why as researchers we must be consciously practicing reflexivity.

Reflexivity refers to the reflective process in which a researcher considers how the findings were produced, taking into consideration the prior knowledge, and our role in it.

6.    Understand the Who?

The second most important consideration is the Who of your research. While quantitative research often samples large groups of people randomly, there is a lot more purpose and specificity to the selection of smaller samples for qualitative research. Because the samples are smaller, they must be less random.

Image source: Slideshare

Data Sampling: There are many ways of sampling data, however, for the purpose of this article, we will talk about two ways:

1. Probability sampling: This is selecting a truly random, yet statistically representative sample that can be generalized to a better understanding of the greater population (market) This is used in quantitative data collection.
2. Purposeful sampling: selecting “information-rich cases” (Patton 1990) for deeper research. From these smaller, yet sought-out samples, you can learn a great deal about issues that are central to the research. The purpose is to “select information-rich cases whose study will illuminate the question under study.” (Patton) This is used in qualitative data collection.

Determine key informants: Again, the aim is at being purposeful when it comes to sampling. Seek out, specific individuals.

Breadth not representatives: You aren’t looking for general public, yet people with the extreme circumstance that make them eligible for the study. For instance, I’m looking to understand what makes individuals loyal to my brand. The average loyal customer shops 2 – 3 times a year. I would seek out those customers who shop over 8 or 9 times a year.

Sample size

• depends on the complexity of inquiry and type of qual research (cannot be clearly determined in advance)
• determined by “theoretical saturation (a point at which no new concepts emerge from data)”

7.    Select the right Qualitative Research method

Image Source: methodsuxdesign

Qualitative research involves different formats and methods, each with a specific use and aim.

The methods include face-to-face or phone interviews, focus groups, observation (natural settings), textual (polls, surveys), listening to audio recordings of conversations between customer support and customers.

According to Ditte Mortensen:

“Qualitative methods such as interviews, ethnographic field studies, and (to some degree) usability tests are often more exploratory and seek to get a more in-depth understanding of the individual users’ or user group’s experiences, motivations, and everyday lives.”

Interviews

Interviews should be considered as “conversations with a purpose.” There are two ways to conducting interviews. Face-to-face or via phone. While in today’s digital world, people are less likely to conduct face-face, the chances of accuracy are always less likely when you can’t view the person and their expressions when they answer.

Topics and issues to be covered are specified in advance, in outline form; the interviewer decides sequence and the outline increases the comprehensiveness of the data and makes data collection somewhat systematic for each respondent. Logical gaps in Important and salient topics may be inadvertently omitted. Interviewer flexibility in sequencing and wording questions can result in the wording of questions in the course of the interview. data can be anticipated and closed. Interviews remain fairly conversational and situational. substantially different responses from different perspectives, thus reducing the comparability of responses.

Useful for:

• Individual perspectives and experiences
• Sensitive topics
• Situations where there is perceived danger of reprisal
• Topics that cannot be investigated through surveys
• Gathering in-depth information about a topic

Focus Groups

When we think focus groups, a lot of times, the picture of a group of people evaluating some sort of product that is going to be rolled out pops into mind.

According to the Marketing Research Association,

 “A focus group is the meeting of a small group of individuals who are guided through a discussion by a trained moderator (or consultant). The goal of the focus group is to get beyond superficial answers and uncover insights on consumer attitudes and behavior.”

The key behind focus groups is trying to generate narrative data in a focused discussion. The nature of this format is about the group dynamics offering a wide range of perspectives and views on a common experience. Because many people are participating, it helps to activate forgotten details. It’s also revealing in the way people speak about issues.

Useful for:

• Characterizing social and cultural norms
• Sharing and comparing
• Revealing how people talk about an issue
• Exploring sensitive topics

Observation

Observation is defined as the recording of the behavior of the sample. Many usability experts use observation and eye-tracking tools when they conduct a study online to specifically gauge more of the participant’s facial and body language. Although participants may say they weren’t frustrated by a feature or experience post the observation, their body language, and facial expressions tell a different story. It is best used when applied to visitors in their natural environments. For instance, if a company that offers teachers classroom management tool wants to understand more about how and when teachers use their product answering questions about the agility of the software, they would send a team to classrooms using the software and observe over a number of days how the teacher interacts with it.

Useful for:

• Individual perspectives and experiences
• Difficult to gauge through interview experiences
• Complimenting usability studies
• Topics that cannot be investigated through surveys
• Gathering in-depth information about a topic

Textual (polls and surveys)

The go-to for most marketers because it’s easiest and least invasive. However, with all of the positives, the responses you can be getting about certain questions may need to be taken with a big spoonful of salt. Of course, if you know how to use polls and surveys, and what type of questions work with this format, it can prove useful. What’s also key to polls and surveys is first identifying, of course, the goal, but then the segment that would best provide you with the answers that have the greatest specificity and accuracy.

Useful for:

• Specific segment perspectives and experiences
• Broad topics
• Situations where participants can feel free to criticize
• Demographical data

Keep your online surveys as short and concise     

The vast majority of the customers have a tendency to avoid long surveys. People do not have the patience nor the time to click through and answer any questions. There’s a higher likelihood that people will fill out short and clear surveys. From my own experience, if I see a question to rate an experience, for example, where I need to click through in order to get to another screen where the question is listed, I will likely opt out. Let it be easy and visible from the outset to increase response rate.

Image source: Econsultancy

Response rate

For qualitative data, the response is more heavily focused on the quality, and not the quantity. For surveys and polls, there is a possibility to tap into a greater number of participants. However, the aim should not be solely focused on the number of respondents, but rather if you were able to get quality responses from the specific segment you were after.

That being said, with some kinds of polls and surveys (especially quality or experience assurance or NPS) you need to factor quantity as well. There are several factors that determine the validity of a survey. Population size, margin of error, confidence level, sample size, and response rate.

The Response rate: Number of valid responses / total number of people invited to respond. When you make an investment about anything, you expect high ROI (return on investment) of a project right? The response rate is the ROI of a survey. You need high response rate to be sure that you are collecting valid data.

Checkout 34 ways to improve your survey response rate:

By implementing some of the above strategies, you can improve your response rates from 30% to 55%.

PeoplePulse, IBM’s innovation center, ran B2B online surveys from 2000 to 2001. After some changes in their survey structure, they managed to double their response rate. The two variables they tested on the surveys were offering incentives and shortening the survey. The incentive increased response rates 10-15%.

8.    Asking the right questions

Your questions will vary depending on the goal and format you will be conducting the qualitative research. Whether it’s interviews, ethnographic research, focus groups, or surveys, questions must be identified clearly before beginning the process.

You’ve considered your goal, thought about the biases that may pollute your reading of the responses and consider who it is you are asking: your job now is to think about the questions.

When it comes to asking research questions, a lot of bias towards the “answer you expect” can sway your approach to constructing sound, unbiased questions.

Here are some tips to consider:

Use a verb: Remember, your role is to ask them to describe, characterize, understand, consider. That line of questioning will put the ball in their court.

Identify the topic of interest: Be clear about what the actual topic is first, then ask the question, in order to give them context.

Non-directional neutral language: Remember, many times we inject our personal feelings into a question which will, in turn, create a bias in the responses.

Define the sample and setting: How will you conduct your research – what method? The question may differ based on those decisions. A statement of why you are conducting the research is helpful at this point: like the goal is to XXX.

As precise as possible: Most of the work happens in the question construction phase, if you rush and create a haphazard question, you are likely to get haphazard answers that add no value to your research.

Samples:

To explore the views of men who read XXX skin research blog

To understand XXX visitor views on XXX

To describe and classify the factors that determine motivations of visitors for xxx

To understand the knowledge, attitudes, beliefs, preferences, and barriers regarding XXX amongst XX visitors

To characterize barriers to XXX among male/female visitors, who are currently enrolled, registered, new, returning, oligarchs,

Mass communication theory has some good ideas on how to structure the questions you will be asking.

They recommend formatting it into what I would describe as almost a data tree structure.

• The Research Problem
• Goal: or qualitative purpose statement
• Determine method of research that is best suited to achieve the goals
• The research questions (that narrow the purpose):
  • Central
  • Sub-questions

To hone in on the right questions, Mass Communication Theory suggests using the following scripts to identify the relation between the question and the qualitative purpose statement:

“Central question script (use only one):

“What does it mean to _________________ (central phenomenon)?”

“How would ______________ (participants) describe (central phenomenon)?”

Sub-question script: “What _________ (aspect) does __________ (participant) engage in as a _____________ (central phenomenon)?”

Other tips for asking questions

Take a look at a chart NN Group mentions how to ask open-ended questions:

Here are the possible questions to be asked upon buying process:

• What matters to you when buying product/service?
• How long did the buying process take on average? Do you think it is short/long?
• What are we doing worse/better compared to our competitors?
• Which hesitations did you have before buying? Are there still any, please clarify?
• What made you buy from us?
• What would you recommend us to better improve our product/service for next time?

Andra Baragan states:

“When surveying your customers, you don’t want to just survey anyone and everyone. The best way to do it is to survey only recent customers because they won’t hold a biased opinion, and their experiences with you are still fresh. If possible, survey them immediately after they purchase. The faster you survey post-purchase customers, the more accurate your results will be.”

9.    Conduct Qualitative Data analysis

The part of digital marketing that always baffles me is the amount of data available for analysis but left unused. There is so much effort that goes into the creation of research, but by the time it’s said and done, marketers feel exhausted and don’t pay close attention to what the results are really saying.

The key to conducting qualitative data analysis is to ensure there is sufficient, quality data. Looking at that data, there are several ways to analyze it. You will have likely considered the analysis needed for your specific qualitative research method previous to receiving the results.

Thematic analysis: identifying themes and patterns of meaning across a dataset in relation to a research question

Grounded theory: Questions about social and/or psychological processes; focus on building theory from data.

Interpretative phenomenological analysis: Analysis aimed at seeking insights into the meanings that events and experiences hold for people.

Discourse analysis: Analysis of understanding more about how certain terms are being used. What is said and why it might be said?

Narrative analysis: Analysis based on seeking greater understanding of unique perspectives brought on by individuals. It is based on seeing how individuals make meaning using stories. How these individuals make sense of their external and internal world.

Types of qualitative research methods in UX design

1. Grounded theory. 

This systematic methodology was developed by sociologists Glaser and Strauss in the 1960s.

Grounded theory is a qualitative research method in which researchers come up with theories and hypotheses that are grounded in empirical data/observations. 

Instead of researchers developing a theory before examining data, grounded theory works the other way around. Researchers come up with theories after examining the data and finding patterns along with categories.

2. Card sorting. 

Card sorting typically presents users with a series of categories or topics written on cards and invites them to organize them in such a way that makes sense to them. Card sorting is a great way to understand what might be intuitive to users and what web or platform design structure users expect. It is also a relatively quick, easy, and cheap method to execute.

3. Diary studies.

 By getting users to log their habits over time, diary studies offer a research approach that yields insightful data that can then help businesses understand users’ daily habits, needs, and behaviors. It also meets users’ where they are—having them document their day as it’s happening instead of asking for them to recount an interaction after the fact.

4. Qualitative data analysis. 

Also known as thematic analysis, this method involves analyzing qualitative data for themes that can help answer a research question or find meaning within a data set. The results of a thematic analysis can influence design decisions by helping developers zero in on user needs.

Wrapping Up

Qualitative research is a type of research in the social sciences that focuses on non-numerical data as opposed to quantitative data that’s full of numbers.

To get the best from conducting any qualitative research, you need to focus on the quality of answers, not the number of respondents. 

Quantitative data is focused on how large the sample size is; qualitative data is focused on the quality of the information from a smaller sample size.

Also, don’t limit yourself to one type of qualitative research method; employ many, like ethnographic research methods, interviews, focus groups, textual analysis, etc.

Whoever said qualitative data was easy or required little precision or attention to detail are the same people that sold AB testing as an easy-to-use tool that any marketer can simply apply to their activities. It is precise, scientific, and driven by knowing who you are targeting, being conscious of your biases, and constructing sound problem statements, goals, and questions.

Additional Resources

1. Open-ended questions and close-ended questions: What they’re and how they affect user research.

2. The Paradox Of User Behavior In Web Design: Novel vs. Routine Tasks

3. A Guide To UX Data Analysis For Actionable Insights

The post 9 Tips to Conducting Accurate Qualitative Research appeared first on Invesp.

Businesses realize the importance of scientific experimentation and testing. Many product-based enterprises have embedded experimentation in their organizational culture. The reason – innovation cannot result without experimentation. Moreover, though there are millions of ideas that spur every minute, every hour, it is a critical business decision to calculate idea feasibility, and even more important to decide which ideas will work the best.

Experimentation and A/B testing

Anyone who has had experience in running experiments on their websites will concur with what’s written above. And they will know that an experimentation framework includes researching for data, data analysis, drawing hypothesis, testing hypothesis, reporting the findings, and giving feedback on hypothesis drawn. Testing is the next logical step.

Because testing is the secret to making sense out of data, businesses are starting to test everything – from their content to visuals to pricing to usability. A/B testing has emerged to be the de facto of all the website conversion testing techniques, being used by companies across the globe.

There are different types of a/b testing tools that serve different business needs. As a business, you can decide and deploy an a/b testing tool that suits your needs – website a/b testing, email marketing campaign a/b testing, social media ad campaign ab/ testing etc.

One of the bases on which you can differentiate and define your a/b testing requirement is client side vs. server side a/b testing. Most businesses often overlook this differentiating aspect, which can turn out to be a disappointment later on if they find a gap between what they expected out of an a/b testing tool and the actual result.

David Isquick explains;

“Client side: conversion rate optimization: traditionally used to optimize conversion rates in marketing or purchase funnels.”

“Server-side: product success: enables teams to optimize products and logged in experiences for engagement, retention, usage, and lifetime value.”

Differentiating between client side and server side A/B testing

Defining simply, you run a client-side test when you are creating page variations directly on the user’s browser as opposed to your server. Smart client-side tools use Javascript to create the test page variations. Because these alterations and changes are done in the visitors’ browser, it is termed as client-side a/b testing.

Gif Source: Javascript_muratoner

On the other hand, in server side testing, the test is rendered on the web server itself. The page variations are then delivered to the user’s browser where no subsequent modifications take place. The pages being tested are fetched randomly from the server and shown to the visitor.

Let’s understand the difference better using the following examples:

The marketing team at an e-commerce website conducts an analysis to determine why visitors are abandoning their cart. Based on the research, they decided to create a new cart page design. This new design removes the website navigation from the page to reduce distraction from CTAs.

Using a client-side AB testing tool, the new cart page design is created in the AB testing tool. The team uses the split testing tool to select the website navigation and remove it.

Now here is what happening when the test page is delivered with the purpose to perform client-side testing:

1. When a visitor navigates to the cart page, first the original cart page is loaded in the browser.
2. The testing software script is loaded in the page.
3. After the page gets loaded, the testing software script sets in motion the changes made to the page and displays the page to the visitor sans the top navigation.

So with client-side AB testing software, the visitor will always load the original page first, and then the changes are applied on the browser.

Here is what would happen when you are instead going to the server side testing tool:

1. When a visitor navigates to the cart page, the visitor’s browser requests the page the server.
2. The testing software randomly determines which of variation it will send to the visitor browser, so a specific set of visitors view the original page on their browser while the rest view the new cart page design.

This brings us to dig deeper into the differences between client side vs. server side a/b testing.

1.Best used when

Image Source: frinkiac_when

• Client-side a/b testing is best used when you are deploying an internal team of marketers to run experiments without the need for them to touch coding.
• Server-side a/b testing is best used when the scope of an experiment is bigger, for example, changes need to be made at the product level. This will require the coding know how.

2. Ease of use

• The client side a/b testing tools make it easier to implement changes. A WYSIWYG editor or any other visual editor can be used by almost anyone who wants to make quick changes to any element on a web page. For more advanced changes, like a different layout or page design, you might still require frontend developers, but the speed is still faster and cost and difficulty levels on the lower side compared to the server side testing.
• Server-side a/b testing tools are multilayered and complex to use. As you need the team of the front end as well as back-end developers.

3. Depth of experimentation

• Client-side experimentation is limited to the look and feel of the website. Experiments are done to see how changes in webpage elements affect conversions: you might change the look and feel of a single element, or change the layout of a page.
• While server side a/b testing tools can also be effectively used to alter the visual elements of the website, client-side testing is more suited to the purpose. The true value of server-side testing tools lies in the fact that they can be used to, test product features and proposed benefits. To explain, a functional change refers to any modification made to the technical functionality of an app or product. For example, integrating Google analytics experiments with your content management provider. In addition, server-side tools can be used on devices that do not support javascript (such as mobile apps, smart TVs, IoT devices).

4. Objectives

• The ultimate goal of client-side a/b testing is to optimize conversion rates on the webpage.
• While increasing conversions is the ultimate goal of server-side testing, there are also a number of mini goals along the way that one should tick off the list; such as enhancing the product performance, optimizing for engagement and overall product usage.

5. Deployment

• It is simple to deploy or implement client-side a/b testing. All you need is one line of code to run site-wide testing experiments.
• Ease or complexity of deployment of the test completely depends upon server-side technology stack.

6. Performance 

• The JavaScript used for client-side testing is susceptible to the characteristic drawbacks of both synchronous and asynchronous code. While asynchronous code can cause flicker effect, synchronous code often leads to increased page load time.
• With server-side testing, you are creating a self-sufficient new variation as opposed to relying on a piece of code to load itself on a third party interface and execute the changes. The direct result is a smooth website performance because any possible impact is indiscernible to the user.

7. Privacy

• The configuration of the experiment being implemented is visible both to the end user as well as to 3rd parties.Obfuscation of privacy is possible to some extent in this case.
• Security is strong in this case as details of the test experiment can be limited to the company’s server.

8. Caching

1. Integration with CDN is not required. When content is stored in CDN, experiments can be run on client-side once the cached content in CDN is delivered to the user. In this scenario, backend must serve the response to ensure there is no flicker effect. The group affiliation of the A/B test will be required by the backend for smooth website performance.
2. The changes made as part of server side testing have to be determined at CDN level. This makes the integration of CDN compulsory for running server-side test.

David Isquick explains;

“Client-side: compatible with any CDN provider: companies storing content in a CDN can run experiments client side after cached content is returned to the user.”

“Server-side: treatments must be cached at CDN layer: server-side implementations will only work if treatments are determined. A CDN integration or separate decision service is needed.”

9. Channels

• Client-side testing and experimentation are restricted to a single channel.
• Server-side testing and experimentation can be run on multiple channels at the same time. It is also suitable for running experiments in any channel at different points of time.

10. Process lifecycle

Image Source:hubprocess

• The process lifecycle in client-side testing is shorter.
  • You first create a hypothesis
  • Build the different variations using the AB testing tool
  • Analyze the results and determine the winner.
  • Only the winning variation is then built into the main application code and rolled out for all website users.
  • If the test does not generate any increase in conversions, then all variations are removed from the AB testing software and the original page design is served to visitors.

• Comparatively, the process lifecycle takes a longer route with server-side testing.
  • You first create a hypothesis
  • you implement each of your test variations on the server.
  • Analyze the results and determine the winner.
  • Since all variations are already built on the server, then when a winner is determined, it defaults to all users
  • If the test does not generate any increase in conversions, then all variations are removed from the server the original page design is served to visitors.

When to opt for client-side a/b testing

Image Source: serversidevsclientside

Even though from the differences that we have highlighted above, it might seem to some that server-side a/b testing is a superior technique over client side a/b testing, or the other way around, this isn’t exactly the way to go about it. Deciding between client-side a/b testing and server-side a/b testing is not always about which one is better. It is rather about your very specific experimentation needs and requirements.

Both these techniques call for a different sort of an experimentation roadmap, have unique objectives, and need very dissimilar organizational models.

For example, in client-side a/b testing, if the variation is a loser then there is no server code release required. On the other hand, in server side a/b testing, every experiment might require code release.

Let’s go back to our cart page example and see how the team handled implementing it using the two different A/B testing platforms:

1. Coding the cart page in a client-side platform is quick. It requires front-end developers to perform the implementation.
2. Coding the cart page in server-side platform requires both front and back-end developers to implement the test. The time is considerably longer compared to the option 1.

Let’s take two post-test scenarios and see how you will have to handle them based on your AB testing platform:

1. You run a test and one of your challengers is a winner

For client side AB testing platform, your dev team will have to now implement the winner in the platform. Depending on the complexity of the test, your dev team will be spending considerable time implementing the winner.

For server-side, the winning variation is already coded in the platform. Your dev team will have to remove all other variations and replace the original design with the winner. This is a quicker task for the dev team to handle.

2. You run a test and none of your challengers is a winner

For client side ab testing platform: you remove all the variations from the AB testing platform.  This is a quick task for the dev team to handle.

For server-side: the dev team will have to remove the all variation code from the server.  This is a longer task for the dev team. In this case, the cleanup required on the server end makes it a comparatively arduous task for companies and they need to make sure that there are no traces of the variation left.

Companies choose client-side a/b testing when there is a need for:

• Ease of experimentation: If the objective is to enable the marketing team to create and run experiments using a visual editor, which does not require code releases. This means client-side experiments don’t need your dev team to oversee execution. Also, it takes lesser time to run such experiments. Less work, more speed, and simple execution – if this is what your experimentation needs are then client-side testing is what you should opt for. And, as pointed out by Andrew Anderson in this post, speed is a critical testing requirement for businesses.
• Need for segmentation: More data is available in case of client-side a/b testing, for segmentation purposes. This is because client-side tests are run post a page has been loaded. Because these tests are done post the page loads.
• Minimum SEO impact: If you do not want the testing and experimentation on your website to negatively impact its ranking and searchability on Google, client-side is what you should be opting for. Typically, changes made through JavaScript are not accounted for or not indexed by Google, leaving absolutely no impact on your search engine optimization. As far as the question of human reviewers is concerned, as long as you are not trying to deceive, you won’t face the penalty. It’s only when it appears you are trying to trick Google that reviewers will flag you.

Juan Vega explains; 

“On the bright side, server-side rendering is great for SEO. Your content is present before you get it, so search engines are able to index it and crawl it just fine. Something that is not so with client-side rendering. At least not simply.”

Problem with client-side a/b testing

Now that you are aware when to pick client side a/b testing, you should also understand the inherent drawbacks of choosing this technique. Knowing what are the cons or problems with client-side a/b testing will help you make a sound decision.

• Flicker effect: In simple words, flicker effect is said to happen when for a very short duration of time your visitor sees the default version of the webpage before the test variation loads. Now why it matters or is counted as a drawback, is because it impacts tests results and can skew results. If people detect even a small hindrance to their onsite experience, their behavior changes. That is why the objective is to reduce flicker to an extent that it is not detectable by human eye.
• Impact on loading time: Loading time affects user behavior. Using synchronous code in client-side is known to cause slower load times.
• User restrictions: Even though client-side testing allows for a lot of ease and simplicity, most complex split tests would still require the involvement of front-end developer to implement.
• Browser related problems: There can be compatibility issues when a visitor’s Javascript has not been updated or updates very slow during a variation is being loaded. In the case where a variant being tested is not compatible with the browser, it results in visitor data becoming polluted.

When to opt for server-side a/b testing

Image Source: serverside_abtest

Next, we move to discussing when it is advantageous for companies to opt for server-side a/b testing. It needs to be clearly understood, and hence is being emphasized in this article again and again, that the limitations of one technique should not be the reason for opting the other.

• Mobile app testing: Server-side a/b testing is more suited for mobile apps because mobile sites serve dynamic content. It isn’t easy to run client-side experiments on dynamic content. Moreover, if phones have slow connectivity and weak browsers client-side Javascript based approach will only be a drawback.
• Where you need greater flexibility in experimentation: The depth and scope of server-side experimentation are higher than that of client-side experimentation. You are neither restricted to testing on a single channel nor to a few modifications that you are otherwise limited to while running client side a/b.

Robin Nichols explains;

“Server-side A/B testing is more oriented towards product teams; whose needs involve more business rules and which are tightly linked to product features. The keyword here is FLEXIBILITY.”

• Minimal website performance impact: Of course, everyone would want their website to run absolutely smoothly, without any flicker or hindrance. Since these experiments are not being rendered on the browser, your visitors hardly detect any changes.
• Enables experimentation on Omnichannel campaigns: The world is going Omnichannel and so should your digital marketing efforts. It does not any longer hold any sense to experiment only one element of a web page at a time. You need to run cross-channel and Omnichannel tests simultaneously to measure performance across all devices and on a unified client journey. This is possible only with server-side experimentation.
• Optimizes for long-term goals: Server-side a/b testing allows optimization for macro goals like customer lifetime value. It is possible to exclude certain conversions in this approach and have complete control on what you are measuring. For example, an e-commerce owner might want to exclude checkout values of above $200, which he considers as an unusually large purchase that does not reflect the true nature of purchases on the website. He does not want to take them into account because they can skew the conversion results. While an experienced developer can implement this in client-side ab testing tool, it is not as straightforward if implemented in a server-side tool.

Disadvantages of server-side a/b testing

Server-side a/b testing is not free of cons. There are some problems associated with this approach as well, which are discussed as follows:

• The server-side tool requires developers at the server end: To deploy codes and to establish any experiment, a development team is required. This means more investment in hiring professional developers.
• Longer development time (Slower execution): The client-side tool, due to its simplicity, has a quicker implementation. The server-side on the other hand requires more coding to run from the start of its operation, this makes it slower in terms of execution. With server side, the development team will actually have to code all the variations on the server side. After the test is conducted, and a winner is chosen all other variations that were coded on the server will be removed.

How to run an a/b test – client-side

The following are typical steps followed when setting up and running a client-side a/b test:

1. Insert code snippet on your website
2. Create test version using a visual editor
3. Create your conversion goals- increase in conversion goals, downloading files, clicking on a link etc.
4. Start the test on your choice of a/b testing tool
5. Keep a track of website version performance on the a/b testing tool dashboard

How to run an a/b test – server-side

To run a server side a/b test you need to follow the given steps:

1. Once a user lands on site, a unique visitor ID is generated using the API.
2. Call the API passing in parameters the unique visitor ID when the test has to be run. Alongside also pass in the test ID.
3. Responding to your request, the a/b testing tool sends the variation ID.
4. The server builds its response built on this variation ID.
5. Data servers are informed once a conversion takes place.

Conclusion

A detailed know-how about client side and server side a/b testing equip companies to make sound business decisions. If you too are deploying an a/b test tool for running tests and experiments, do read as much as you can on the uses, advantages, and downsides of both.

The post Server-Side A/B Testing Vs. Client-Side A/B Testing appeared first on Invesp.