Creating A Conversion Roadmap: How to Prioritize Conversion Problems on Your Website

Khalid Saleh

Khalid Saleh

Khalid Saleh is CEO and co-founder of Invesp. He is the co-author of Amazon.com bestselling book: "Conversion Optimization: The Art and Science of Converting Visitors into Customers." Khalid is an in-demand speaker who has presented at such industry events as SMX, SES, PubCon, Emetrics, ACCM and DMA, among others.
Reading Time: 13 minutes

Disclaimer: This section is a TL;DR of the main article and it’s for you if you’re not interested in reading the whole article. On the other hand, if you want to read the full blog, just scroll down and you’ll see the introduction.

  • A conversion optimization project is typically divided into four different phases that intertwine:
  • As you go through the process of identifying conversion issues on your website (scrutinize), you can easily identify anywhere from 150 to 250 possible test scenarios.
  • The scrutinize phase of our SHIP process contains several activities designed to identify possible conversion blockers on a platform:
  • As you conduct your analysis, there are four different types of issues you will identify; 1, research opportunities, 2. Issues to investigate further, 3. Instrumentation and 4. Issues to fix right away.

Things To Look Out For During A Conversion Rate Audit.

  • Website bugs: arrears where the website is broken and not working as expected.
  • Website usability issues: issues where website usability is not ideal.
  • Conversion blockers: issues related to getting customers to convert

Understanding your initial AB test plans: the research opportunities list

  • It is not enough to identify research opportunities list for a website. You will need to go a step further and determine our initial test hypothesis for each item in the list.
  • For us at Invesp, we have the initial hypothesis and the concrete hypothesis.
  • The initial hypothesis is the first stab our team takes on how we should address a potential problem on the website/webpage.
  • The concrete hypothesis is developed from further research (more on this in the article)

How To Prioritize Your A/B Test Ideas

  • CRO practitioners have used several frameworks to classify testing ideas; 
  • PIE by Widerfunnel: evaluates items based on three different factors: potential, importance, and effort.
  • The Hotwire framework: Pauline Marol, the Lead Product Manager for optimization at Hotwire introduced this model back in 2015.
  •  The PXL framework by CXL: it uses 7 criteria to score items on your research opportunities list (more on this in the article).
  • At Invesp, we have our own prioritization framework that contains 18 criterias for your conversion research.

Here’s A Longer And More Detailed Version Of The Article


A conversion optimization project is typically divided into four different phases that intertwine:

  • Scrutinize stage: You conduct various activities to determine possible testing areas on a platform or campaign
  • Hypothesis: you create a hypothesis for the different testing ideas identified during the scrutinize phase
  • Implement: you validate your hypothesis through the process of AB or usability testing
  • Propagate: you look for insights, share them within the organization, across marketing channels and verticals

As you go through the process of identifying conversion issues on your website (scrutinize), you can easily identify anywhere from 150 to 250 possible test scenarios. Shana Rusonis states:

“Prioritization is the million-dollar question in optimization programs. It gets at the heart of our most frequently asked questions: “What should I test?”

The next logical question to ask is how do you prioritize the testing ideas?

Before we get into the framework we use to prioritize the results of the scrutinize phase; it will be valuable to provide insights on how we structure some of the processes to prioritize our final list.

What do you look for when optimizing for conversion?

The scrutinize phase contains several activates designed to identify possible conversion blockers on a platform:

  1. Identifying visitor experience/personas
  2. Conducting expert reviews
  3. Conducting quantitative research
  4. Conducting qualitative research
  5. Conducting competitive analysis
  6. Conducting usability testing

As you conduct your analysis, you will identify four different types of issues:

  1. Research opportunity: These are items you want to implement on your platform using split testing or usability testing.
  2. Investigate further: You identified a problem, but you are not sure why visitors behave a certain way. You will need to conduct further research to understand visitor behavior.
  3. Instrumentation: missing analytics tracking
  4. Fix Right away (stop the bleeding report): bugs identified on the website that must be fixed right away.

You should immediately implement Items that you classified as “Instrumentation” and “fix right away” issues.

Your “investigate further” list requires further research and will typically turn into research opportunities and fix right away items.

Your prioritization should be focused on the research opportunities list. But you are not ready yet to do that.

Bugs, usability issues, and conversion blockers

I will take a quick detour to talk about different items we uncover as we conduct a conversion rate audit for a website.

  1. Website bugs: areas where the website is broken and not functioning as expected. You should classify the bugs you identified in terms of priority (P1: blockers to P4: small items). Bugs are typically put into the “fix it right away” bucket.
  2. Website usability issues: issues where website usability is not ideal. Usability issues are typically classified as either “fix it right away” or “research opportunities” buckets.
  3. Conversion blockers: issues that relate to persuading visitors to convert. Conversion issues are typically classified as “research opportunities” bucket.

Many conversion consultants focus their effort on identify bugs and usability issues on a website and rarely pay close attention to conversion issues. Without getting into a philosophical debate, I firmly believe that is a vast difference between creating a user-friendly website and a highly converting site.

Usability issues:

  1. Fixing usability issue on a website creates a user-friendly website.
  2. Usability issues focus on making the site easier to use
  3. Usability issues focus on top of mind issues for website users

Conversion issues:

  1. Fixing conversion issues on a website creates a user-friendly website (Similar to fixing usability issues)
  2. Conversion issues focus on making the site more persuasive
  3. Conversion issues focus on psychological items that persuade visitors to convert

Every usability issue is a conversion issue but not the other way around.

Beyond problems – adding new website features

CROs focus a lot on issues that stop visitors from converting on a website. However, your research during the scrutinize phase should expand beyond that to include possible new features that you can implement on the platform to further enhance conversions.

Understanding your initial AB test plans: the research opportunities list

It is not enough to identify research opportunities list for a website. You will need to go a step further and determine our initial test hypothesis for each item in the list. The initial test hypothesis identifies how we think the particular issue could be tested/fixed.

A hypothesis is a predictive statement about a possible change on the page and its impact on your conversion rate.

Now, notice how I used the term “initial hypothesis.”

An initial hypothesis is the first stab our team takes on how we should address a potential problem on the website/webpage.

Example of initial hypothesis:

“Adding social proof will enhance the visitor trust in the website and increase conversions”

We use the initial hypothesis to help us assess each line item on our research opportunities list. When we decide to implement a particular test, we further develop our initial hypothesis to a

“concrete hypothesis.” Let’s take the initial hypothesis above and see the concrete hypothesis for it:

Based on qualitative data collected from online polling, we observed that website users do not trust the brand and are unaware of how many users are using it. Adding social proof on the homepage will increase visitors trust and improve conversion rates by 10%.

A concrete hypothesis

  • states how we identified the issue
  • indicates the problem identified on the page
  • states the potential impact of making the fix (we set an actual numerical goal for the test)

Now, that you have your research opportunities list and each item has an initial hypothesis attached to it, we are ready to go through the exercise of prioritization items on our research opportunities list.

Prioritizing your AB testing ideas (the research opportunities list)

CRO practitioners have used several frameworks to classify testing ideas. I will cover some of the common frameworks and suggest a new model use.

Karl E. Wiegers explains:

 “When setting priorities, you need to balance the business benefit that each function provides against its cost and any implications it has for the product’s architectural foundation future evolution.”

Pie by Widerfunnel

Pie framework evaluates items based on three different factors: potential, importance, and effort.

  1. Potential: Not all of your testing ideas will present the same potential in terms of increasing conversion rates. A good example of this is testing different color of a CTA on a page (no real impact to little effects) compared to changing page layout to emphasize value proposition.

While the potential is significant, it is very subjective. What I might think will have a significant impact on bottom line, someone else might think it will have no effect.

  1. Importance: How important is the page you are optimizing?

The Widerfunnel team defines Importance as follows:

“How valuable is the traffic to the pages? Your most important pages are the ones with the highest volume and the costliest traffic. You may have identified pages that perform terribly, but if they don’t have a significant volume of expensive traffic, they aren’t testing priorities.”

According to Chris Goward:

“How do I figure out where to test first?”It’s a smart question to ask. You can’t test all pages at once. With limited time and resources to commit and, most importantly, limited traffic to allocate to each test, test prioritization is an important part of your conversion optimization plan.”

  1. Ease: How easy is it to implement the test? Widerfunnel defines ease as:

There are two elements to consider when it comes to ease:

  • Technical implementation
  • Organizational or political barriers”

Both of these elements impact how easy it is to implement a testing idea, albeit, they are somewhat subjective.

Hotwire framework

Pauline Marol, the Lead Product Manager for optimization at Hotwire introduced this model back in 2015.

Here are the elements in this framework:

  • Main Metric
  • Location
  • Fold
  • Targeting
  • New Information
  • Benchmarking
  • Conversion Veins
  • Strategic Topic
  • Mobile
  • Opaque

What I like most about this model is the specificity it provides the potential problem and potential fix. It also brings the idea of how strategic a test to the overall business objective for a business. According to Pauline Marol: 

“Prioritization is a critical part when building an efficient, impactful testing program. Creating a framework to make smarter choices and thinking deeply about the key factors will make your program successful in the long term.”

PXL framework from CXL

The PXL framework uses the following criteria to score items on your research opportunities list:

  • Is the change above the fold? Changes above the fold are noticed by more people, thus increasing the likelihood of the test having an impact
  • Is the change noticeable in under 5 seconds? How fast does a user notice the change?
  • Does it add or remove anything? The idea is that “removing distractions” or “adding key information: will have more impact
  • Does the test run on high traffic pages? Changes to “high traffic page(s)” will have a larger impact on the bottom line.
  • Designed to increase user motivation: does the introduction of your solution increase visitor motivation? Highly motivated visitors are more likely to convert
  • How was the problem identified?
    • User testing
    • Quantitative data
    • Qualitative data (polls, surveys, etc.).
    • Mouse tracking heat maps or eye tracking
  • Ease of implementation

The PXL framework is a step in the right direction.  Peep listed three main advantages to using it:

  1. It makes any “potential” or “impact” rating more objective
  2. It helps to foster a data-informed culture
  3. It makes “ease of implementation” rating more objective

The central theme is the focus on trying to establish a more accurate model to score different line items on your list.

I believe that there is room for improvement for the following reasons:

  1. All of the prioritization frameworks use a mix of points to score the identified issue and the potential fix. But a single problem could have multiple possible solutions to it. And each potential solution should be assessed with a different score.
  2. A potential problem could be identified in various ways which give it a higher score. But does identifying a problem in 4 ways make that problem a more significant issue compared to a problem identified in 3 different ways? At some point, adding more ways to how a problem is detected should not give the problem more weight.
  3. There is not enough focus on how a potential solution could impact visitors view of the website. PXL does an excellent job of using motivation, but there are other criteria that we should consider when scoring a possible solution.
  4. The models do not distinguish enough between the different types of pages (top of the funnel, mid of the of the funnel and bottom of the funnel
  5. Finally, the models use “high traffic pages” as a way to score a problem. Since analytics gives a specific number to identify what is a high traffic page, we should use that.

Invesp Prioritization framework

Let me state this again; a particular problem can have multiple initial hypotheses of how we can fix it. Some of our team members like to call these as testing ideas. This is important because each problem and each initial hypothesis are called out as separate line items in our prioritization framework.

You can address a single research opportunity in different ways that require different time to implement and will have a different impact on the bottom line.

Next is the criteria we use to evaluate each research opportunity:

  1. What is the potential Impact?
  2. Is the problem above the fold?
  3. Was the problem discovered through expert review?
  4. Was the problem identified through qualitative research?
  5. Was the problem identified through user testing?
  6. Was the problem identified through analytics?
  7. Was the problem identified through heatmaps/Video recording?
  8. Known top competitor feature?
  9. Does the test idea enhance primary conversion goal?
  10. Is implementing the change easy (politically, LOE)?
  11. Adding or removing an element?
  12. Does the test idea increase visitor trust?
  13. Does the test idea reduce FUDs?
  14. Does the test idea incentivize visitors to act right away?
  15. Does the test idea increase engagement?
  16. What type of change?
  17. What type of page?
  18. The percentage of page views

Not all of these items carry the same weight.

Let’s discuss some of these elements:

  1. Potential Impact: we do think that this is a subjective measure. However, we did not delete it. The value of the impact is high (score of 4), mid-impact (score of 2) and low impact (score of 1).
  2. Is the problem above the fold: where did you uncover the problem? The higher the location of the problem on a page, the more likely visitors will notice it.
  3. How did you discover the problem: most CROs will conduct a mix of activities to uncover issues on a site. Each of these activities should be called out. These are
  4. Expert review (some might refer to it as heuristic analysis),
  5. Analytics data
  6. Heatmap/session
  7. Qualitative data
  8. User testing

Most conversion blockers are uncovered through a combination of the above elements. So, we decided to take a weighted average as follows:

If an item discovered through one method                     score of 3

If an item identified through two methods                      score of 6

If an item identified through three methods                    score of 9

If an item identified through four or five methods        score of 12

  1. Known competitive feature: there are instances where you want to test an item because you observed that one of your competitors have implemented. By no means should you understand this as a way to copy your competitors, but you should not ignore them either.
  2. Does the test idea enhance primary conversion goal: what is the primary conversion goal for the page? Is your testing idea going to increase that conversion goal?
  3. Is implementing the change easy (politically, LOE)?
  4. Adding or removing an element: Your test can do several things at the same time:

Add or remove an element: scored at 3

Replace an element: scored at 2

Change the location of an element: scored at 1

Emphasize an element: scored at 1

  1. Evaluation of visitor-centric factors that would impact how a visitor perceives a page:
  • Does the test idea increase visitor trust?
  • Does the test idea reduce FUDs?
  • Does the test idea incentivize visitors to act right away?
  • Does the test idea increase engagement?

Two or more elements gives you a score of 6. One gives you a score of 3.

  1. Type of change: This focuses on the testing idea at hand. Are you:
  • Focused on an element level (changing a single element) scored at 1
  • Focused on page level (changing layout or several elements on the page) scored at 2
  • Focused on visitor flow around the website scored at 3
  1. Type of page: How is the page classified in your funnel?
  • Top of funnel page scored at 1
  • Mid-funnel page scored at 2
  • Bottom of the funnel page scored at 5
  • Global Change scored at 5
  1. % of page views: By looking at the % of unique page views the page gets compared to the overall website page views, you can select one of the following values:

Is there a perfect prioritization framework?

I do not think there is a perfect framework out there. We continue to tweak this framework as we work on different projects. On a recent project, our team identified close to 200 research opportunities on a project. We ran it through a previous version of the framework and looked at the results. Some of the top items confused us. It did not make sense that we should start with them. We went back and reworked some of the formulas. I am sure that someone can take the same framework and tweak it further. According to Chris Goward:

“There are no standard rules for which pages are best to prioritize. Your website lives in a unique target market, including factors like your competition, seasonality, and internal cultural environment; all of these affects how your site is used and should be optimized. The priority rating you give each of your potential test pages will depend on this unique business environment.

It is a step in the right direction. We tried to build on some of the already existing frameworks out there and add more elements that impact prioritization on our different projects.

You can download the excel for the prioritization framework from here.

As always, if you have suggestions, please do let’s know!

Frequently Asked Questions

What is Conversion Rate Optimization (CRO)?

  • Answer: Conversion Rate Optimization, or CRO, is the practice of increasing the percentage of website visitors who take a desired action, such as making a purchase or filling out a form. It involves analyzing user behavior and implementing changes to improve the user experience and boost conversions.

What is a Conversion Funnel?

  • Answer: A conversion funnel is a series of steps that a website visitor goes through before completing a desired action. It usually starts with attracting the visitor to the site and ends with them taking an action like making a purchase.

What is A/B Testing?

  • Answer: A/B testing is a method where you compare two versions of a webpage or app against each other to determine which one performs better in terms of conversions.

How Do I Prioritize My A/B Testing Ideas?

  • Answer: Prioritizing A/B testing ideas involves evaluating the potential impact, ease of implementation, and relevance of each test. Frameworks like PIE (Potential, Importance, Ease) or the Invesp Prioritization Framework can help in this process.

What are FUDs in CRO?

  • Answer: FUDs stand for Fears, Uncertainties, and Doubts. These are psychological barriers that can prevent a visitor from converting. Identifying and addressing FUDs can significantly improve your conversion rate.

What Does ‘Above the Fold’ Mean?

  • Answer: ‘Above the fold’ refers to the portion of a webpage that is visible without scrolling. Elements placed here are more likely to be seen by visitors and therefore have a higher impact on conversions.

How Long Does It Take to See Results from CRO?

  • Answer: The time it takes to see results from CRO can vary widely depending on the changes made and the metrics you’re tracking. However, a good rule of thumb is to allow at least a few weeks for meaningful patterns to emerge.
Share This Article

Join 25,000+ Marketing Professionals!

Subscribe to Invesp’s blog feed for future articles delivered to receive weekly updates by email.

Khalid Saleh

Khalid Saleh

Khalid Saleh is CEO and co-founder of Invesp. He is the co-author of Amazon.com bestselling book: "Conversion Optimization: The Art and Science of Converting Visitors into Customers." Khalid is an in-demand speaker who has presented at such industry events as SMX, SES, PubCon, Emetrics, ACCM and DMA, among others.

Discover Similar Topics