• Creating A Conversion Roadmap: How to Prioritize Conversion Problems on Your Website

    A conversion optimization project is typically divided into four different phases that intertwine:

    • Scrutinize stage: You conduct various activities to determine possible testing areas on a platform or campaign
    • Hypothesis: you create a hypothesis for the different testing ideas identified during the scrutinize phase
    • Implement: you validate your hypothesis through the process of AB or usability testing
    • Propagate: you look for insights, share them within the organization, across marketing channels and verticals

    As you go through the process of identifying conversion issues on your website (scrutinize), you can easily identify anywhere from 150 to 250 possible test scenarios. Shana Rusonis states:

    “Prioritization is the million-dollar question in optimization programs. It gets at the heart of our most frequently asked questions: “What should I test?”

    The next logical question to ask is how do you prioritize the testing ideas?

    Before we get into the framework we use to prioritize the results of the scrutinize phase; it will be valuable to provide insights on how we structure some of the processes to prioritize our final list.

    What do you look for when optimizing for conversion?

    The scrutinize phase contains several activates designed to identify possible conversion blockers on a platform:

    1. Identifying visitor experience/personas
    2. Conducting expert reviews
    3. Conducting quantitative research
    4. Conducting qualitative research
    5. Conducting competitive analysis
    6. Conducting usability testing

    As you conduct your analysis, you will identify four different types of issues:

    1. Research opportunity: These are items you want to implement on your platform using split testing or usability testing.
    2. Investigate further: You identified a problem, but you are not sure why visitors behave a certain way. You will need to conduct further research to understand visitor behavior.
    3. Instrumentation: missing analytics tracking
    4. Fix Right away (stop the bleeding report): bugs identified on the website that must be fixed right away.

    You should immediately implement Items that you classified as “Instrumentation” and “fix right away” issues.

    Your “investigate further” list requires further research and will typically turn into research opportunities and fix right away items.

    Your prioritization should be focused on the research opportunities list. But you are not ready yet to do that.

    Bugs, usability issues, and conversion blockers

    I will take a quick detour to talk about different items we uncover as we conduct a conversion rate audit for a website.

    1. Website bugs: areas where the website is broken and not functioning as expected. You should classify the bugs you identified in terms of priority (P1: blockers to P4: small items). Bugs are typically put into the “fix it right away” bucket.
    2. Website usability issues: issues where website usability is not ideal. Usability issues are typically classified as either “fix it right away” or “research opportunities” buckets.
    3. Conversion blockers: issues that relate to persuading visitors to convert. Conversion issues are typically classified as “research opportunities” bucket.

    Many conversion consultants focus their effort on identify bugs and usability issues on a website and rarely pay close attention to conversion issues. Without getting into a philosophical debate, I firmly believe that is a vast difference between creating a user-friendly website and a highly converting site.

    Usability issues:

    1. Fixing usability issue on a website creates a user-friendly website.
    2. Usability issues focus on making the site easier to use
    3. Usability issues focus on top of mind issues for website users

    Conversion issues:

    1. Fixing conversion issues on a website creates a user-friendly website (Similar to fixing usability issues)
    2. Conversion issues focus on making the site more persuasive
    3. Conversion issues focus on psychological items that persuade visitors to convert

    Every usability issue is a conversion issue but not the other way around.

    Beyond problems – adding new website features

    CROs focus a lot on issues that stop visitors from converting on a website. However, your research during the scrutinize phase should expand beyond that to include possible new features that you can implement on the platform to further enhance conversions.

    Understanding your initial AB test plans: the research opportunities list

    It is not enough to identify research opportunities list for a website. You will need to go a step further and determine our initial test hypothesis for each item in the list. The initial test hypothesis identifies how we think the particular issue could be tested/fixed.

    A hypothesis is a predictive statement about a possible change on the page and its impact on your conversion rate.

    Now, notice how I used the term “initial hypothesis.”

    An initial hypothesis is the first stab our team takes on how we should address a potential problem on the website/webpage.

    Essentials of AB Testing

    Example of initial hypothesis:

    “Adding social proof will enhance the visitor trust in the website and increase conversions”

    We use the initial hypothesis to help us assess each line item on our research opportunities list. When we decide to implement a particular test, we further develop our initial hypothesis to a

    “concrete hypothesis.” Let’s take the initial hypothesis above and see the concrete hypothesis for it:

    Based on qualitative data collected from online polling, we observed that website users do not trust the brand and are unaware of how many users are using it. Adding social proof on the homepage will increase visitors trust and improve conversion rates by 10%.

    A concrete hypothesis

    • states how we identified the issue
    • indicates the problem identified on the page
    • states the potential impact of making the fix (we set an actual numerical goal for the test)

    Now, that you have your research opportunities list and each item has an initial hypothesis attached to it, we are ready to go through the exercise of prioritization items on our research opportunities list.

    Prioritizing your AB testing ideas (the research opportunities list)

    CRO practitioners have used several frameworks to classify testing ideas. I will cover some of the common frameworks and suggest a new model use.

    Karl E. Wiegers explains:

     “When setting priorities, you need to balance the business benefit that each function provides against its cost and any implications it has for the product’s architectural foundation future evolution.”

    Pie by Widerfunnel

    Pie framework evaluates items based on three different factors: potential, importance, and effort.

    1. Potential: Not all of your testing ideas will present the same potential in terms of increasing conversion rates. A good example of this is testing different color of a CTA on a page (no real impact to little effects) compared to changing page layout to emphasize value proposition.

    While the potential is significant, it is very subjective. What I might think will have a significant impact on bottom line, someone else might think it will have no effect.

    1. Importance: How important is the page you are optimizing?

    The Widerfunnel team defines Importance as follows:

    “How valuable is the traffic to the pages? Your most important pages are the ones with the highest volume and the costliest traffic. You may have identified pages that perform terribly, but if they don’t have a significant volume of expensive traffic, they aren’t testing priorities.”

    According to Chris Goward:

    “How do I figure out where to test first?”It’s a smart question to ask. You can’t test all pages at once. With limited time and resources to commit and, most importantly, limited traffic to allocate to each test, test prioritization is an important part of your conversion optimization plan.”

    1. Ease: How easy is it to implement the test? Widerfunnel defines ease as:

    There are two elements to consider when it comes to ease:

    • Technical implementation
    • Organizational or political barriers”

    Both of these elements impact how easy it is to implement a testing idea, albeit, they are somewhat subjective.

    Hotwire framework

    Pauline Marol, the Lead Product Manager for optimization at Hotwire introduced this model back in 2015.

    Here are the elements in this framework:

    • Main Metric
    • Location
    • Fold
    • Targeting
    • New Information
    • Benchmarking
    • Conversion Veins
    • Strategic Topic
    • Mobile
    • Opaque

    What I like most about this model is the specificity it provides the potential problem and potential fix. It also brings the idea of how strategic a test to the overall business objective for a business. According to Pauline Marol: 

    “Prioritization is a critical part when building an efficient, impactful testing program. Creating a framework to make smarter choices and thinking deeply about the key factors will make your program successful in the long term.”

    PXL framework from CXL

    The PXL framework uses the following criteria to score items on your research opportunities list:

    • Is the change above the fold? Changes above the fold are noticed by more people, thus increasing the likelihood of the test having an impact
    • Is the change noticeable in under 5 seconds? How fast does a user notice the change?
    • Does it add or remove anything? The idea is that “removing distractions” or “adding key information: will have more impact
    • Does the test run on high traffic pages? Changes to “high traffic page(s)” will have a larger impact on the bottom line.
    • Designed to increase user motivation: does the introduction of your solution increase visitor motivation? Highly motivated visitors are more likely to convert
    • How was the problem identified?
      • User testing
      • Quantitative data
      • Qualitative data (polls, surveys, etc.).
      • Mouse tracking heat maps or eye tracking
    • Ease of implementation

    The PXL framework is a step in the right direction.  Peep listed three main advantages to using it:

    1. It makes any “potential” or “impact” rating more objective
    2. It helps to foster a data-informed culture
    3. It makes “ease of implementation” rating more objective

    The central theme is the focus on trying to establish a more accurate model to score different line items on your list.

    I believe that there is room for improvement for the following reasons:

    1. All of the prioritization frameworks use a mix of points to score the identified issue and the potential fix. But a single problem could have multiple possible solutions to it. And each potential solution should be assessed with a different score.
    2. A potential problem could be identified in various ways which give it a higher score. But does identifying a problem in 4 ways make that problem a more significant issue compared to a problem identified in 3 different ways? At some point, adding more ways to how a problem is detected should not give the problem more weight.
    3. There is not enough focus on how a potential solution could impact visitors view of the website. PXL does an excellent job of using motivation, but there are other criteria that we should consider when scoring a possible solution.
    4. The models do not distinguish enough between the different types of pages (top of the funnel, mid of the of the funnel and bottom of the funnel
    5. Finally, the models use “high traffic pages” as a way to score a problem. Since analytics gives a specific number to identify what is a high traffic page, we should use that.

    Invesp Prioritization framework

    Let me state this again; a particular problem can have multiple initial hypotheses of how we can fix it. Some of our team members like to call these as testing ideas. This is important because each problem and each initial hypothesis are called out as separate line items in our prioritization framework.

    You can address a single research opportunity in different ways that require different time to implement and will have a different impact on the bottom line.

    Next is the criteria we use to evaluate each research opportunity:

    1. What is the potential Impact?
    2. Is the problem above the fold?
    3. Was the problem discovered through expert review?
    4. Was the problem identified through qualitative research?
    5. Was the problem identified through user testing?
    6. Was the problem identified through analytics?
    7. Was the problem identified through heatmaps/Video recording?
    8. Known top competitor feature?
    9. Does the test idea enhance primary conversion goal?
    10. Is implementing the change easy (politically, LOE)?
    11. Adding or removing an element?
    12. Does the test idea increase visitor trust?
    13. Does the test idea reduce FUDs?
    14. Does the test idea incentivize visitors to act right away?
    15. Does the test idea increase engagement?
    16. What type of change?
    17. What type of page?
    18. The percentage of page views

    Not all of these items carry the same weight.

    Let’s discuss some of these elements:

    1. Potential Impact: we do think that this is a subjective measure. However, we did not delete it. The value of the impact is high (score of 4), mid-impact (score of 2) and low impact (score of 1).
    2. Is the problem above the fold: where did you uncover the problem? The higher the location of the problem on a page, the more likely visitors will notice it.
    3. How did you discover the problem: most CROs will conduct a mix of activities to uncover issues on a site. Each of these activities should be called out. These are
    4. Expert review (some might refer to it as heuristic analysis),
    5. Analytics data
    6. Heatmap/session
    7. Qualitative data
    8. User testing

    Most conversion blockers are uncovered through a combination of the above elements. So, we decided to take a weighted average as follows:

    If an item discovered through one method                     score of 3

    If an item identified through two methods                      score of 6

    If an item identified through three methods                    score of 9

    If an item identified through four or five methods        score of 12

    1. Known competitive feature: there are instances where you want to test an item because you observed that one of your competitors have implemented. By no means should you understand this as a way to copy your competitors, but you should not ignore them either.
    2. Does the test idea enhance primary conversion goal: what is the primary conversion goal for the page? Is your testing idea going to increase that conversion goal?
    3. Is implementing the change easy (politically, LOE)?
    4. Adding or removing an element: Your test can do several things at the same time:

    Add or remove an element: scored at 3

    Replace an element: scored at 2

    Change the location of an element: scored at 1

    Emphasize an element: scored at 1

    1. Evaluation of visitor-centric factors that would impact how a visitor perceives a page:
    • Does the test idea increase visitor trust?
    • Does the test idea reduce FUDs?
    • Does the test idea incentivize visitors to act right away?
    • Does the test idea increase engagement?

    Two or more elements gives you a score of 6. One gives you a score of 3.

    1. Type of change: This focuses on the testing idea at hand. Are you:
    • Focused on an element level (changing a single element) scored at 1
    • Focused on page level (changing layout or several elements on the page) scored at 2
    • Focused on visitor flow around the website scored at 3
    1. Type of page: How is the page classified in your funnel?
    • Top of funnel page scored at 1
    • Mid-funnel page scored at 2
    • Bottom of the funnel page scored at 5
    • Global Change scored at 5
    1. % of page views: By looking at the % of unique page views the page gets compared to the overall website page views, you can select one of the following values:

    Is there a perfect prioritization framework?

    I do not think there is a perfect framework out there. We continue to tweak this framework as we work on different projects. On a recent project, our team identified close to 200 research opportunities on a project. We ran it through a previous version of the framework and looked at the results. Some of the top items confused us. It did not make sense that we should start with them. We went back and reworked some of the formulas. I am sure that someone can take the same framework and tweak it further. According to Chris Goward:

    “There are no standard rules for which pages are best to prioritize. Your website lives in a unique target market, including factors like your competition, seasonality, and internal cultural environment; all of these affects how your site is used and should be optimized. The priority rating you give each of your potential test pages will depend on this unique business environment.

    It is a step in the right direction. We tried to build on some of the already existing frameworks out there and add more elements that impact prioritization on our different projects.

    You can download the excel for the prioritization framework from here.

    As always, if you have suggestions, please do let’s know!

Khalid Saleh

Khalid Saleh is CEO and co-founder of Invesp. He is the co-author of Amazon.com bestselling book: “Conversion Optimization: The Art and Science of Converting Visitors into Customers.”
Khalid is an in-demand speaker who has presented at such industry events as SMX, SES, PubCon, Emetrics, ACCM and DMA, among others.

View All Posts By Khalid Saleh
Khalid Saleh

Join 25,000+ Marketing Professionals

If you enjoyed this post, please consider subscribing to the Invesp blog feed to have future articles delivered to your feed reader. or,receive weekly updates by email:

Leave a Reply

Your email address will not be published. Required fields are marked *