There is no such thing as a perfect website.
No designer, developer, or marketer can spot the full range of problems on a website alone.
Why? One direct reason is that only real users can identify some of these problems.
The market today is a fierce battlefield, as everyone is striving to deliver the best they can to their users. At the front lines, marketers need techniques to better understand website visitors and customers.
There are many articles that tackle usability testing and its benefits. But most of these articles treat usability as a broad, theoretical term. Most fail to suggest concrete steps for detecting usability problems and propose potential solutions.
One of the very first steps when implementing the 12-step conversion optimization system to increase a website’s conversion rate is to conduct a website assessment. This article will provide 7 concrete techniques used by UX and CRO experts to uncover problems on your website.
1. The Guerrilla Testing -In Person Usability Testing
Some VPs of marketing and CMOs fail to understand that guerrilla testing can occur throughout the entire life cycle of a website.
The term usability, according to Nielsen, refers to the quality attribute that assesses how easy an interface is to use for both novices and experts. Ease of use has been a goal for designers, developers, and marketers all along.
These usability tests are usually described as laboratory-based studies in which users perform tasks in a setting similar to their environment, while observers take notes.
With more than 80% of subject matter experts using it as their first and primary resort for problem detection, guerrilla testing is one of the most commonly used methods for improving user experience, as shown in the graph below.
Image source: measuringu
How to Conduct a Usability Test?
Our detailed guide on conducting usability testing breaks the process into 3 stages:
- Step1: Planning the Test
- Identify the Goal of the Research
- Select the Research Method
- Design Your Research Questions
- Recruit Participants
- Step 2: Conducting Your Research
- Step 3: Analyzing Research Results
- Transcribing Your Data
- Data Mining: Looking for Insights
- Defining Themes & Looking for Patterns
The Usability Testing in Action
Let’s take a look at this case study of a usability test for an e-commerce website, in which three users were recruited to complete tasks using the existing web services.
The main objective of this testing was to identify pain points users experienced while browsing the website under its current design and to determine whether users could navigate it easily, regardless of what they were looking for.
Scenario and Tasks:
Tasks were presented in a contextual scenario for users to complete, as the one below:
Christmas is coming, and you want to buy presents for your family. Your mother has always wanted a rose-gold Michael Kors watch, so you want to buy it for her.
The aim of using a scenario was to keep the tasks between users as coherent as possible, so the results could be compared efficiently in the analysis.
Tasks:
1. Browse and find a Michael Kors watch in rose gold and its details, such as price, warranty, and the material used for production.
2. Add the product to the cart and proceed to checkout.
3. Find the information on the company’s policies, including its delivery and return practices.
Execution and Observation:
First, participants were asked to browse the website to find the “Michael Kors watch in rose gold.” When asked to find a specific product using the navigation, two of the three participants couldn’t locate it right away and began to question whether that option existed at all.
Regarding the second task, the participants had no trouble adding the product to the cart and completing checkout.
For the third task, all participants in the usability study had difficulty finding information about the website’s delivery and return policies.
Advantages of Usability Testing
- Help you identify and uncover problems, as user testing can clarify any issues you noticed during analysis of analytics data or heatmaps/video recordings.
- Test the validity of a certain feature on a page or on a site.
- Uncover unexpected problems within your website that you may not have picked up on earlier.
- Help you inform the design process.
- Quick and easy to conduct, and it doesn’t require much effort.
Disadvantages of Usability Testing
- Data contamination: it’s a bit scripted and controlled, so, of course, that contaminates the data to a certain extent.
- Subjectivity: The person conducting the user test may make prompts or suggestions that can affect the test’s validity and further compromise it.
- Users’ awareness of the test: because it isn’t a normal online experience but rather a test, users skew their behaviors.
2. Heuristic Evaluation: Critique Expertise and Feedback
Heuristic evaluation relies on subject matter experts (SMEs) to predict interface problems (website/mobile/mobile app) and rate them. It’s still one of the most widely used methods among experts.
Different CRO companies approach heuristic evaluation in different ways. But all heuristics are rooted in the same principles.
At Invesp, we combine the conversion framework, a form of qualitative data, with our heuristic evaluation process to ensure we cover as many problems as possible.
Nielsen uses 10 basic heuristics as broad rules that cover many possible user interface problems. You can work your way around these rules by adding anything that works for your website, as well as by eliminating whatever does not apply to your situation.
So, how many experts should be conducting the evaluation? You need more than one expert to uncover heuristic problems. The more experts you add, the more problems you will be able to uncover.
We usually recommend conducting the heuristic evaluation with 3 to 5 experts. Having more than 5 experts conduct the evaluation does not seem to add much more value.
Severity Rating in Heuristic Evaluation
As we evaluate a website, two things happen. We uncover problems with the website. These problems receive a rating as follows:
- P1: Catastrophic and nearly unusable for all users, imperative to fix
- P2: Major problems are present, but users are still able to perform tasks
- P3: Minor problems are present for some users
- P4: Cosmetic issues only
These items are then sent to the client or the development team for fixing. At the same time, the team will uncover research opportunities. These are items we need to investigate further, develop better designs for, or improve visitor flow. Research opportunities identified during the heuristic evaluation form the basis of our CRO work.
How to Conduct a Heuristic Evaluation?
The knowledge and experience of reviewers are crucial to heuristic evaluation because they can drastically affect the findings.
You go through three fundamental steps when conducting a heuristic evaluation:
The Preparation phase:
- Agree on who will do the review.
- Determine the key principles or guidelines of the evaluation.
- Select the key content areas for evaluation that will cover a wide range of evaluation types.
The Execution phase:
- For each heuristic, experts, individually or in a group, assess the severity of each problem and assign a severity rating from 0 to 4.
The Consolidation phase:
- Discuss the aggregated findings and identify problem areas; these can be specific details or broader pain points.
- Look for patterns.
- Suggest appropriate recommendations and allocate resources to fix those problems through realistic solutions.
The Heuristic Evaluation in Action
The following case study evaluates the user interface of the Gateway to Electronic Media Services (GEMS) system.
Image source: GEMS
The study focuses on identifying problem areas on the website and making informed decisions about areas that need improvement.
For these purposes, a questionnaire based on Nielsen’s 10 heuristics was designed and administered to a sample of 25 NTU students.
The evaluation included four different tasks for the users to complete:
- Finding the title of a book.
- Finding the author(s) of the book.
- Finding the title of the book using the ISBN.
- Finding the ISBN number of a book with the help of a title.
The overall impression of the users:
The users were asked to give their impressions of the GEMS system, as you can see below:
This case study poses an interesting question. Who should conduct the heuristic evaluation? Should CRO/usability experts conduct the review, or should we recruit users of the system? If you have the budget and time, do both. If you absolutely have to choose one, we believe conversion experts are better than non-specialists at conducting heuristic evaluation.
Advantages of Heuristic Evaluation
- Obtain feedback in the early stages of the design.
- Obtain more accurate and detailed results.
- Easily obtain meaningful results in a very short time.
- Evaluate potential issues on your website.
Disadvantages of Heuristic Evaluation
- Based on experts’ personal inferences, which makes it subjective to some extent.
- The evaluation may identify minor issues as well as some major ones.
- The fact that only experts can undertake this evaluation makes it obligatory for you to assign and hire experts, in case you lack the needed knowledge, which requires resources.
3. Cognitive Walkthrough
A cognitive walkthrough traces the mental processes users are expected to follow to accomplish tasks.
This technique’s primary focus is ease of use on a specific website, and it’s based on a theory that users learn through exploration rather than formal training.
After selecting a task from the suite the website supports, the expert’s main objective is to determine one or more sequences of actions that users will actually follow.
How to Conduct a Cognitive Walkthrough?
The process of conducting a cognitive walkthrough evaluation can be done through three major steps:
Step 1. Preparation
- Define an assumed user background and their level of knowledge about your website.
- Choose a task sample that is both realistic and important.
- Specify correct action subsequences for the task.
- Determine interface states along the sequence(s).
Step 2. Analysis
- For each correct action, come up with a successful story that can explain and give reasons why a user would choose to take that action, or a failure story on why a user wouldn’t choose to take that action.
- Record problems, reasons, and assumptions.
- Suggest alternatives and solutions for the defined problems.
Step 3. Follow-up
- Modify your website based on the findings to eliminate the problems.
The Cognitive Walkthrough in Action
This case study was conducted using the “História do dia,” a children’s website.
The method used to validate the cognitive walkthrough for children included five main steps:
1. Performing the cognitive walkthrough on all the tasks available on the website (performed first by an expert).
2. Creating a list of typical tasks that children perform on the website.
3. Getting two other experts to perform the cognitive walkthrough on the same list of tasks.
4. Testing the tasks with children and recording their results.
5. Comparing the difficulties experienced by the children and the problems predicted in steps 1 and 3.
Users:
Children aged 5 to 7 with some knowledge of using a mouse and keyboard. Users can also read and learn the basics of web navigation.
List of Tasks:
Read the daily story:
- Click the picture associated with the story.
- Click the icon to start the story.
- Click the icons to navigate through the story.
- Results:
Hear the daily story:
- Click the picture associated with the story.
- Click the icon to start the narration.
- Results:
Read a story from the archive:
- Click the “Arquivo” icon.
- Click the desired month.
- Click on the period within the chosen month in which the story was first made available.
- Click the picture associated with the story.
- Results:
Vote on the daily story:
- Click the “Votar” icon.
- Choose how many stars we will award to the story.
- Results:
Conclusion:
As shown in the table below, the cognitive walkthrough is a reliable method for predicting usability problems and for identifying areas for improvement in an interface.
Advantages of Cognitive Walkthrough
- It’s an effective method for identifying problem spots, if any, and suggesting possible causes.
- It uses a more detailed, more complete procedure with more practical instructions for experts, and it uncovers almost 40% of the problems and hurdles identified by user testing.
Disadvantages of Cognitive Walkthrough
- Requires an expert’s knowledge to be conducted.
- It provides no quantitative data because it’s primarily theoretical.
- Time-consuming at the level of conducting a complete and thorough analysis.
- It depends on analysis and theory rather than user testing.
4. Checklist Review (Guidelines Review)
The checklist review is similar to the heuristic evaluation, but more detailed, based on concrete test statements and a set of best practices and principles.
How to conduct a Checklist Review?
In his book, “Designing Web Navigation: Optimizing the User Experience,” James Kalbach states that the checklist evaluation or the guideline review is conducted throughout three main stages:
- Preparation: This stage involves recruiting subjects once they are familiar with the site’s key pages and determining which items need evaluation.
- Execution: At this stage, there are only major steps; you review the test statements and rate both those checkpoints and how they are met.
- Consolidation: This stage is fully devoted to a thorough analysis to identify patterns in the findings, how each key page works, the main problems identified, and appropriate recommendations to address them.
Because of their detailed nature, these statements and guidelines tend to grow over time to cover all areas where problems might occur on the website, which requires more attention and focus when implemented.
These statements are time-consuming but still easy to conduct. They can replace usability tests if budget constraints exist.
| Evaluation Point | Rating | Comment |
| Navigating | ||
| All the major parts of the website are accessible from the home page | ||
| Critical content is located high in the structure of the site | ||
| Content is within three clicks of the home page | ||
| Alternative navigation mechanisms are available | ||
| An exit point appears on every page | ||
| Further navigation suggestions on every page apart from a global navigation | ||
| Related information is linked together | ||
| Navigation links behave consistently and predictably | ||
| Labelling | ||
| Links are labeled accurately with mutually exclusive terms | ||
| The language used is simple and in terms that the site visitors can understand | ||
| The meaning of navigation is clear, consistent, and useful | ||
| The destination of navigation links is predictable | ||
| Abbreviations are not used, or, when they are used, they are clear and obvious to target audiences | ||
| Each page has a browser title that is coordinated with the navigation and page title | ||
| Each page has a clear page title, related to other labels around it | ||
| If the site supports multiple languages, the navigation is flexible to accommodate translations | ||
| Visual Designs | ||
| Navigation options are clear and visible | ||
| Navigation options are readable and quickly scannable | ||
| There is a clear hierarchy of options, labels, and headers on each page | ||
| The navigation mechanisms are pleasing and attractive | ||
| The layout is clear with a sufficient amount of white space | ||
| Colors are used effectively to prioritize and organize navigation | ||
| Browser | ||
| Back buttons and other assumed browser functions are operable | ||
| Each page has a human-readable URL | ||
| The URL is related to the name of the company and shows a predictable structure within the site via directory structure | ||
| There are no broken navigation links |
Advantages of the Checklist Review
- Quick feedback to designers with relatively direct answers
- The detailed nature, which limits the scope of interpretation and guessing
- Relatively inexpensive
- Doesn’t require knowledge and expertise to be conducted
Disadvantages of the Checklist Review
- Time-consuming due to its detailed nature
- Requires a lot of intention and focus when being conducted
5. The PURE Method: A Substitute for Usability Testing
The term PURE stands for “Pragmatic Usability Rating by Experts.” This new method, according to Christian Rohrer, is used to estimate the amount of ‘friction’ a typical user is likely to experience when using an interface. The focus is on examining the ease of use.
To estimate this friction, the method provides a scoring mechanism, and the experts’ job is to rate it according to predefined rubrics and criteria.
To simplify scoring, Christian Rohrer introduced colored bars of varying heights. Each bar and color has significance.
Keep in mind that the numbers and colors shown in the scorecard represent the level of friction; the more friction there is, the higher the number and the ‘’hotter’’ the color becomes.
| Bar heights | Significance |
| The green bar represents steps that can be easily accomplished by the target users due to low cognitive load and, mainly, because the patterns are known. | |
| As for steps that fall in the yellow range, they require some effort from users and impose a significant cognitive load. | |
| The red bar indicates steps that users could accomplish but require a humongous cognitive load. However, most of the steps in this stage are challenging. The majority of users are likely to fail. |
As the name suggests, the PURE method first quantifies a website’s ease of use by aggregating ratings against a set of criteria. Next, they provide qualitative insights on how to fix the issues they rated.
The PURE method combines both behavioral and attitudinal measures. In this sense, it enables you to fill the gaps left by traditional methods, whether behavioral or attitudinal.
Below is a scorecard showing the aggregated scores for each task.
The numbers and colors also indicate an overall rate for the task, usually determined by the worst score.
For instance, in the following example, a single step rated (3) on ‘browser extension’ led the entire task to be labeled red, indicating failure points in a fundamental task.
On the scorecard, green and yellow indicate tasks that are easy to fulfill, with less friction.
Image source: app.box
How to conduct the PURE evaluation?
The PURE evaluation is quick, cheap, and reliable. However, it takes around ten detailed, specific steps to obtain valid results.
Identify target user profiles: defining your user profile types will help experts maintain consistent ratings. For this task, we usually recommend that our panel review the website personas.
Defining fundamental tasks: identify tasks that are fundamental to your evaluation by asking questions such as:
- “What are the critical tasks for the business to succeed?”
- “What kind of steps allow the target users to meet their core needs?”
Uncover your users’ happy paths: the most desired way they accomplish their tasks.
Having more than one path sounds reasonable, especially with multiple users, but it will require significant work and data analysis in later stages.
- Setting your steps’ boundaries: this stage is dedicated to defining your steps’ beginnings, where users are provided with a set of options, and their endings, when they expect a response from the interface.
- Review by 3 expert raters: A panel of 3 CRO specialists gathers in a room to conduct the PURE review. We try to select experts who have no prior involvement with the design in hand or with any other products, and we also include product professionals to avoid bias and subjectivity when rating. The lead in the panel starts performing each of the fundamental tasks. The rest of the panel will observe and record a score. Each member of the panel will record their score silently without sharing it with others. Again, each task will be rated on a scale of 1 (easy) to 3(challenging).
- Determine the PURE score for each task: In this step, the panel reviews each task, and each member reveals their score for it. The panel must agree on the score for the task.
The panel might have different opinions and rate differently, which will lead to the following cases:
- Case 1: all three decide on the same individual score; in this case, the decided score is obvious, and the margin of disagreement is absent.
- Case 2: 2 out of 3 agreed on a certain score; it’s obvious that the score represents the majority, unless the discussion revealed insights the majority missed and the panel chose the less popular score as their final decision.
- Case 3: all three experts have different scores; in this case, the discussion will clarify any wrongful assumptions or misunderstandings about the method or interface.
Summing up the PURE scores: in the final step, the scores for each task are added together. As mentioned earlier, these bar heights indicate the simplicity or complexity of the task at hand. Knowing where your users encounter hurdles gives you more room to improve your product.
Advantages of the PURE Method
- Easy to conduct and cost-effective.
- You can use it on user experiences that haven’t been completely built yet.
- Provides regular assessments and tracks long-term improvement.
Disadvantages of the PURE Method
- Requires knowledge and expertise to be conducted.
- It is time-consuming and requires significant effort and planning.
6. The Navigation Stress Test
The navigation stress test was first introduced by Keith Instone. This test asks tough questions about your website, helps focus on navigation issues, and forces you to make conscious design decisions.
These questions revolve around three basic points:
- Where am I?
- What’s here?
- Where can I go?
How to Conduct a Navigation Stress Test?
The stress test method is quite easy and simple:
- The first step is about randomly choosing a webpage.
- After removing the URL from that page, print it in black and white.
- Pretend to be a first-time visitor, or ask someone who has never seen the site before to participate in the test, and try to answer the questions included in the chart below:
| Navigation Questions | Mark up on the paper |
| What is this page about? | Draw a rectangle around the title of the page or write it on the paper yourself |
| What site is this? | Circle the site name, or write it on the paper yourself |
| What are the major sections of this site? | Label with X |
| What major sections is this page in? | Draw a triangle around the X |
| What is one level up from here? | Label with U |
| How do I get to the home page? | Label with H |
| How do I get to the top of this section of the site? | Circle the major groups of links and label:D: More details, sub-pages of this one N: Nearby pages, within the same section as this page S: Pages on same site, but not as near O: Off-site pages |
| What does each group of links represent? | Write the set of selections as: Choice 1 > Choice 2 > … |
Chart source: instone
- Start marking up the question you answered with the symbols in the chart.
- Prepare recommendations and solutions for the questions you didn’t answer based on the findings.
Applying the Navigation Stress Test
Advantages of the Navigation Stress Test
- Among the cheapest methods to evaluate your website.
- One of the quickest techniques to apply.
- No need to hire experts to conduct the test.
Disadvantages of the Navigation Stress Test
- The major problem with this method is the limitation to the specific page you are evaluating. It is hard to generalize the findings to the other pages of the website.
7. UX Benchmarking Studies
UX Benchmarking studies, also called summative studies, help you position your website features against industry standards.
With benchmarking, you can test, measure, and track how well your website is doing in correlation to users’ experience metrics over time.
These studies combine quantitative and qualitative data to measure the user experience.
Benchmarking studies can be divided into two categories:
Stand-alone Benchmarking: This type of benchmarking is an extremely effective method for establishing your own best practices and identifying problem areas by tracking your website’s evolution over time.
Competitive analysis: this type of analysis is based on adding another component to the equation, usually your most important competitor, to help compare your site key performance indicators, such as:
- success rates of revenue-driven tasks
- efficiency metrics such as time on tasks
Net Promoter Scores (NPS) with competitors to recognize and see the areas that they need to focus on more in comparison to them.
How to conduct a Benchmark?
Image source: SlideShare
1. Set a plan:
- Determine and define your comparators—what you want to benchmark and why. For instance, it could be all your Key Performance Indicators (KPIs) or just some of them.
Examples of KPIs:
- Satisfaction
- The average number of errors
- Task success rate
- The average time of tasks
- Define your target audience.
- Select competitors to benchmark against.
2. Conduct research using task-based testing:
- The rationale for choosing task-based testing is to ensure that your baseline metrics are available for later measurements.
3. Gather both quantitative and qualitative data:
- Use the quantitative data for testing.
- Use the qualitative data to support your findings and support the why.
4. Re-test and research at least twice a year: to measure and track the changes and see if your expectations are met.
UX Benchmarking Studies in Action
In the following case study, you can see a UX benchmarking study conducted between American Airlines and Delta Airlines.
The plan
The plan for this study included pre-task questions, two tasks to be performed, and follow-up questions:
Image source: How to conduct UX Benchmarking
The tasks
The two tasks users had to perform were:
- Go through the entire process of booking a flight.
- Locate the total price for checking in two bags.
The post-tasks included:
- Self-reported success
- How satisfied were you with the process of booking a flight?
- Ease of use
- Ease of navigation
- Clear where to start
- Satisfaction with the amount of time to complete the task
The wrap-up included:
- Overall rating
- Post-task branding attributes
- What did you like the most about the airline website?
- What did you like the least about the airline website?
- Likelihood to recommend
The results
For the first task, most participants reported successfully completing the entire purchase process.
Image source: How to conduct UX Benchmarking
Image source: How to conduct UX Benchmarking
For the second task, most participants reported being unable to find the total price for checking two bags.
Image source: How to conduct UX Benchmarking
These are the heatmaps of users’ paths when trying to complete the second task:
Image source: How to conduct UX Benchmarking
Both websites had negative recommendation scores. Most participants reported they would not recommend the sites to family and friends.
Advantages of Usability Benchmarking
- Puts you in continuous improvement mode.
- It helps you understand how much improvement you need to achieve significant performance.
- Enables you to outperform your competitors.
Disadvantages of Usability Benchmarking
- Enables you to measure the overall effectiveness.
- It puts your business at risk if you choose the wrong type of competitor.
Over to You
Choosing the most appropriate method to enhance the user experience on your website can be overwhelming.
Given the large number of methods out there, keep in mind that, no matter which one you choose, you need to make sure you have chosen the most feasible one for your situation.
Test for the right reasons, and you’ll have a good chance of achieving great outcomes. Do the opposite, and you’ll end up with some misleading insights, wasting hours, resources, and putting your whole business at risk.
