• The 7 Step Roadmap For Effective Usability Testing

    • Posted in UX
    a couple looking at a map for a better travel experience, depicting a roadmap for better usability testing on a website.

    How often do you conduct a usability test to realize middle-way through you should have taken a different route?

    I wouldn’t be surprised if the answer was very often!

    Conducting a usability test is by no means cut and dry, and there are many methodological pitfalls that can turn the whole test into a flop if not conducted properly. It is also an important step when conducting a conversion optimization project.

    With the different kinds of usability testing, you should keep an eye on the issues that may rise along the way.

    This article will focus on answering questions such as:

    • How do I get started with usability testing?
    • What are the steps I need to follow to obtain valid results?
    • How do I make sure not to miss anything?
    • What are some of the pitfalls that I need to avoid?
    • How do ensure that I will collect valid and significant results?

    Over the past years, usability studies have undergone some tremendous changes. They are now more sophisticated yet easier to run, offering more actionable insights and better results to improve user experience.

    Before diving into the know-hows, keep in mind that every usability test is different in terms of their goals and their requirements. However, these tests do share common elements and can follow the same overall roadmap.

    Let’s begin with the planning phase.

    1. Define Your Goals

    Defining the goals of your study is probably the most crucial part of the process.

    You should have a clear result for the study and decide whether you want to gather:

    1. Qualitative, or
    2. quantitative data, or
    3. address issues related to the design of your system.

    Having a clearly defined goal limitation is key.

    The more focused your tests are, the more accurate and relevant the results will be. On the flip side, the more questions you commit to, the more room there is for errors and misinterpretations.

    The scope of your test could be broad and addresses general questions such as:

    • Which navigation page works best and provides a more user-friendly experience?
    • How users are currently using my app?

    Or even more specific questions, such as:

    • What exactly is stopping users from proceeding to checkout?
    • Which element on checkout attracts users’ attention the most?

     2. Determine and Recruit Users

    The greatest and most useful insights derive from real users. Make sure you know your users well, so you can recruit the right participants.

    The users to be included in a usability test must match closely your website visitors and your target market. There are instances where website visitors do not match your target market. In most cases, recruiting website visitors for a usability test (when possible) is the best option.

    To get representative participants, you must know who the users of a current system are, or who the targeted users of a newly developing system will be. Only participants who truly represent the actual users can provide the kind of valid feedback you need to make meaningful improvements to a design.

    Deborah Hinderer Sova and Jakob Nielsen

    Image source: measuringu

    Decide on the number of participants you need to recruit and how many you can afford within your budget. The number of participants can affect drastically the results of your test.

    We typically like to limit the number of users within a usability test to anywhere from 3 to 8 users.  Some companies recommend conducting larger scale usability tests including 50 to 150 users.

    Years of experience have shown us that there is marginal benefit from adding more users.

    3. Choose the Format of Your Test

    There are different formats for conducting the usability test.

    In-Person Usability Testing

    The Silent Observer or Natural: the observer stands silently behind the users and doesn’t interact with them in any sort of way to be able to analyze their behavior in their own environment.

    Think Out Loud Method: this method might be uncomfortable for users since they describe everything they do or think of out loud, but it is extremely insightful and useful for the experimenters. The think out loud method allows you as an experimenter to have a clear vision about what’s going on inside the users’ mind as every part of their thought process is articulated.

    Constrictive Interaction Method: using this method you can either have two users working together and helping each other completing the task, or you can ask questions and offer suggestions to them without helping them or guiding them to complete the task. This method allows interaction between the users and the experimenters and you can see how they work through the hurdles they face during the test.

    Remote Usability Testing

    Remote usability testing can be an alternative for in person testing as it is time and cost effective and the users can complete tasks in their natural environment.

    Moderated: these sessions allow the interaction between the observer and the test user. It is somehow similar to in-person testing as both the moderator and the participant are online simultaneously.

    Unmoderated: these testing sessions are completed by the participants independently behind their screens, without an interaction with the test moderator, and there is no opportunity for the experimenters to ask questions or have any sort of involvement during the task even if needed.

    FigPii Heatmaps

    Editor’s Note: We highly recommend in-person usability testing as they allow you to read the users’ body language and be in contact with them in real life rather than trying to infer their movements and watch them remotely.

    4. Define the Tasks Within the Test

    Your users’ responses and reactions are impacted by almost everything you present to them during the test, so you have to be careful not to create biased tasks for your participants to complete.

    Tasks may either open/exploratory or closed/specific, and finding a middle ground between these two extremes would provide a healthy mixture for users. You can suggest scenarios for the participants, or direct provide the tasks for them to take.

    Open/Exploratory Tasks: tend to be more broad, research-oriented. In this case, you provide participants with the minimal amount of information on how to perform the task.

    These kind of tasks may have more than one answer as the answers vary from one users to another.

    Closed/Specific Tasks: are more focused and goal-oriented. As they have an end point, they limit the scope of interpretation and confusion for the users and produce more adequate and accurate results.

    Characteristics of Good Usability Tasks

    When building tasks, follow these guidelines to create meaningful results:

    Propose a task not a question

    Avoid making the task look like a question rather than an actionable task that users are supposed to do.

    A good task is NOT a question which participants are supposed to answer. A bad task is asking participants direct questions with predictable answers.

    Make sure to choose the most meaningful tasks within your system, according to the goal you set for the test.

    Be specific

    Specificity is an essential component of your tasks. Do not leave users with questions in mind, wondering what they are supposed to do and what he next step is.

    A good task is neither vague nor tiny, you want to make sure that you are not focusing on tiny and miner features and make main goals in your task and should cover and highlight only the important features of your system.

    Avoid instructions

    Going to the other extreme and start giving instructions and indications to users to get the test right is another mistake.

    By doing so, you ruin your test. Instead of receiving accurate and real results, you will obtain the results you want. Biased results will not help you in assessing your system usability.

    Give participants space

    The core of testing a system usability relies on giving the users space and providing similar conditions to the real use of the system. Within similar conditions, you can assess participants’ frustration, struggle, and relief.

    Editor note: a best practice we follow is to create the steps within a test and then conduct one remote usability test using these tasks. That will help us uncover any problems in the design of our tasks. It also allows us to refine our process.

    5. Build Questionnaires

    You can use two main types of questionnaire in usability tests.

    Pre-test questionnaires: these questions help you understand your users better. With 7 to 10 questions on relevant background information, as users’ age, gender, educational background, beliefs, and their experience with the system (whether a novice or an expert), you can collect important data to evaluate how a target audience engages with your product. Keep in mind that these questions should have a direct relationship to your users, the functionality of the tasks chose, and the goal you set for your test.

    Here is an example of the kind of questions you might ask in the pre-test questionnaires:

    How often do you shop online? (i.e. Amazon, EBay)

    • Never
    • I shopped once or twice last year
    • I shop approximately 3 to 7 times per year
    • I shop regularly (How often?) ________

    Post-test questionnaires: these questions will also help you understand the experience which the participants went through as they completed the tasks.  You can ask participants to give their feedback on how easy or difficult it was to use the system, and how much they enjoyed or disapproved the experience. Keep in mind that you should give your users space to judge the usability of your system.

    A good example would be the following question:

    How did you find the application?

    • Easy to use
    • Hard to use

    Can you explain why? ______________

    6. Try a Dry Run

    Rehearsing for the study is crucial to make sure that everything is ready and to avoid any kind of complications during the real test.

    Running a pilot test will help you estimate the time necessary to complete the real test. You can make a schedule for each task, in case of many, and prioritize them in accordance to their importance and relevance to the main goal of the study.

    Through conducting a pilot test, you assure that the real test will run smoothly, without problems or major surprises. The dry run is also an opportunity to test if the tasks are clear to the participants.

    We often run a pilot test with our team members. One or two pilot users are more than enough to detect severe problems and prepare ourselves for the real test.

    These tests usually give us different perspectives on how users engage with a website. Just because it is a pilot test, it doesn’t mean you need to dismiss any findings. If everything goes well, the pilot test can also provide meaningful data to your study. Pilot tests can reveal interesting, unexpected findings on dry runs.

    Editor note: When conducting an internal pilot test, make sure to include users who are not familiar with the system you are testing.

    7. Report the Findings

    If all goes well, after you finish your usability test, it’s time to gather and compare your findings.

    There is no specific or single method to analyze your data. You can choose and agree with your team on whatever method works for you. For example, you can summarize the difficulties encountered by each user and then gather the common ones.

    You can also classify these problems, if any, and rate them in accordance with their severity or importance, to determine which ones to address first and which to drop out of your list.

    Based on the findings, you’ll be able to spot problem areas within the system and create solutions that will make it more useful.

    Case Study: The Usability Test in Action

    To illustrate the steps above, let’s take a look at the usability test we conducted for large e-tailer that provides digital photo finishing solutions and customized photo products.

    Goal: The main focus of the study was to find answers for the following design questions and make informed changes while redesigning particular sections on the website.

    • how easy it is to locate a particular photo product?
    • how easy it is to customize it and add to cart?

    The users: The study was conducted with 6 users. After completing the tasks, they were asked to answer follow-up questions.

    The format: Users were using Chrome browsers because the application only supports that, and they were required to speak throughout their experiences as we have chosen the think out loud method in this test.

    The scenarios and tasks: You would like to create a customized baby photo gift. You have different photos that you think will be lovely for some kind of wall decor in your bedroom.

    Task 1: Find a customizable a wall decor photo you like and can imagine in the bedroom.

    Task 2: Create 3 different customization options for your photo.

    Task 3: Add the 3 different customized items to your cart.

    Task 4: Create an invitation for a baby meet and greet using 2 of the customization you just created.

    Task 5: Complete your purchase

    Questionnaire and findings

    Question 1: How would you rate your overall experience on this site?”

    • 5 users rated the experience as fair and excellent.
    • Only 1 user rated the experience as poor.

    Question 2: What would you say is the most frustrating aspect on the website?

    Main Comments of the participants:

    • Builder functionality problems
    • There were several website glitches
    • Slowness of the builder

    Question 3 : What would you say is the most frustrating aspect on the photo builder?

    FigPii Heatmaps

    Main Comments of the participants:

    • Navigating the builder
    • No guidance or informative indicator about process
    • Users could not find CTAs

    Question 4: What is the first thing we should improve on the site?

    Main Comments of the participants:

    • Improve or change the builder
    • Highlight next steps
    • Add more design and filter options to the builder

    This testing revealed problems in the following areas:

    1. Builder functionality
    2. Builder “Add to Cart” and “Save Project” CTAs
    3. Unclear user flow
    4. Overlooked functionalities

    Over to You

    Preparation is key when it comes to the validity and significance of usability studies. Defining the scope of the test and what you are trying to accomplish to start with.

    Note that words are powerful and it takes only one wrong word to change the whole task conception and only a  good one to correct it. Hence, it’s fundamental to be careful and avoid bias when creating your tasks.

    It’s highly advisable to conduct a pilot test before having any test users take the real one.

    Be open minded and get ready to be surprised, because at this stage you can come across findings that you never guessed or thought about.

Khalid Saleh

Khalid Saleh is CEO and co-founder of Invesp. He is the co-author of Amazon.com bestselling book: "Conversion Optimization: The Art and Science of Converting Visitors into Customers." Khalid is an in-demand speaker who has presented at such industry events as SMX, SES, PubCon, Emetrics, ACCM and DMA, among others.

View All Posts By Khalid Saleh
Khalid Saleh

Join 25,000+ Marketing Professionals

If you enjoyed this post, please consider subscribing to the Invesp blog feed to have future articles delivered to your feed reader. or,receive weekly updates by email: