Usability Metrics: A Guide To Measuring The User Experience
We were hiring for a UX/UI senior position at Invesp. Of the 20 resumes we liked, we were only to make 10 phone interviews. Of the 10 interviews, only one made it to a face to face.
The interviewee came in talking about all the great design and apps he had created. But then we started drilling further asking about some specific usability guidelines this individual follows and he started to stumble. Then we asked; how do you measure the usability of a product you created?
The basics of CRO lie in usability. Without it, we can’t design and innovate a superior experience for the visitor. Yet, designing and preparing an experience in the digital world and having more control over the user experience is no easy feat.
Whether you’ re marketer or a designer, the goal is to create a useful, enjoyable and easy to use application for the visitor. Yet, website after website, application after application, we find those values are lost amongst their creators.
Conversion optimization and UX have soared in importance in the past couple of years. Companies are hiring experts within these fields to fix their processes and ensure that the experience for the end user is fully optimized and engaging. But hiring and applying the standards are two different things. In our own experience, we find that many companies enjoy the concept and will hire to improve, however are sometimes unwilling to take that greater leap that will lead to the better experience. Very often better UX and CRO means going against what the HIPPO thinks or wants.
Digital marketing provides such a wide range of collecting and processing data. If used correctly can help turn the company around. But very often, collecting data is the easy part, but processing, deciphering and coming up with actionable insights is the challenge. And even collection process can become cumbersome, because you need to decide which data points are relevant to what you are trying to achieve.A major part of our process is combining and collecting data about the visitors and their behaviors from a multitude of sources. We take qualitative data such as user testing, and try to match the metrics and visitor behaviors in analytics to our testing to validate the information further in order to solve a usability problem.
But before delving into tools and how to use them, you need to understand what exactly is usability? What is the framework for better usability? And what are the metrics we look at to determine usability success? First, here are the fundamentals of the usability framework for greater context.
Usability FrameworkImage source: ui-designer
As seen in the framework above, “usability is a multidimensional concept that aims into the fulfillment of certain set of goals, mainly; effectiveness, efficiency and satisfaction” and without these goals, usability cannot be achieved.
- Effectiveness: this term refers to the accuracy and completeness of the user goal achievement.
- Efficiency: refers to the resources exhausted by users in order to ensure an accurate and completed achievement of the goals.
- Satisfaction refers to the subjective thoughts of the user regarding their attitude, level of comfort, relevance of application and the acceptability of use.
A system or a product is totally dependent on its specific and distinct context of use, the nature of the task, the users appointed to take the task, and finally the equipment used to perform it.
Measuring the usability of a certain system can be done through the measurement of the three goals using a number of observable and quantifiable usability metrics.
In the light of the three goals mentioned earlier, we’ll go through the different metrics used to measure each goal, however, our main focus will be on the success rate or the completion rate because it’s gives a general idea about the performance of the system.
Usability MetricsImage Source: disciullodesign
It can be measured through using two usability metrics: Success rate, called also completion rate and the number of errors.
Success rate/ completion rate: is the percentage of users who were able to successfully complete the tasks.
Despite the fact that this metric remains unable to provide insights on how the tasks were performed or why users fail in case of failure, they are still critical and are at the core of usability.
The success rate is one of the most commonly used metric for most of practitioners, where 79% of them reported using the success rate as the first metric to consider for ease of use and during data collection and interpretation.Image source: measuringu
“The success rate metric can be measured by assigning a binary value of 0 and 1 to the users; where 1 is assigned to those who successfully complete the task and 0 to the ones who fail to do so.”
Once the test is over and you have all the data you need to calculate your success rate, the next step would be to divide the total number of correctly completed attempts by the total number of attempts multiplied by 100.
The completion rate is easy to measure and to collect but with one major pitfall to consider; it happens frequently when a user stops at some point during the task and fails to finish it or even finishes it but not in the expected way.
Taking into account that they have completed some steps successfully in the task, how would you score what they have accomplished as an evaluator?
I am going to dive a little bit into the details on how to score you users taking into account the different stages of their success or failure, using an example to illustrate.
Let’s consider, for instance, that your user task is to order a box of dark chocolates with a card to their mother for mother’s day.
The scoring might seem simple at first glance, and you can easily say; if the mother receives the box of dark chocolate with the card then it is a case of success. On the other hand, if the mother does not receive anything then we can simply say, that this is a case of failure.
However, it’s not that simple, there are other considerations:
- Ordered a box of chocolate but not the dark one (white or milky or a variety of these) along with card.
- Ordered the right chocolate box without a gift card
- Ordered more than one box of chocolate by mistake and a gift card
- Ordered a box of chocolate but didn’t add delivery information or address
- Ordered a box of chocolates and gift card successfully but to the wrong address
All these cases entail a percentage of success and failure in the process of fulfilling the task, their failure is partial as well as their success, and that simply means that as an evaluator you need to engage your own personal opinion in the scoring.
If you decided that there are no middle grounds in the estimated scoring, your success rate would be different from that obtained when you appreciate the effort they have made in spite of the task you planned for them.
The fact that there is not a steady rule when it comes to scoring your users, and oftentimes success rates become subjective; because different evaluators won’t have the same scoring and estimate the same percentage of failure or success for the above cases, in the same way. However, in order to mainstream the process, you need to determine the important aspects of the task and what score you would allot each part of it.
Success rate remains the simplest usability metric and the easiest among the whole range of these usability signals, mainly because it’s quick and easy and does not require much preparation and time to collect and most importantly it enables you from tracking the progress within your system
However, being one of the general areas commonly used by marketers and designers all along, to see the big picture of how well their system is doing at the level of user experience, this does not change the fact, that it remains subjective.
2. The Number of Errors
This metric provides an idea about the average number of times where an error occurred per user when performing a given task.
Image source: Invesp
These errors can be either slips; where the user accidently types the wrong email address or picks the wrong dates when making a reservation or booking a flight, or they can be mistakes where the user clicks on an image that’s not clickable or even double clicks a button or a link intentionally.
Normally any users of any interactive system may make errors, where 2 out of every 3 users err, and there is absolutely no such thing as a ‘’perfect’’ system anyway. The rationale behind using this metric is not to eliminate errors but rather to lessen their numbers within the system.
To help you measure and ensure obtaining great diagnostic results, it is highly recommended to set a short description where you give details about how to score those errors and the severity of a certain of an error to show you how simple and intuitive your system is.
3. Time-Based Efficiency
Or referred to as time on task, this metric helps in the measurement of the time spent by the user to complete the task or speed of work. This consequently means that there is a direct relationship between the efficiency and effectiveness, and we can say, that efficiency is actually the user effectiveness divided by the user time spent.
Image source: stock.adobe
In order to illustrate the calculation of the time-based efficiency, I am going to use an example to make it simple.
Let’s consider, for example, that we have 3 different users performing the same task, where 2 managed to complete it successfully in a considerable time -2,3 seconds respectively- while the third user took 7 seconds and never finishes the task.
Now that we know that the number of task is N=1 and the number of our users is actually R=3, and we know also time spent of each one of them, it becomes easy to determine the value of our Time-Based Efficiency.
N= Number of tasks (in this case N=1)
R= Number of users (in this case R=3)
Nij = The result of task i by user j; if the user successfully completes the task, then Nij = 1, if not, then Nij = 0
Tij = The time spent by user j to complete task i. If the task is not successfully completed, then time is measured till the moment the user quits the task
Using the time-based efficiency equation, we end up with something like this:
User 1: Nij = 1 and Tij = 2
User 2: Nij = 1 and Tij = 3
User 3: Nij = 0 and Tij = 7
Time-based Efficiency = (1/2+1/3+0/7)/1*3
= 0,71 goals/sec
4. The Overall Relative Efficiency
This is actually measured through users who successfully completed the task in relation to the total time taken by all users.
Let’s consider that we have 2 users where each one of is supposed to complete a different task.
The first user has successfully completed task (1) yet failed to complete task (2). While the second user has failed to complete task (1) but completed task (2) successfully.
In this case the overall efficiency can be calculated as the following:
The Overall Relative Efficiency = E = ((1*1+1*1)/ (2*2)) *100% = 50%
5. Post Task Satisfaction
Once your users have finished the task and it doesn’t matter whether complete it successfully or not, it’s time to hand them over a questionnaire to have an idea about the difficulty of the task from the users point of view.
Generally, these tasks consist of 5 questions, and the idea behind them give your users a space to judge the usability of your system.
6. Task Level Satisfaction
This metric helps into investigating the general impression of users confronted with the system. To measure the level of satisfaction you can either use the smiley scale method where the user is expected to choose one of the 5 smileys as a reflection of their satisfaction or lack of satisfaction.
Image source: Smiley Scale
The Word Method is also use to measure the user’s level of satisfaction through listing a series of positive and negative connotations highlighted in green and red respectively. As shown in the graph below.
In light of the conceptual framework we have discussed earlier, the user experience is highly influenced by everything that surrounds it.
In the digital world, especially in the context of user experience and the usability of a system, it’s all about how accurate and exact your numbers and stats are. And because there is so much data and variables that are completely out of control, usability metrics represent a great approach to compromise and combine these data, in a quantitative and qualitative way, and help you understand what you are doing against your system’s objectives and goals to finally provide a better user experience.
Once you have collected the metrics, it’s time to use them and form a conclusion about the overall usability of your system and make sound and data driven decisions about the future changes within your system for a better user experience.
My name is Ayat Shukairy, and I’m a co-founder and CCO at Invesp. Here’s a little more about me: At the very beginning of my career, I worked on countless high-profile e-commerce projects, helping diverse organizations optimize website copy. I realized, that although the copy was great and was generating more foot traffic, many of the sites performed poorly because of usability and design issues.View All Posts By Ayat Shukairy
Join 25,000+ Marketing Professionals
If you enjoyed this post, please consider subscribing to the Invesp blog feed to have future articles delivered to your feed reader. or,receive weekly updates by email:
Connect with us
The Art and Science of Converting Prospects to Customers
By Khalid Saleh and Ayat Shukairy
- Project Management for CRO – Process, Software, and Resource Allocation
- How to Leverage The Price Anchoring Effect (With Examples)
- Expert Answers to 9 Tough CRO Questions
- What they don’t tell you about A/B testing velocity
- Google Optimize: The Good, the Bad, and the Ugly
- How to Use Data to Identify Problems on Form Fields
- The Importance of Customer Lifetime Value In eCommerce
- Optimizing Conversion Funnels: Where Should You Start First?
- How to Build a Brand Community from Scratch
- Should you hire a CRO Agency, Or should you use CRO Tools?