Usability Metrics: A Guide To Measuring The User Experience

Ayat Shukairy

Ayat Shukairy

My name is Ayat Shukairy, and I’m a co-founder and CCO at Invesp. Here’s a little more about me: At the very beginning of my career, I worked on countless high-profile e-commerce projects, helping diverse organizations optimize website copy. I realized, that although the copy was great and was generating more foot traffic, many of the sites performed poorly because of usability and design issues.
Reading Time: 9 minutes

We were hiring for a UX/UI senior position at Invesp. Of the 20 resumes we liked, we were only able to make 10 phone interviews. Of the 10 interviews, only one made it to a face-to-face.

The interviewee came in talking about all the great designs and apps he had created. But then we started drilling further, asking about some specific usability guidelines this individual follows, and he started to stumble. Then we asked; how do you measure the usability of a product you created?

Complete silence.

The basics of CRO lie in usability. Without it, we can’t design and innovate a superior experience for the visitor. Yet, designing and preparing an experience in the digital world and having more control over the user experience is no easy feat.

Whether you ‘re a marketer or a designer, the goal is to create a useful, enjoyable, and easy-to-use application for the visitor. Yet, website after website, application after application, we find those values are lost amongst their creators.

Conversion optimization and UX have soared in importance in the past couple of years. Companies are hiring experts within these fields to fix their processes and ensure that the experience for the end user is fully optimized and engaging. However, hiring and applying the standards are two different things. In our own experience, we find that many companies enjoy the concept and will hire to improve; however, they are sometimes unwilling to take that greater leap that will lead to a better experience. Very often, better UX and CRO means going against what the HIPPO thinks or wants.

Digital marketing provides a wide range of data collection and processing. If used correctly, it can help turn the company around. Very often, collecting data is the easy part, but processing, deciphering, and coming up with actionable insights are the challenges. Even the collection process can become cumbersome because you need to decide which data points are relevant to what you are trying to achieve.

A major part of our process is combining and collecting data about the visitors and their behaviors from a multitude of sources. We take qualitative data, such as user testing, and try to match the metrics and visitor behaviors in analytics to our testing to validate the information further in order to solve a usability problem.

But before delving into tools and how to use them, you need to understand what exactly usability is. What is the framework for better usability? And what are the metrics we look at to determine usability success? First, here are the fundamentals of the usability framework for greater context.

Usability Framework

Image source: UI-designer

As seen in the framework above, “usability is a multidimensional concept that aims to fulfill a certain set of goals, mainly effectiveness, efficiency, and satisfaction,” and without these goals, usability cannot be achieved.

  • Effectiveness: this term refers to the accuracy and completeness of the user goal achievement.
  • Efficiency refers to the resources exhausted by users in order to ensure an accurate and complete achievement of the goals.
  • Satisfaction refers to the subjective thoughts of the user regarding their attitude, level of comfort, relevance of application, and the acceptability of use.

 A system or a product is totally dependent on its specific and distinct context of use, the nature of the task, the users appointed to take the task, and finally, the equipment used to perform it.

Measuring the usability of a certain system can be done through the measurement of the three goals using a number of observable and quantifiable usability metrics.

In light of the three goals mentioned earlier, we’ll go through the different metrics used to measure each goal. However, our main focus will be on the success rate or the completion rate because it gives a general idea about the performance of the system.

Usability Metrics

Image Source: disciullodesign

Effectiveness

It can be measured by using two usability metrics: Success rate, also called completion rate, and the number of errors.

Success rate/ completion rate: the percentage of users who were able to complete the tasks successfully.

Despite the fact that this metric remains unable to provide insights on how the tasks were performed or why users fail in case of failure, they are still critical and are at the core of usability.

The success rate is one of the most commonly used metrics for most practitioners, where 79% of them reported using the success rate as the first metric to consider for ease of use and during data collection and interpretation.

Image source: measuringu

“The success rate metric can be measured by assigning a binary value of 0 and 1 to the users, where 1 is assigned to those who successfully complete the task and 0 to the ones who fail to do so.”

Once the test is over and you have all the data you need to calculate your success rate, the next step would be to divide the total number of correctly completed attempts by the total number of attempts multiplied by 100.

The completion rate is easy to measure and collect but has one major pitfall to consider; it happens frequently when a user stops at some point during the task and fails to finish it or even finishes it but not in the expected way.

Taking into account that they have completed some steps successfully in the task, how would you score what they have accomplished as an evaluator?

I am going to dive a little bit into the details of how to score your users, taking into account the different stages of their success or failure, using an example to illustrate.

Let’s consider, for instance, that your user’s task is to order a box of dark chocolates with a card to their mother for Mother’s Day.

The scoring might seem simple at first glance, and you can easily say that if the mother receives the box of dark chocolate with the card, then it is a case of success. On the other hand, if the mother does not receive anything, then we can simply say that this is a case of failure.

However, it’s not that simple. There are other considerations:

  • I ordered a box of chocolate but not the dark one (white or milky or a variety of these), along with the card.
  • Ordered the right chocolate box without a gift card
  • Ordered more than one box of chocolate by mistake, and a gift card
  • Ordered a box of chocolate but didn’t add delivery information or an address
  • Ordered a box of chocolates and a gift card successfully but to the wrong address

All these cases entail a percentage of success and failure in the process of fulfilling the task; their failure is partial as well as their success, and that simply means that as an evaluator, you need to engage your own personal opinion in the scoring.

If you decided that there is no middle ground in the estimated scoring, your success rate would be different from that obtained when you appreciate the effort they have made in spite of the task you planned for them.

The fact that there is no steady rule when it comes to scoring your users, and oftentimes, success rates become subjective because different evaluators won’t have the same scoring and estimate the same percentage of failure or success for the above cases in the same way. However, in order to mainstream the process, you need to determine the important aspects of the task and what score you would allot each part of it.

Success rate remains the simplest usability metric and the easiest among the whole range of these usability signals, mainly because it’s quick and easy and does not require much preparation and time to collect, and most importantly, it enables you to track the progress within your system.

However, being one of the general areas commonly used by marketers and designers all along to see the big picture of how well their system is doing at the level of user experience, this does not change the fact that it remains subjective.

2. The Number of Errors

This metric provides an idea about the average number of times where an error occurred per user when performing a given task.

These errors can be either slip, where the user accidentally types the wrong email address or picks the wrong dates when making a reservation or booking a flight, or they can be mistakes, where the user clicks on an image that’s not clickable or even double clicks a button or a link intentionally.

Normally, any user of any interactive system may make errors, where 2 out of every 3 users err, and there is absolutely no such thing as a ‘’perfect’’ system anyway. The rationale behind using this metric is not to eliminate errors but rather to lessen their numbers within the system.

To help you measure and ensure obtaining great diagnostic results, it is highly recommended to set a short description where you give details about how to score those errors and the severity of a certain error to show you how simple and intuitive your system is.

3. Time-Based Efficiency

Referred to as time on task, this metric helps in the measurement of the time spent by the user to complete the task or speed of work. Consequently, this means that there is a direct relationship between efficiency and effectiveness, and we can say that efficiency is actually the user effectiveness divided by the user time spent.

Image source: stock.adobe

In order to illustrate the calculation of the time-based efficiency, I am going to use an example to make it simple.

Let’s consider, for example, that we have 3 different users performing the same task, where 2 managed to complete it successfully in a considerable time -2,3 seconds respectively- while the third user took 7 seconds and never finished the task.

Now that we know that the number of tasks is N=1 and the number of our users is actually R=3, and we also know the time spent on each one of them, it becomes easy to determine the value of our Time-Based Efficiency.

N= Number of tasks (in this case N=1)

R= Number of users (in this case R=3)

Nij = The result of task i by user j; if the user successfully completes the task, then Nij = 1, if not, then Nij = 0

Tij = The time spent by user j to complete task i. If the task is not successfully completed, then time is measured till the moment the user quits the task

Using the time-based efficiency equation, we end up with something like this:

User 1: Nij = 1 and Tij = 2
User 2: Nij = 1 and Tij = 3
User 3: Nij = 0 and Tij = 7

Time-based Efficiency = (1/2+1/3+0/7)/1*3

= 0,71 goals/sec

4. The Overall Relative Efficiency

This is actually measured through users who successfully completed the task in relation to the total time taken by all users.

Let’s consider that we have 2 users, and each one is supposed to complete a different task.

The first user has successfully completed the task (1) yet failed to complete the task (2). The second user failed to complete task (1) but completed task (2) successfully.

In this case, the overall efficiency can be calculated as the following:

            The Overall Relative Efficiency = E = ((1*1+1*1)/ (2*2)) *100% = 50%

5. Post Task Satisfaction

Once your users have finished the task, and it doesn’t matter whether they complete it successfully or not, it’s time to hand them over a questionnaire to have an idea about the difficulty of the task from the user’s point of view.

Generally, these tasks consist of 5 questions, and the idea behind them is to give your users a space to judge the usability of your system.

6. Task Level Satisfaction

This metric helps investigate the general impression of users confronted with the system. To measure the level of satisfaction, you can use the smiley scale method, where the user is expected to choose one of the 5 smileys as a reflection of their satisfaction or lack of satisfaction.

Image source: Smiley Scale

The Word Method is also use to measure the user’s level of satisfaction through listing a series of positive and negative connotations highlighted in green and red respectively. As shown in the graph below.

Image source: UI-designer

Conclusion

In light of the conceptual framework we have discussed earlier, the user experience is highly influenced by everything that surrounds it.

In the digital world, especially in the context of user experience and the usability of a system, it’s all about how accurate and exact your numbers and stats are.  And because there is so much data and variables that are completely out of control, usability metrics represent a great approach to compromise and combine these data in a quantitative and qualitative way and help you understand what you are doing against your system’s objectives and goals to provide a better user experience finally.

Once you have collected the metrics, it’s time to use them to form a conclusion about the overall usability of your system and make sound and data-driven decisions about the future changes within your system for a better user experience.

Share This Article

Join 25,000+ Marketing Professionals!

Subscribe to Invesp’s blog feed for future articles delivered to receive weekly updates by email.

Ayat Shukairy

Ayat Shukairy

My name is Ayat Shukairy, and I’m a co-founder and CCO at Invesp. Here’s a little more about me: At the very beginning of my career, I worked on countless high-profile e-commerce projects, helping diverse organizations optimize website copy. I realized, that although the copy was great and was generating more foot traffic, many of the sites performed poorly because of usability and design issues.

Discover Similar Topics