Writing open-ended and closed-ended questions for user research
- Posted in UX
Editor Note: We highly recommend that you implement the different ideas in this blog post through AB testing. Use the guide to conduct AB testing and figure out which of these ideas in the article works for your website visitors and which don’t. Download Invesp’s “The Essentials of Multivariate & AB Testing” now to start your testing program on the right foot.
So you have decided to conduct a usability test for your product —so as to understand your product from the users’ perspective. You then realize that you have to come up with relevant questions to ask participants.
“That sounds like a piece of cake,” some may presume.
Writing effective usability questions seems deceptively easy, but the harsh reality is it’s not as simple as you may think it is. There are quite a number of mistakes to avoid.
How you phrase your questions will directly impact the quality and value of your user testing results. Ask your participants wrong questions, or use the wrong words to structure the questions, and you will gather incorrect feedback. Wrong questions can contaminate the whole research —leading to misleading insights.
Before you come up with any usability questions, there is a great need to think of what you intend to learn from the test. Begin by asking yourself this: what information do I need from this research?
Taking time to answer that question will help you narrow down all the possible wrong directions you might head before getting to the insights you need. The ultimate goal could be as simple as finding out if users will click on your search result listing.
When you have a clearly defined goal, writing usability questions isn’t a task hard to do. By default, your usability questions can either be open-ended or closed-ended.
Whenever we conduct any user research, at Invesp, our conversations with our participants have a natural rhythm. We ensure this by using open-ended and closed-ended questions in unison. This article will explore open and closed-ended questions in great detail, focusing on how to write them and when to use them.
Defining open-ended questions
An open-ended question is what it is: a question that is open to any answer. In the context of user research, open-ended questions are questions that do not limit users to one- or two-word answer. Instead, they have multiple potential responses, and they often give room for further probing by the moderator.
Open-ended questions are versatile in nature, and they prompt users to describe their feelings and thoughts in their own voice. In this regard, the Digital Marketing Evangelist for Google, Avinash Kaushik says:
The greatest nuggets of insights are in open-ended questions because it is the Voice of the Customer speaking directly to you (not cookies and shopper_ids but customers).
There is something about asking open-ended questions that makes participants feel more comfortable during a usability test. People, in general, tend to open up and express themselves better when they are given room to answer in their own words.
For example, rather than asking, “Was the new feature easy to use?” You can try something like, “How would you describe your experience of using the new feature?”
The most common response to the first question would have been, “Yes, it was ” or “No, it wasn’t” and there was no way you could have understood the context behind the user’s response. But the second question allows users to respond freely, and there is a high chance of getting unique answers that you might not have anticipated.
Open-ended questions are ideal for starting and holding a conversation in any circle. They empower users, giving them 100% control of what they intend to say —and this is something that cannot be done using closed ended-questions.
Tips for writing effective open-ended questions
Coming up with the right kind of questions requires practice. There’s an art to asking questions that prompts people to think before giving a responding. Unless it’s an open-ended question, not every question you ask your users will motivate to give a detailed answer. Here are some few tips that will help you write effective open-ended questions.
Begin your question with how, why, and what
What makes a question an open-ended is the wording of the sentence. For an effective open-ended question, start the question with words such as how, what, why, and can. This way, you give your users freedom to say more and in the process, there is a possibility of uncovering rich insights.
Avoid using more specific words such as did, would, which, when and was —these usually prompt one-worded answers.
For example, a question like “Which part of your experience was unsatisfactory?” does not evoke users to give in-depth details about what they found unsatisfactory with the application. Instead, you can elicit users to reflect on their experience and give a more insightful response if you ask it this way “What challenges did you face during your experience with XYZ?”.
Here are a couple more examples of how to attain a more precise answer by changing the wording of your question:
Closed-ended Questions Open-ended Questions Was our product easy to use? What do you think about our product? Did you enjoy using our product? How would you describe the experience you had using our product? Did anything persuade you to use our product? What persuaded you to use our product among other products?
Clarity and Analysis
Ensure that your question requires users to be analytical and to clarify their points. One defining trait of an open-ended question is its ability to propel users to put more thought to their responses.
Analytical questions do not require users to generalize their answers. For instance, in a Conversion Optimization research, you might ask your participants these questions:
- What is the significance of a certain element in a website?
- How important is the new feature on the site?
- Why did you choose to use this service/product?
All these three questions are different, but they all have one thing in common: they would require participants to be clear and describe their answer in more detail.
By motivating users to give clarity in their responses, open-ended questions can also be used as a way of motivating their participants to verify their answers, especially when your previous question was a closed-ended.
Suppose you ask this closed-ended question: “Did you find the product you were looking for?” you can then verify the given answer by asking this follow up open-ended question: “Why were you looking for that product?”
Avoiding leading participants into a certain answer
In any research approach, open-ended questions are asked so as to elicit valuable insights from users not to confirm the moderator’s existing beliefs. So, if your questioning subtly prompts users to answer in a certain way or gives hints at the expected answer, then you need to revise the phrasing of your questions.
The wording of the question shouldn’t be suggestive of any answers to the participants as this biases the users into giving a predetermined answer.
Let’s say you ask this question: “Which feature made you visit our site?”
The problem with this question is that already suggests an answer for the users. It implies that it’s a feature that made users visit the site. Come to think of it, what if it wasn’t a feature but a service that lured the users to the site?
When to use open-ended questions
In some situations, the only way to get valuable insights is to give respondents some sense of control over the conversation by allowing them to answer in their own words. There is a high chance of bumping into something completely unique and valuable if you allow users to have the freedom to express themselves.
Open-ended questions in sales
SalesHacker made an interesting observation about interactive discussions prompted by open-ended questions: when you have a conversation with your potential customers and they talk for at least 30% of the time, your conversion sales are likely going to increase. But if they talk for less than 30% of the time, your sales conversion rates will drop.
With open-ended questions, you are not only guaranteed an increase in sales, but they can also help you:
- Explore the needs of your customers.
- Provide you a better idea of what your customers think about your product.
- Foresee and minimize risks.
- Trigger a meaningful and insightful conversation with your customers.
- Discover new opportunities.
- They can play a significant role in building a good rapport with your customers.
However, not all open-ended questions are good. In an article written by Business 2 Community, they gave examples of some of the “bad open-ended questions” that won’t work well in sales:
- How much are you willing to spend?
- What is your worst pain?
- What kind of goods or services are you ready to pay for
- What don’t you like about our service?
- What don’t you like about our service?
Open-ended questions in conversion optimization research
The essence of an effective CRO program is not only based on getting tactics right and testing this and that, but it’s also about knowing the mindset of your customers. In order to deliver a product or the services they desire, you first have to see your product or service from the customer’s lenses.
One way of getting into the customer’s head is by asking open-ended questions.
In this regard, JeremySaid put out a handy list of questions you can ask your current customers when you intend to increase your conversion rate. Here are some of the open-ended questions they recommended:
- How was your overall experience?
- Why are you here today?
- What about this product/service struck out to you?
- What do you know about our company?
- What would you like to know about our company?
- What problems have experienced in the past with similar products?
- What would you like to see us do more online?
The more you let your customers feel comfortable, the more they will reveal what drove them to consider purchasing your product. So, whether you are conducting a usability test, focus group, customer interviews, or surveys, keep your customers TALKİNG and use the information to your advantage. And the best way of doing this is opening up the conversation.
Defining closed-ended questions
If you can imagine a question that restricts participants to a set of predefined answers, then that’s a closed-ended question. It aims to get precise and clear-cut answers — without leaving any room for users to express themselves.
According to Wikipedia:
A closed-ended question refers to any question for which a researcher provides research participants with options from which to choose a response. Closed-ended questions are sometimes phrased as a statement which requires a response. A closed-ended question contrasts with an open-ended question, which cannot easily be answered with specific information.
Asking closed-ended questions will give you specific answers. Do they want to purchase your product? Are they shopping around with your competitors for the same service? The answers are simple and direct.
Although closed-ended questions provide limited insights, that doesn’t make them any less important. In most cases, closed-ended questions are used in a quantitative research approach where insights gathered are numerical.
Closed-ended questions have their place in user research and they are wonderfully effective in guiding participants into giving certain answers.
For example, let’s say you conduct a usability test to find out if your app store listing will convert well. So you ask your participants this question:
Which of the following piece of information made you download this app on play store?
- App Icon
- Customer reviews
From this example above, the question eliminates any element of surprise by setting boundaries for the participants’ responses. Participants are not expected to give an answer outside the set of predefined responses.
So, if closed-ended questions do not require participants to express themselves, then isn’t that a disadvantage? Well, it is. Respondents are biased into responding in a certain way. But sometimes it’s necessary to use these types of questions in user research as they make it easier and quicker for respondents to answer.
Although closed-ended questions have different forms, they all have this in common: they are similar in the kind of an answer they draw out from the respondents —clear-cut answers.
Specific questions are precise, clearly defined and they leave no room as to the intended meaning. At times, they come as multiple-choice questions that consist of two sections: (1) the stem which is the question itself and (2) a list of response alternatives, choices or answers that respondents will have to select an answer from.
Example: Suppose you want to evaluate your marketing channels and find out which platform is your brand more visible in. So you ask your participants:
How did you first learn about our product/website?
As alternative responses to the question, you can give these to your participants:
Among all kinds of questions, specific or multiple-choice questions are considered to be the most versatile type of questions. In user research, you can use specific questions to discover facts or to gain an understanding of user behavior.
If anything is said to be implicit, then it means that it is not directly pointed out, but it is somehow suggested in the statement. So, an implicit statement is as an expression that prompts a certain reaction.
With that said, what then is an implicit question?
An implicit question can be defined as a leading question that gives hints about the type of answer needed. Think of them as leading questions, that pushes participants to respond in a specific manner.
Example: If you ask users this question: How many times do you visit our website?
Using “how many times” in the above question implies that the participants have visited the website before. There is an element of conjecture and assumptions and in this case, participants are persuaded to give a numerical answer.
If the question was to be phrased as a direct question, it would have been:
Have you ever visited our website?
This second question doesn’t influence the participants’ responses —and this means that the question doesn’t cultivate any biases in respondents.
Tips for writing effective close-ended questions
Closed-ended should not always be thought of as a simple question that anyone can easily answer merely because they do not require a detailed answer.
Begin sentences with Where, Which, When, Did
To make a question a closed-ended, there is a certain way you should phrase it. In his book, Conversationally Speaking, Alan Garner suggests that you use these few words to begin closed-ended questions:
Using these words, here’s a list of some examples of closed-ended questions you can use:
- Are you happy with your experience when using our site?
- Would you recommend our product/service?
- What challenges did you face when you were using this website?
- Which elements on our mobile applications were easy to use?
All these closed-ended questions do not prompt participants to give detailed answers. They all can be answered with a one-word answer, as they aim to find out the ‘what’ and not the ‘why’.
Be clear and simple
Needless to say, when you ask a clear and simple question you allow the possibility of a clear-cut answer. So the starting point is to remove extra verbiage that may end up distracting or confusing the respondents.
Good example: Would you recommend our website?
Bad example: You have used the website for more than 10 minutes and you have visited all the pages and clicked on all, so does it mean that you will recommend our website to other people?
Once you frustrate users with wordy questions, you risk compromising the value of your feedback. Here’s a list of clear and simple closed-ended questions as given by Hotjar:
- Did you experience good customer service?
- Would you consider using our product or service again?
- Did you like our product or service?
- What product or service were you looking for today?
- Are you happy with your experience with us?
- Did you find what you were looking for today?
Having clear and simple closed-ended questions, you won’t just make it easy for users to infer the intended meaning, but you, as a moderator, will understand the given answers without any hassles.
Relevant answer choices
If you intend to use multiple-choice questions, then make sure that your suggested answers are plausible. Participants usually have different experiences even after using the same product, and this is why you should have several alternative answers that best describe answers.
Use at least four alternatives in each multiple-choice question so as to give users a variety of alternatives.
When to use closed-ended questions
Generally, an online poll can have both closed and open-ended questions as long as they require short feedback. But to give users a simple experience, you can use closed-ended questions as they are easy to answer and do not require a detailed answer. Since they require limited answers, Susan Farrell from Nielsen Norman Group says this:
Closed-ended questions are often good for surveys, because you get higher response rates when users don’t have to type so much. Also, answers to closed-ended questions can easily be analyzed statistically, which is what you usually want to do with survey data.
Here’s an example of one of the closed-ended questions we use on the FigPii polls. The question simply asked users the reason for their visit on the site and it allowed them to select one answer from four alternatives.
Such closed-ended questions which have multiple-choice form have higher completion rates as users do not have to come up with their own responses. Although the answers given provide a general sentiment of insights, you can always follow up with an open-ended question so as to see things from the users’ perspectives.
For instance, you can ask users to further elaborate their answer by asking this as a follow up:
What is the most important feature of our product/service for you? This way you can understand the context behind the users’ decisions.
Considering that they don’t demand much explanation from the respondents, closed-ended are perfect in quantitative usability research where you’d need to measure usability metrics such as task completion rates, error rates, and post-task satisfaction.
Insights gained using closed-ended questions allows researchers to categorize respondents based on the answers they have selected. How so?
Let’s say you have an online store and you need to know the demographics of people who visit(ed) your site and left without completing a purchase. To decipher this demographic information, you can conduct an online survey that asks these closed-ended questions:
Question 1: Can you please specify your gender?
Question 2: You are in which age group?
- 18-24 years
- 25-34 years
- 35-44 years
- 45-54 years
- 55-64 years
- 65+ years
Question 3: What is your annual income range?
This knowledge would help you target the right kind of marketing campaign to the exact customers you’d want to attract.
Similarly, the Nielsen Norman Group gave this list that indicates when to use closed-ended questions. Here are the situations where you should use this type of questions:
- In quantitative usability studies, where you are measuring time on task and error rates, and you need to compare results among users
- In surveys where you expect many (1000+), respondents
- When collecting data that must be measured carefully over time, for example with repeated (identical) research efforts
- When the set of possible answers is strictly limited for some reason
- After you have done enough qualitative research that you have excellent multiple-choice questions that cover most of the cases.
Whenever you hear any CRO consultant or agency saying that they will fish out all the ‘barriers that inhibit conversion’ on your site, all they mean is that they will ask relevant questions until they achieve better results.
In this CRO business, assumptions can ruin what might have been a good relation with your customers. Foster the culture of asking questions, after all, the source of valuable knowledge is attained through asking questions.
So, whether it’s open-ended or closed-ended you decide to use, make sure you can answer Yes to the following questions before coming up with any questions:
- Do your participants have relevant prior knowledge needed to respond to your questions?
- Does your question address one of the important aspects that your users may have experienced during the course of the research?
- Is your question clearly outlined, using the appropriate language that can be easily interpreted by your customers/users?
- Are your questions grammatically correct?
Simba Dube is the Growth Marketing Manager at Invesp. He is passionate about marketing strategy, digital marketing, content marketing, and customer experience optimization.View All Posts By Simba Dube
Join 25,000+ Marketing Professionals
If you enjoyed this post, please consider subscribing to the Invesp blog feed to have future articles delivered to your feed reader. or,receive weekly updates by email:
Connect with us
The Art and Science of Converting Prospects to Customers
By Khalid Saleh and Ayat Shukairy
- How to Analyze A/B Test Results
- JTBD and using it on Conversion Projects
- How much does it cost to hire a CRO firm?
- Road Map to Effective Personalization: When, how and what to do!
- 6 Usability Pitfalls that Developers Should Avoid
- Crunching UX Data for Actionable Insights
- Which Comes First? Content or Layout?
- 6 uncomfortable thoughts on AB testing | #CRO
- Influencer Marketing Fraud – Statistics and Trends [Infographic]
- Compliance gaining: What is it and how it can generate higher conversions?