5 Costly User Research Mistakes (and How to Avoid them)

From an outside perspective, conducting user research can seem like child’s play. You ask users a few questions, or observe how they use your product…and BAM! As if by magic, you know exactly how to knock their socks off next. But the reality, as you know, is user research isn’t as easy as it looks.

Getting useful insight from your UX research is a delicate balancing act of art and science that can take years to master. And if that’s not challenging enough, user research mistakes can be extremely costly. Seemingly innocuous things can lead to bias, inconclusive results, or the development of products destined to fail. Fortunately, the vast majority of these mistakes are easy to avoid if you know what they are.

In this article we’ll share the 5 most common user research mistakes and how to avoid them.

5 User research mistakes to avoid

1. Asking users what they want

Asking users what features they prefer or need seems like the best way to gain customer insight, doesn’t it? As the old adage goes “the customer is always right.” But when it comes to user research that’s rarely the case.

Why? Most customers have no clue what they want.

In fact, this lack of insight is one of the key traits of our human behavior.

Consider Emily Pronin’s Introspection Illusion theory, for instance:

“People wrongly think they have direct insight into the origins of their mental states, while treating others’ introspections as unreliable. In certain situations, this illusion leads people to make confident but false explanations of their own behavior or inaccurate predictions of their future mental states.”

An additional theory of “casual theories” confirms that “there are very few genetically driven causes for behavior for humans in general and none for individual traits”. It states that a person typically doesn’t notice the real reasons for their behavior. We are however good at trying to provide explanations.

And lastly, in their 1977 paper, “Telling More that We Can Know” (PDF version) Richard E. Nisbett and Timothy DeCamp Wilson described a series of studies on cognitive processes and confirmed that:

“When people attempt to report on their cognitive processes, that is on the processes mediating the effect of a stimulus on a response, they do not do so on the basis of any true introspection. Instead, their reports are based on priori, implicit casual theories or judgments about the extent to which a particular stimulus is a plausible cause for a given response.”

In other words:

We humans have absolutely no understanding of our mental processes. And thus, there’s no point in asking us about what we want.

What to do instead of asking customers what they want

Avoiding this user research mistake is simple. Don’t ask what users want. Observe.

Learn more about how your users interact with your product. Dig deeper into their motives and ask them open-ended questions to slowly uncover the customer truths hidden in between the lines.

2. Your user research is biased

Most people don’t like being wrong. After all, who wants their research to confirm that users don’t need a particular feature the team spent months and months developing? Or that the layout you feel so proud of is actually unusable?

But effective user researchers need to accept that their hypotheses will not always be correct.

Getting new insight from user research requires you as the researcher to not only keep your own cognitive biases out of things, but also prevent bias from infiltrating the research methods themselves.

That said, the unfortunate truth is, bias is everywhere and can be difficult to avoid. Here’s a few common ways bias can find its way into your user research.

You ask users leading questions

Leading questions are those which suggest a particular answer, or imply what kind of information you expect to confirm.

For example:

  • “What do you dislike about feature X?”­ –prompts the subject to think negatively about a particular feature and thus include only negatives in their answer.
  • “How good is feature X” – prods subject to respond only in relation to the feature’s usefulness.

A non leading question on the same topic would be asked this way:

  • “What do you think about feature X?” – doesn’t suggest anything about the feature or prompt the subject to think about it in any particular way.

You suggest the answer

If you incorporate your own point of view into your research questions, you leave the user with two options:

They can either agree with you OR face having to defend their ideas.

For instance, I once saw a website quality survey featuring this question:

“We love our site. Do you?”

The survey allowed me to reply either yes or no (and only the latter included a text field where I could explain my choice).

Now think about it:

My involvement with the site was minimal. I landed on it to find information or satisfy a particular need, but my attitude towards it was neutral at best.

Forced to respond to a survey, I preferred to agree rather than face having to explain why I disagreed with the company. And I’m sure I wasn’t the only one leaving the company with completely useless feedback.

You preselect the answer

In addition to not truly knowing what we want, we humans are also awful at decision making.

Decisions take effort and labor. And we’ll go to great lengths to avoid making decisions. Or, often, we’ll simply skip the decision thought process completely by going with the default option we’re presented with.

(Dan Ariely has a great TEDtalk on the topic if you’re curious.)

In user research, pre-selecting or suggesting an answer in any way skims the research. Simply because many users will naturally go for the default option… if only to skip that torturous task of making a decision.

3. Selecting the wrong user research method

When you set off to conduct user research, usually the first thing on your mind is not how but what and why. You want to know what the users of your app need to be successful, and you want to know why they need those things. And (as often is the case), a customer survey comes up as a viable user research method (the how) for getting those answers efficiently.

Now let’s look at this hypothetical situation from the perspective of a user.

You launch your favorite app, one you just can’t live without…and you are greeted with a survey request. You shrug at it but since you use the app so much, you decide to help. But what seemed like a simple survey turns out to be pages upon pages of questions…leaving you stuck and frustrated instead of working on your stuff.

And tell me, at what point do you start picking responses at random to get to the end quicker?

You see, not every research medium is suitable for every situation.

For instance, users are more prone to filling in a quick survey while using an app, but they’d be more likely to provide long and in-depth answers outside of it.

Be considerate of the method you use to obtain your research. As a helpful tool, consider reviewing Christian Rohrer’s 3-dimensional framework for identifying when to use different user research methods.

The 3 dimensions include:

  • Attitudinal vs. Behavioral
  • Qualitative vs. Quantitative
  • Context of Use

And include the following contextual methods to use:

  • Natural use of product,
  • Scripted (lab-based) use of product,
  • De-contextualization (not using the product), and
  • A hybrid of them all

Check out his thorough explanation for the framework, it’ll help you decide when to use different research methods.

4. Reusing user research studies

This one should be pretty self explanatory. Just like we want to avoid building features for the sake of building, we want to avoid research for the sake of research. The purpose of user research is to learn; to find out what works, what to iterate on, and then, move to a new problem.

Reusing old ideas isn’t conducive to learning, at least not anything new.

Don’t start a new research study unless you have clearly defined what it is you want to learn from it and why.

5. Not testing your user research method

It may seem a bit odd to suggest testing your testing method, but trust me, you cannot neglect this step.

Imagine spending hours designing a user research study, writing the perfect user research questions, and building your survey. You launch the new research project, put up a survey in the app for all users to see. A day later, you discover that the script is broken…or that your questions confused most users…

As a result, you receive no feedback, or feedback that is simply not useful.

All that trouble could have been prevented. And that’s why you must pilot test your user research methods. A pilot test can help establish whether your research will generate viable feedback, and can also be used to check that all technical parts are working too.

Whenever you launch a new research project, send it to a single participant first. If they get confused or can’t complete it for any other reason, revise it. Make sure that users can understand your tasks or questions and that you receive their feedback. Then, rinse and repeat the test until you’re 100% sure your research is going to produce meaningful results.