Customer Feedback Best Practices: Usability Testing

Usability testing can shines a spotlight on parts of your product which might confuse or mislead users. Of all the types of feedback you can get from customers, usability feedback can be the most tedious to obtain, yet also the most concrete. It helps clarify whether real people will actually be able to use your real product…and whether they will want to keep on using it.

Sit down with your favorite user experience designer, buy them a cup of coffee (maybe even a muffin), and ask them about usability testing. Then let the debate begin about when, how, and how often to perform usability tests and whether conducting a usability test is a professional skill or something amateurs should be doing.

What is Usability Testing and Why Use It to Gather Customer Feedback?

When should you gather feedback via a usability test? “If you want to learn how quickly and effectively a participant can complete tasks with your product, then you should run Usability Testing. Watch your participants use your product to see where they are successful and where they struggle in order to improve the user experience.”

In order to run a usability test, though, you need to have a working product. Paper prototypes, non-functional html mock-ups, buggy beta versions…they will get you some feedback, but will be skewed compared to the findings from running usability tests with working products.

What kind of feedback can you get from usability testing?

Usability tests will classically show you how real users interact with your product. What areas do they intuitively understand? Where do they stumble? Where do they do things you did not expect them to do, or take actions you had not anticipated?There are two very valuable parts of this feedback.

• The first is that the user can explain their thought process as they explore the product, and you can get a better understanding of WHY. Why was a particular area confusing? Why did the user take certain actions to complete a task? Why were the UI or instructions misleading?  (You might also get recommendations about HOW it could be improved. Users are notoriously bad at solving the HOW, so focus on the WHY and give your designers and developers leeway to figure out how.)
• The second is that you can SEE what people are actually doing, in real time. One of the disconnects you often find with some types of feedback is that a user’s self-reported facts often differ from what they actually did. Or, as Bartosz Mozyrko wrote on Boxes and Arrows, “What people say versus what people do…and what they say they do.” (The full post is a great reference, btw, as is the entire Boxes and Arrows site.)

Usability Testing Facts

Collecting the Feedback: Moderate-Difficult. Usability Tests are often intimate, one-on-one conversations with users, where the user is given some context and then asked to complete a task (or tasks). You need to spend the time to plan out a usability test, and conducting them takes time. Even if you’re conducting remote or automated usability tests, set up takes time.

‍Analyzing the Feedback: Easy. Assuming you have taken the time to plan out the test, tasks, approach, hypothesis and metrics, assessing the success or failure of a usability test is relatively easy.

‍Reach: Deep and Narrow. Usability tests are generally focused on optimizing specific tasks.

‍Scalability: Difficult. These tests are typically one-on-one conversations. Not so scalable, even if they are scripted.

‍Cost: Varies. You can run usability tests on the cheap by simply giving a user access to your software and remotely observing what’s on their screen. Alternatively, you can hire a firm specializing in this sort of thing (bigger bucks) or use an automated tool.

Benefits of Usability Testing

• Seeing is Believing. Since you can see what a user is doing and where they are stumbling, you do not need to take their word for it.
• Usability Testing is a Professional Skill. There are people and organizations which specialize in usability testing, and have finely honed skills and technology suited to the purpose. While it is something which can be performed on the cheap, it can also be outsourced. Consider this option for various stages in the product lifecycle, and when professional intervention may be most valuable.
• Remote User Testing. Sure, it’s beneficial to conduct usability tests in person and see how the product is used, but firing up a web conference to conduct a usability test has many advantages: “1. You are not inconveniencing your participants as much as you would be if you were asking them to come to you, so you are more likely to get higher quality participants. 2. You can conduct tests with people all over the country (or the world) without travel expenses. 3. You can invite multiple team members from multiple offices sit in on the tests and observe.”
• It’s a Test. An Actual Test! As opposed to other forms of customer feedback, usability testing can actually be structured as a (dramatic pause) test. You can define up front what it is you want to test, how you will go about it, the metrics you will use as validation, and systematically take your subjects through the test. Jerry Cao’s article, “Knowing ‘Why’ before ‘How,’” is an excellent reference for anyone planning a usability test. Your product might ace a given usability test, or it might fail miserably. Revel when you pass, learn when you fail.
• Avoid Normalization of Deviance. Usability tests, whether frequent or sporadic, help keep you on track and should prevent you from thinking that clearly bad decisions are actually good. The “normalization of deviance” theory, coined by Diane Vaugn after the space shuttle Challenger explosion, was intended to highlight how people can make terrible decisions even when they have the best intentions. “People grow more accustomed to the deviant behavior the more it occurs. To people outside of the organization, the activities seem deviant; however, people within the organization do not recognize the deviance because it is seen as a normal occurrence.” Usability tests are a great antidote to this behavior.
• Learn More. And More. There are a lot of references available on usability, usability design, and usability tests. If you want to go deep on the subject, you can and should. Every product manager should have more than a passing familiarity with design best practices. If you don’t yet know a little about the discipline of user experience design, stop being an embarrassment to product managers everywhere and learn. (It’s a foundational pillar of being a product manager. Josh Elman said so on Medium, and Martin Eriksson said so on Mind The Product, so it must be true.)

Weaknesses of Usability Testing

• Usability Testing is a Professional Skill. There are people and organizations which specialize in usability testing, and have finely honed skills and technology suited to the purpose. It’s important to acknowledge that creating and running a usability test is a real skill; a professional will extract more (and different) information than an amateur. Product Manager, know thyself. If you’re fortunate enough to work with professional usability people, give them enough direction to get their job done, then step back and observe closely.
• “One user struggled to complete the first task. Let’s change it.” Product managers want to solve problems. Usability sessions are very effective at highlighting shortcomings in our products, and they are usually problems which can be solved. That’s catnip to a product manager. Resist the urge to just take the first feedback and quickly implement it. Be thoughtful and patient. Demetrius Madrigal and Bryan McClain wrote on UXMatters: “The feedback just one or two participants provide might not reflect the reaction your entire target market would have toward the product and its design. Researchers look for trends rather than focusing on individual comments or isolated incidents.”
• You’re Biased. You can try, but your biases will creep into your usability tests. It’s hard to avoid any biases, so as a starting point, be aware of the different types of biases and how they might influence your interactions. Some which are more pertinent to usability tests: The Hawethorne Effect, Task Selection Bias, Confirmation Bias.
• Optimizing the Shit. There are so many tools available which will help you optimize any part of a product, feature or interaction. You can A/B test just about anything. That said, usability tests can also be used to keep you honest, and ensure that the overall experience remains pleasurable to a user. First make sure it is worth doing. Then, in your effort to optimize it, do not optimize out the surprise and delight.

How to use user testing throughout the product development lifecycle

At what point(s) in the PDLC will this type of feedback be most useful?The Product Development Lifecycle:

• Phase 1: Conceive - imagine, specify, plan, innovate
• Phase 2: Design - describe, define, develop, test, analyze, validate
• Phase 3: Realize - manufacture, make, build, procure, produce, sell, deliver
• Phase 4: Service - Use, operate, maintain, support, sustain, phase-out, retire, recycle, dispose

Tapping into Usability is helpful only at specific points in the Product Development Lifecycle.

• For Usability testing to be truly useful, you need to have a product or feature that works and can be tested. Ideas, vague concepts, brainstorms, non-functional prototypes, buggy alpha versions… there is very little value in running a usability test at that point. Stick with focus groups and other less specific feedback mechanisms throughout the brainstorming parts of Phase 1.
• Once you have working versions of your product, as you progress through Phase 2, get it in front of customers and users and run usability tests. Refine it before if gets to market. Refine it once it has already been released. Usability tests are most valuable as you iterate pre-release, and as you create new versions of an existing product.

Usability Testing Best Practices

Hey, You Got Your Usability Testing in My Analytics!

To get better results, don’t just rely on usability testing or analytics, let them inform each other. As Ryan van der Merwe of Jive Software writes, “in order to know where to start to improve the user experience of your product, you’re going to need more than a single data source. No single method can tell you the whole story you need to make good decisions. Usability testing will show you where there are interaction problems, but it won’t show you the magnitude of those problems. Web analytics will show you lots of numbers, but it can’t tell you how to fix issues you encounter.” He goes on to make the case for the two approaches: 1) conducting usability studies first, then using those results to home in on areas worthy of further analytics or 2) start with your existing analytics to develop a hypothesis and use usability testing to validate it.

Seeing, Believing (Part 2)

There is some great advice in the post User Testing isn’t (Always) Rocket Science, and the most important theme is that until you see how people use your products, you cannot always predict what will work or why they might fail. “‘Words and phrases matter to the users. With content, you think you’ve got it nailed down, then you see users get stuck on something really simple. Without hearing [the feedback] on the way, we wouldn’t know why the site was not working later…It’s hard to drill down into why people are doing or not doing something. It’s hard to do with just analytics. When you hear and see people react, you can get into their heads and make decisions based on this.’ says Margret Schmidt, VP of User Experience & Design at TiVo.”

‍Dana Chisnell, co-author of Handbook of Usability Testing, agrees, “I contend that 80% of the value of testing comes from the magic of observing and listening as people use a design. The things you see and the things you hear are often surprising, illuminating, and unpredictable. This unpredictability is tough to capture in any other way.”

Go To Your Users

Anthropologists and ethnographers make a point of observing their subjects in their natural habitats, since they can learn so much more when they have the surrounding context. The same can be said of usability testing: you can conduct a usability test in a sterile usability lab, yet imagine how much richer the results will be when they are in the context of how and where the subject will actually be using the app. Test a shopping comparison app in the aisles of a store (poor lighting, dodgy connectivity). A sports gamecast app in a bar or stadium (extremely loud and incredibly close). A CAD program used on a construction site (large plans viewable by many hard-hatted stakeholders on an itty-bitty mobile form factor). What works well in a contained usability lab may fail in the real world. And unless you’re creating products that will be used in an actual lab, they will not actually be used in a lab setting.

“The point is: let context drive the work,” writes David Peter Simon, Former User Experience Designer at ThoughtWorks.You have permission to be all Jane Goodall with your usability tests.

That said, there are other ways to gather real usability feedback from real users. Services such as UserBrain and Usability Hub  might just be the next best thing, since the feedback comes directly to you.

User-Provided Soundtrack

During usability tests, ask test subjects to narrate what they are thinking and doing as they do it. Rather than trying to divine their intentions by asking a lot of questions, you can allow them to provide the talk-track. Prepare to be surprised and to follow up once the task is completed with additional clarifying questions.

The Value of Usability Data

If Big Data is about gleaning macro insights from many micro data points, Usability Data is about discovering micro insights from a few macro interactions. And in the world of usability testing, you only need a few data points to draw a substantive conclusion.

[Tweet "Usability Data is about micro insights from a few macro interactions"]

“As you add more and more users, you learn less and less because you will keep seeing the same things again and again. There is no real need to keep observing the same thing multiple times, and you will be very motivated to go back to the drawing board and redesign the site to eliminate the usability problems. After the fifth user, you are wasting your time by observing the same findings repeatedly but not learning much new,” says Jakob Nielsen in an article on the Nielsen Norman Group blog.

TL;DR

Usability tests provide feedback on how real customers use particular parts of your product to complete specific tasks. By watching them perform the tasks, you can learn whether your product is well suited to the task, and how it might be improved. Usability tests cannot tell you if your business model or price point are correct, but they can validate if you’ve designed your product to be easily useable. They will help you understand why users are (or are not) continuing to use it and how you might be able to make it better.