8 Scalable User Research Methods for Product Development

So you launched a new user research project.You prepared tasks for users to complete, invited participants then ran the test, and a week later you’re still trying to come up with some meaningful insights from the results.Don’t you just hate it when that happens?

You thought that inviting additional users would deliver deeper insight and better data. And, in turn, help you draw better conclusions. Yet it only broke your research method.What if you needed to test a larger sample size, perhaps to make a point across or test a major hypothesis?Then you’d need to use a research method that can scale. And that’s exactly what I’m going to address in this post, 8 scalable research methods you can use to test samples of any size.But first…

What exactly is a scalable method?

Investopedia defines scalability as:“[…] a system, model or function that describes its capability to cope and perform under an increased or expanding workload. A system that scales well will be able to maintain or even increase its level of performance or efficiency when tested by larger operational demands.”When speaking of Scalability in Information Technology however, we often refer to two distinct meanings:One relates to the definition I cited above. “It is the ability of a computer application or product (hardware or software) to continue to function well when it (or its context) is changed in size or volume in order to meet a user need.” (source)But it can also relate to:“The ability not only to function well in the rescaled situation, but to actually take full advantage of it. For example, an application program would be scalable if it could be moved from a smaller to a larger operating system and take full advantage of the larger operating system in terms of performance (user response time and so forth) and the larger number of users that could be handled.” (source)And in the context of user research, I’d follow the IT’s definition and say that:A scalable research method is capable of delivering results even under the pressure of a higher sample size.In other words, it can provide meaningful insight even if you significantly increase the number of participants.

8 Scalable Research Methods for Product Development

A/B Testing

The idea behind an A/B test is simple:You identify a problem element (for example, a page has a particularly high form abandonment rate),then based on research and analysis come up with a hypothesis as to what might be causing it, and launch a variant of the page including a reworked element you want to test.And:A/B testing delivers the best results if you scale the number of visitors seeing it.Pinterest continuously runs scalable A/B tests. As one of their software engineers, Chunyan Wang explains:“We have hundreds of A/B experiments running at anytime, as we launch new products and improve the quality of the user experience. To power these experiments and make timely and data-driven decisions, we built a system to scale the process.”Furthermore, when defining goals for their testing framework, the company specified that it should be “Scalable to process and store a large amount of data generated by ever increasing number of experiments, users and metrics to track.” (source)

Clickstream Analysis

[caption id="attachment_2003992" align="aligncenter" width="600"]

A sample clickstream report – image courtesy of Opentracker[/caption]Clickstreams, also known as clickpaths, show the route a visitor chooses when navigating through a site.A clickstream report shows when and where a person came to the site, what pages he or she visited, the time they spent on each page, and where they left.Clickstreams help remove the guesswork about what people are doing within your app or on your site.Gathered together this data can help you identify the most frequently used app functionalities and pinpoint potential problems such as users spending too much time on a particular task.And since the whole process of collecting this data is automated, it can perform well regardless of the sample size.

Polls and Surveys

Asking users what they want is a recipe for research failure.(I recently explained why in this post.) But...asking users to provide information about feature preferences for instance could help you plan your product roadmap and deliver on users’ expectations.

Polls and Surveys let you tap into a user’s experience and preferences and provide you with meaningful feedback.You can use polls and surveys to validate ideas, collect feedback about a new feature, run a new idea by users, and much more.And often, they deliver the best insight when run on a large user sample.

Heatmaps, Clickmaps, and Screen Recordings

By visually representing users’ clicks, taps, and scrolling behavior all three methods help understand:

• What users want from your app,
• What sections or functionality they care about,
• What grabs their attention, and
• What options they may find confusing or out of place.

Screen recordings, on the other hand, let you observe customers using your app. You can see what they click (or attempt to click), how they navigate through the user interface, and how they interact with various features.

Surveys (Online and Email)

Surveys have become one of the most popular quantitative research methods for a reason. They allow us to research problems on virtually unlimited test samples. In fact, the larger the test sample, the better results you can expect.Naturally, the quality of survey results correlates to your sample as well as variables used to construct research questions. But provided you target the right users with well-constructed questions, surveys can be the most scalable way to test any hypothesis.In product development, the two most common survey types include online (like website or in-app surveys) or questionnaires sent via email to a selected sample.

And there you have it…

8 research methods you could use on a larger sample size and still get meaningful and usable data.