If product development were a long road trip, then beta testing would be like ensuring that you arrive at your destination looking and smelling fresh. Ignoring or short-shrifting beta testing can lead to the same reaction you might receive when arriving unshowered and smelling of beef jerky and french fries. Sure, you got there, but after one awkward hug, no one wants to get close to you. Effective beta testing ensures your impending product release hits the mark and is ready for prime time.
To torture this analogy a bit more, at what point in the adventure should you be opening the door to some beta tester hitchhikers, and what qualities do you look for before inviting them along? Where do you find beta testers? In a dark alley off a dusty desert road holding a sign that reads “The end of development is near”? Probably not.
Beta testing is most valuable when done early enough in your product release cycle that you can incorporate feedback from beta into your product, yet late enough in the development cycle that a legitimate version of the product is being tested.Given the timing and value of beta tests, you’ll likely need to be able to find great beta testers on short notice for short money.
As with all good road trips, detours happen. So before we get to the intersection of Where and How, let’s spend a little time on Why, When, and What Roads.
You’ve run the pre-release product through its paces internally, design has run focus groups and usability testing on portions of the product, QA has been hammering on it non-stop, and engineering is so ready to ship it. A beta test validates that users see the product in the same light. It’s an opportunity to get real people using the almost-real product. You will then have the opportunity to respond to that beta feedback in a more subdued way than you might if the product were already released.
And those “real people” I referred to above? Those are your beta testers. The quality of the beta feedback will rest in large part on your ability to recruit and motivate the right group of testers.The type of beta testers you need may depend on your goal for the beta. For example, as Joel Spolsky wrote, “Don't confuse a technical beta with a marketing beta.” He continues, “[The goal of a technical beta] is to find bugs and get last-minute feedback. Marketing betas are prerelease versions of the software given to the press, to big customers, and to the guy who is going to write the Dummies book...With marketing betas you don't expect to get feedback”
Beta testing is typically performed on a version of software which is substantially feature complete, with few bugs or known issues. Plan on having multiple rounds of beta testing. The feedback from each round needs to be vetted, prioritized, and potentially incorporated into subsequent beta versions of the product.If you do not intend to allow time to incorporate feedback, it’s not really a beta program, and Evan Hamilton just might scold you: “If you’re going to ask customers for feedback, you need to seriously consider that feedback. To seriously consider that feedback, you need to plan in time to potentially act on it. If you’re doing a beta right before your code complete date, you’re doing it wrong.” (What’s the outcome of a poorly run beta? I’ll share a limerick in an upcoming post which pretty much sums it up.)
In 2004, Joel Spolsky explained that a beta test should run for no fewer than 8 weeks, and that you should plan on releasing a new version every 1-2 weeks during that period. Today, with more rapid development cycles and more sophisticated devops tools, it’s possible to release more frequently and to even incorporate feedback quicker. The key takeaway is that a beta is a multi-week process of incorporating feedback from people who are not a part of the team that created the product in the first place. Those people, beta testers, are the ones who make this process run. They’re the ones looking at the product with a beginner’s mind, and have the ability to see what you no longer can.We’ve established why you should run a beta test, when to run one, and what a beta test and beta tester are. Now on to the pragmatically fun stuff: where you find appropriately qualified beta testers, and how you inspire them to test (in a beta sense) your product.
So how do you find, qualify, and inspire a group of beta testers? There are plenty of ways to identify and recruit beta testers. Start by articulating your target audience and goals:
Once you’ve nailed those questions, expand your criteria to encompass product maturity and budget:
Mature: If you have a product, customers, and user base, you can immediately tap into your existing community. Depending on the size of your company and installed based, you may simply be able to reach out to the community through already established channels (social media, email lists, blog) and customer-facing functions within the company (sales, marketing, customer support, professional services). While your beta tester recruiting costs may be lower, the cost of managing your beta program may be higher since you’ll likely want a large beta panel.
Adolescent: If your product is less mature (but you still have a product which is on the market), or you do not have a broad customer or userbase to leverage, then don’t you worry, there are services which you can use to help with the recruitment. They range in price from free (yay!) to gold-plated (yay!), so choose the option that fits you budget. You get what you pay for (yay!).
Infant: If you’re just getting started and have not yet released a product to market, or if your pool of friends and family is too small or ill-suited to the task to be effective, it’s time to go another route. Let’s say that you’re really bootstrapped and want to spend little or no money, go the guerilla route. If you’ve just raised lots of money to bring your product to market, why skimp? That’s why you raised all that cash, no?
Liam Gooding at Trackio has compiled a great list of places where you can recruit beta testers for free, and also included some pros and cons of each option. Sites like Betali.st, Erli Bird and StartupLi.st require little investment to get started and allow you to post your needs, then further qualify the respondents. Note that some of these sites have “premium” options where your posting will received preferred placement. Some sites, such as Quora and Hacker News, can be a goldmine, but you need to invest the time to build a reputation within those communities.David Arnoux of twodoo points out that, while Gooding’s list is a good starting point, free is not really free. For example, he writes:
He goes on to expand on how to exploit some truly free alternatives, such as posting to appropriate online groups (Facebook groups, LinkedIn groups, Google+ communities (RIP)), publishing infographics, seeking out people who are unhappy with your competitors, and soliciting commenters on appropriately targeted blogs.Michael Siegler, Founder at ErliBird, lists other resources, depending on the goals of your beta tests. Typical goals are UX/UI feedback, user acquisition, and bug testing, and the services he highlights range from UserTesting and Lookback, to Google AdWords, Facebook Ads and TestElf.
Emily Hossellman, a marketer at Centercode and self-proclaimed nerd, posted to Quora that in order to get beta testers that give in-depth feedback you should “Qualify your testers before you let them into your beta program. The easiest/best way to do this is with a qualification survey. The survey will give you valuable information about their demographics as well as how they'll behave at a beta tester.”
She goes on to explain, “Some people are afraid of making their testers fill out a survey to join their beta. They're concerned that they'll scare people away. But you want to scare away the testers that aren't willing to put time and effort into beta testing. Adding some friction to the beginning of your beta test forces your potential testers to work a little bit to get into your test, which makes them more invested and makes it more apparent who's willing to give some time to your product.”Joel Spolsky agrees: “The best way to get a beta tester to send you feedback is to appeal to their psychological need to be consistent. You need to get them to say that they will send you feedback, or, even better, apply to be in the beta testing program. Once they have taken some positive action such as filling out an application and checking the box that says "I agree to send feedback and bug reports promptly," many more people will do so in order to be consistent.”
Spolsky has other great advice on finding beta testers in his post Top Twelve Tips for Running a Beta Test. Points 7, 9, 10, and 11 are spot on, the sort of advice that can only come from a person who has suffered through too many beta test cycles.Is a beta tester just someone who tries an early version of your product and provides feedback? It’s a missed opportunity if that’s all you want from your testers, says Mikal Belicove in Entrepreneur Magazine. “Many view these testers as the early adopters of your offering. So not only can they help suggest improvements and features, they're often the first ones to generate excitement about your launch.”It’s the early adopters, the ones who have a stake in the outcome and who are your target users, who are the most likely to provide actionable feedback, and who will most likely stay engaged throughout the beta period. The hope is they will also turn out to be early evangelists. Ultimately, a core group of product evangelists could be the difference between WTF and FTW.
Some people claim that the hardest part of beta testing is recruiting a reliable, responsible, representative slate of beta testers, and that once you’ve qualified them and gotten them signed on, it’s all open road from there.
That’s the perfect set-up for my next post, which will be on how to motivate beta testers into giving regular, actionable feedback. Get ready.