How to Create a Scalable Product Validation Testing Process

The average salary of a full-stack developer in the US is more than $100,000 per year. To build new features into your product, you need multiple developers. Add to that the investment that goes into infrastructure, research, technology stack, testing, maintenance, and then marketing, and you are looking at hundreds of thousands of dollars’ worth of effort that goes into building a new feature. You want to protect that investment by building features that have a greater chance of being successful.

‍Continual product validation helps you make sure you are building the right features into your product by collecting feedback from your users at all stages of the product lifecycle. You earn stakeholder trust, attract investors, and keep your team focused, all the while reaffirming your belief in your idea. By building a scalable product validation testing process, you can use the same process for new features and other products in the future. Here’s how:

Validate the Severity of the Problem

Your product or new product feature aims to solve a problem. At this stage, you need to gauge the severity of the problem. If it’s not a big enough problem for your target market, it may not be worth pursuing. Similarly, if your user base already has a solution to the problem, they may never adopt your feature, no matter how great it is.

Back in 2013, Facebook launched an app called Facebook Home. It replaced the regular home screen of Android users with a Facebook home screen. The app didn’t work out, and they had to shut it down the same year. The reason: they overestimated the people’s need to see Facebook updates. Facebook users already had the Facebook app, and they could also access Facebook through their browsers. The home screen feed was a solution to a problem that didn’t exist. Facebook would have benefitted from validating the severity of the problem.

To see if the problem you have set out to solve is big enough, plan a survey targeting your user base. Aim for a statistically valid sample size. In a user base of 100,000, you’ll need about 400 participants. Use closed-ended questions designed around the problem. To come up with the questions, first, have a brainstorming session with your team and write down all of your assumptions about the problem. For example, if your idea is to build a new feature that helps your users do their taxes, your assumptions about the severity of the problem can be:

• People spend too much time doing their taxes
• People struggle doing their taxes and have to hire tax preparers/advisors
• People are willing to use a software product that fetches data from their bank and eCommerce accounts

Some of the questions designed around these assumptions can be:

• Do you spend 5–10 hours or more doing taxes?
• Do you hire a tax advisor to file your tax returns?
• Would you like to do it yourself if the process was easier and took less time?
• Would you be interested in an app that would do it for you?

Since these are closed-ended questions, you can validate your idea before investing any resources into it. If 90% of your participants said yes to all questions, you know you are onto something. If, for example, more than 50% said yes to all questions, you are close, but your idea needs to be tweaked and refined. Maybe people don’t spend that much time doing their taxes, or maybe they are perfectly happy with tax advisors unless this new feature does their taxes in just a couple of clicks.

This technique scales with every new feature you want to add to your product. Every time you want to add a feature, go back to the drawing board and validate the severity of the “problem” the new feature aims to solve.

While we recommend using closed-ended questions for this stage, you can also go for open-ended questions, Likert scale questions, and so on. But collecting the data is only part of the validation process. Make sure you know how to interpret data from different types of questions, so you don’t find yourself in a situation like Facebook Home.    

Validate the Size of the Market

Validating market size will help you determine whether it’s worth it for you to invest time in development. It is very easy to fall in love with your idea, validate it on a surface level, and start building, only to find out there isn’t a big enough market for your product to be successful. Developing a new feature without validating is like adding a meat department to your grocery store in a predominantly vegan neighborhood. Yes, there are a few people there that’d regularly buy from you, but not enough for that initiative to be profitable.

To validate market size for your new feature, first determine a target market. This can be the exact same target market that uses your product, or you can be branching out to attract more people with the help of a new feature. You can do this based on job titles, interests, age, location, and other criteria related to the problem. Let’s say you have an app that helps gym managers/owners manage their gyms, and you want to introduce a new feature that helps them post new media to social networking sites. Your initial target market would be gym managers and owners that currently use the app, as well as any gym managers located in your state. So, you will depend on user status, job title, and location as the three variables to define your target market. The other variables, such as age and gender, may not be as important.

Once you have defined your target market, look for data from reliable sources in terms of the number of potential customers. You can find the total number of gyms in your local area from Google maps. You can also use Yelp and BBB.org to find gyms in your state. You can look up gym managers and owners from online fitness forums (such as r/Fitness on Reddit). For current users, you will rely on your own database.

The next step is to determine price. Sometimes, you want to add a new feature for free just to keep up with the competition. On other occasions, you want to charge the users to gain access to the new feature. In either case, you have to consider product price to calculate market size.

If you plan to give the new feature to all users for free, use your current product price in the following equation:

Market size = (Total number of potential buyers) × (product price)

If you plan to charge the users for the new feature, calculate the new product price by taking cost of development and upkeep into account. You also have to consider profit margins and market demand. If there are similar products out there, you can look at their pricing for comparison and see if you want to start with a lower price to increase market penetration.

Once you have determined a price, you can calculate the market size using the following equation:

Market size = (Total number of potential buyers) × (product price)

Let’s say there are 2,500 gyms in your state. If each were to buy your product at a $1,200 one-time fee, your market size would be $3,000,000 (2,500 × 1,200). If your new feature costs $25 extra, your market size would increase by $62,500 (2,500 x $25).

To scale this aspect of the validation process, use it every time you want to introduce a new feature or advertise the product to a new market. Going back to the example of the gym management app, every time you want to introduce a new feature, calculate the market size again and see if the number of potential buyers and the new product price make it worth developing the new feature.

Validate the Market’s Willingness to Pay

You also need to know how much your target customer is willing to pay for your feature because sometimes users will only use a feature for as long as it is free. In the previous section, we calculated the market size for your new feature for the gym application as $3,062,500, presuming that each gym would pay $1,225 for the product (the additional $25 due to the new feature). Now we need to find out if gym owners are willing to pay $25 (or anything at all) for the new feature.

One way to validate the market’s willingness to pay is to conduct another survey. You can go with the same sample size as before, and this time, structure your questions around pricing, budget, and affordability.

Another way to go about it is through crowdfunding and pre-orders. Money coming in through both tactics serves as a source of validation because you are asking for people to spend money on a feature that hasn’t been developed yet, as well as fund development. You can put your idea on Kickstarter or a similar website and set up campaigns for crowdfunding.

Back in 2012, Oculus set up a Kickstarter page for Oculus Rift, a VR headset. The company raised $250,000 in a matter of hours. That’s how the company knew the market was willing to pay for its product. While this is an example of raising money for a new product, you can do the same for a new feature you are building for an existing product.

You can also use pre-order sales strategies to the same effect. Set up a pre-order campaign on your website, advertise it to your target market, and see how many people are willing to pay for your product before it’s available. In either case, you are asking the market to back your idea with actual money. Sometimes, data collected through surveys can give you a false sense of the market’s willingness to pay for your product, but the money you raise through crowdfunding or pre-orders will prove to you that the market is actually willing to pay.

Validate at the Pre-Alpha, Alpha, and Beta Stages

Validating the product or feature during development helps you improve it on the go and keep it in line with user expectations.

Validating a feature in its pre-alpha stage means sharing sketches, designs, and a non-working version of the product/feature with the users. You can use A/B testing at this stage to compare two versions of the same design and see which one your users like better. Highly successful companies like Netflix rely heavily on A/B testing before making any changes to their UX or developing new features in general. You already do it internally, i.e., discuss the non-working version of the feature with your team, so take it to the users and include them in the conversation too.

Validating a feature in its alpha stage means sharing a non-tested but working version of the feature with a select group of users. Usually, developers do alpha testing, but getting some end-user perspective at this stage can help you make sure your new feature is headed in the right direction. The developers can find bugs in the code, but end-users can find issues with how different features work vs. how they expected them to work.

Beta testing is standard pre-launch practice, but it’s good to make it part of your scalable product validation and testing process to work out any usability issues. Beta testers let you know if the finished product is as per their expectations or not, making it possible for you to make last-minute changes if needed.

You can find users willing to test your pre-alpha, alpha, and beta products/features on relevant forums. For example, members of the SEO subreddit would love to take a look at new features for your SEO product. You can find users more than willing to test your new video game features on Discord, Twitch, and so on.

Create a Scalable Validation Testing Process With UserVoice

Scaling your validation process can be overwhelming at first. UserVoice gives you the tools you need to collect user feedback at every stage of a product’s lifecycle, from the very early stages of the new feature to launch and post-launch. Get unlimited micro surveys, unlimited end-user responses, and unlimited seats along with a 21-day free trial. Sign up for a free demo today.