How to Measure Product Launch Success

Pulling off a successful product launch is no easy feat. During the planning and lead up there are lots of things to pay attention to and keep track of, whether it’s scheduling, meeting sprint goals or ticking off items on your sales and marketing launch checklists. Once your product is out the door and released into the world, it's time to measure your product launch's effectiveness; you and your management team will start asking much different questions and priorities will shift from “how many days will we slip?” to “what’s our adoption looking like?” That's where product launch KPIs come into play.

Today we'll look at how to determine the best KPIs for your product launch. "While running toward a final goal is great, knowing if you are on the right path along the way is just as important," Chris Ayala, CEO of smartphone breathalyzer tool Alcohoot, tells Business News Daily, "It also means that everyone understands there is accountability — either for success or failure. Without measurements, then after-the-fact analysis is almost impossible." Before you start measuring arbitrary items and throwing numbers around, it’s critical that everyone involved cares about the same key metrics.

This is why your key metric definition process should take place before you even launch, this means not only setting and agreeing to a specific set of goals, but also determining how you will measure your progress toward them.

“Reporting on the results of your product launch should be a natural outcome of establishing launch goals,” explains Dave Daniels of Launch Clinic on the Pragmatic Marketing Blog, “Define metrics that support the goals and then a meaningful Launch Status Report can be developed.” In theory, these goals should be relatively obvious, as they would align with the same reasons you created a new product or released a new version of an existing one.

You saw a market opportunity with a set of potential customers that are looking to solve a problem or satisfy a need. Your next step is translating those goals into actual usage behavior:“No matter what your site is, there are actions you're hoping visitors will take - from tweeting a link to your post to leaving a comment to buying a product or subscribing to an email list. Whatever those actions might be, you need to record the visits that make them through your analytics tool,” says Rand Fishkin, cofounder of Moz in an article on website launches, “Once action tracking is in place, you can segment traffic sources and visit paths by the actions that were taken and learn more about what predicts a visitor is going to be valuable.”

Start by Measuring Marketing’s Launch Efforts

Your product’s launch success is just as much about attracting users to try the product as it is about the product itself, which means measuring performance of your marketing efforts is a good starting point.

‍After you launch, “be vigilant about keeping track of what efforts are starting to lose effectiveness so that you can make the appropriate changes,” says Martin Lünendon, founder of Entrepreneurial Insights “In the same manner, keeping track of which media tools were the most successful in converting potential customers into sales will give an indicator of where your efforts should be spent. Dropping the ineffectual methods and replacing them with renewed efforts in the productive methods will help not only generate income by identifying new customers; it will maximize the customer awareness.” One of the easiest ways to track marketing channel and tactic effectiveness is through liberal usage of landing pages.

“When driving traffic to our website, instead of pointing a link to our main site, we use one of our landing pages based on what we know about that visitor,” Sami Linnanvuo, CEO of Screenful explains a tactic that works to measure results both during and after a launch, “We can compare the effectiveness of different channels on different visitor segments. We’re curious to learn how these alternate landing pages convert and which are the best channels for each page.” Setting up a landing page for each channel or campaign makes it incredibly simple to see where traffic is coming from, and if you are able to flag those users for future cohort analyses, you’ll be able to see the long-term quality of those visitors as well and track their progress through the sales funnel.

Measure User Engagement

Marketing and PR are just the beginning, so don’t rely solely on marketing metrics to measure your launch’s success. “A solid, well-executed PR and social media campaign can drive traffic to your website, but they can’t control what happens next. After the media blitz and buzz, the real challenge rests in keeping users around and compelling them to come back. That’s why in the early days, it’s critical to focus on the user’s experience and engagement,” says Boris Wertz of Version One in an article on launch metrics, “It’s far easier to bring in more traffic than it is to create a happy customer.” An essential tip-off for user engagement and whether your product is meeting its goals is examining specific feature usage instead of just “traffic” and other, less revelatory, statistics.

“Measure the coverage of the products, i.e. which features are being used by users and how much. More often than not, this will throw up many surprises, especially features you thought users will use the most/least,” says Rajat Harlalka, cofounder Bellurbis Technologies. “It will also help prioritize your product backlog.”

Create a User Engagement Pyramid

More than likely there are many different things a user can do with your product - just because they visit your site or open up your app does not mean everyone will do the same thing once they start using it. So, think about what is the most important activity for your product. If your goal is simply “increase engagement”, what activity shows that they are truly engaged? Is it posting content or a picture? Is it adding items to a shopping cart? This is the activity that goes at the top of your user engagement pyramid.

One suggestion for the top of the pyramid is completion of a key workflow, item according to Aaron Severs of Pivotal labs. In an article on startup metrics he explains how to identify a key workflow: “Without doing this, your users will fail.” he writes, “For example, for an email marketing tool, this would be sending your first email campaign.”  Then, think about all of the other things they could be doing with your product. Place these in descending order of importance until you get to the base of your pyramid.

Now, look at your total activity and divide it into the various layers of your pyramid. The most important usage info is from the users completing the activities in the top layers of your pyramid, while users that don’t make it as high represent less engaged users.

“An engagement pyramid grounds your business metrics in something real,” says Angela Tran Kingyens of Version One, “It drives home your ultimate user engagement and helps you develop the right processes to get there.” An added benefit is being able to visualize the correlation between the various potential actions a user can take. For example, you may have assumed that users will be frequent commenters before posting content, but the data will show whether this is really the case (and therefore whether you should be worried so much about how often people comment if the ultimate goal is getting them to post content).

The Importance of Cohort Analysis Can Not Be Overstated

Your product debuts, your marketing campaigns kick in and WHAM!, you’ve got new users and registrations and all kinds of activity. Then the press gets bored, your AdWords budget runs out, the early adopters are all done adopting and suddenly those metrics that were so fun to look at for the first few weeks start to dwindle and fade.

Don’t panic! This is completely normal. It’s now time to move into the slightly more sophisticated and more useful phase of cohort analysis.

“Your task as a product manager is to figure out what stops less engaged users from committing to your work, writes Evgeny Lazarenko, a data analyst at Paktor. “Looking at time series user actions data for different categories of users can help you with that.” Cohort analysis isn’t particularly complicated, you simply divide up users based on a commonality (most often when they started using your product). So, for example, you put the users who started using the product in Week 1 into Cohort A, Week 2 into Cohort B and Week 3 into Cohort C.

Instead of looking at all of these users as one giant blob of usage data, you can more easily identify trends. So, by Week 6, only 10% of Cohort A users may be using your product three times per week, while 30% of Cohort C are using it three times per week. This kind of segmentation shows you that something is happening around a month in to drastically cut usage (perhaps your free trial ran out). Or, you may find that by Week 6 the length of time the remaining Cohort A users spend with your product is higher than Cohort C, which indicates your product gets more addictive over time.

As you make incremental changes to your product, whether they’re UX tweaks, bug fixes, or implementation of new features, cohort analysis becomes even more important since it is one indicator that these changes are having a positive (or negative affect) on your key measurables.

Of course, cohorts aren’t necessarily limited to start dates; you could also include acquisition channel, industry vertical, demographic data…whatever will help you find a meaningful subset of users to examine.

Lazarenko recommends using the following measures, but these concepts can be adapted for your particular product:

• Number of sessions per user (cohort of users), over time
• Session duration for a cohort, over time.
• Number of user actions per session, for cohort, over time.
• Action completion rate & differences between different cohorts

Know Which Metrics to Track & Which to Report

Track everything, but report only what is important. If you report every little thing, it all becomes noise, which is why you must limit reporting to metrics that matter. This is not a particularly novel concept, but an important one to keep in mind at product launch.

“While hundreds of performance indicators and thousands of metrics are possible, the phrase “key performance indicators” shows the way,” says Dr. Simone Setiter of Launch Excellence. “By their very description, KPIs focus on only the most important and pertinent metrics. They must be crucial and relevant to the organizational or brand strategy, be actionable for managing brand performance and initiating positive change, and—when measured against specific criteria or thresholds—yield insights towards the desired objective.”

You’re a product manager, not a psychic, and no one expects you to nail 100% of your KPIs before your product hits the shelf. So while you may only be actively looking at and reporting on a subset of metrics that seemed important on Day 1, you want to store every possible data point because you never know what is going to ultimately move the needle. This way you can always go back into the logs instead of bemoaning the elusive data point you wish was available.

It’s also a great reason NOT to invest in coding complicated dashboards until you know what’s really worth tracking:“Although you should track/store as much usage data as you can, don't waste time creating fancy (or non-fancy) reports just yet,” Dharmesh Shah, Founder of HubSpot, advises in an On Startups post, “Just capture it. Some simple mechanism to get a sense of usage is fine, but don't try to build ways to look at all the data you're tracking. It's a distraction. Focus on what will make the users happy. You can work on reports later.”

Of course, it’s also important that you establish what’s actually trackable before establishing it as one of your core metrics.

“A trackable metric is something like sales or revenue or subscriptions or newsletter sign ups. An un-trackable metric is something like “usability”. Usability will be very hard to quantify and it makes it hard to quantify,” says Michael Rutledge, Product Manager at Audible.com in a post on The Product Manager Club. “Another example of a metric that isn’t trackable is any metric that your company literally cannot track. Maybe your site isn’t tagged or there’s some other issue, but if it’s not trackable it’s pretty obviously a poor choice.” Oftentimes overlooked by engineering teams driving hard toward deadlines, error handling and exception logging are often a treasure trove for identifying problems worth addressing, but if you’re not capturing that data, there’s no way to know if a certain event is happening or how often users are encountering it and abandoning your product out of frustration.

There are Multiple Sources of Data & Feedback

“Great metrics tools allow us to audit their accuracy by tracing reports back to the individual people who generated their data. This improves accuracy, but its more important effect is that it lets us use the same customers for in-depth qualitative research,” says Eric Ries, author of The Lean Startup. “Not sure what the numbers mean? Get the customers on the phone and ask them.” Yes, talking to customers--the same thing you did when figuring out what to build in the first place--is still an important part of your post-launch analysis.

"Conducting surveys and focus groups is always helpful and something I strongly encourage to gain insight into consumer behavior," says Devaraj Southworth, CEO and founder of on-demand wine and spirits delivery service Thirstie. "However, a word of caution: The feedback should be used as guidance [instead of] validation."

Customer service volume is also a great source of early stage feedback. If a user took the time to proactively provide feedback it means a) they were really annoyed and b) there are probably far more users experiencing the same thing who didn’t bother to tell you or just quit using your product altogether.

“Nothing irritates customers more than poor quality products, particularly ones they’ve paid for. And they won’t be shy about telling you about those problems,” says Saeed Khan of On Product Management. “This is one category where a deep analytic assessment is possible, and that should factor into how quality is measured. i.e. consider measures such as new customer reported bugs,  bugs addressed, severity of bugs reported, # of customers reporting bugs etc. All of these can be looked at over time to see if the problems are increasing or decreasing. Based on this assessment, additional investigation into specific problem areas can be conducted.”

Avoid Data Overload

Too much data can be worse than no data at all, and if you have metrics that simply aren’t telling you anything significant, it’s time to stop worrying about them. Focus on the metrics that matter. “By forcing yourself to have a target for killing metrics you are ensuring that you'll focus on an important activity once a quarter,” explains Avinash Kaushik, author of Web Analytics 2.0 You'll re-visit your assumptions and what's important to the business.”

You should also limit your reporting to the metrics you might actually do something with and could inform future actions, vanity metrics are nice to have but are usually not very useful or actionable.

“Avoid the temptation to add "interesting stats" to your list,” says Kerry Rodden of Google in an article on UX metrics, “Will you actually use these numbers to help you make a decision? Do you really need to track them over time, or is a current snapshot sufficient? Stay focused on the metrics that are closely related to your goals to avoid unnecessary implementation effort and dashboard clutter.”

The TL;DR: Define Key Metrics Before Launching

If you want to properly track the success of a product launch (or a feature launch) you’ll want to set goals and identify key metrics and ensure they are being tracked before you launch said product or feature.

‍