Surprise Insights: Coping with Unexpected Product Feedback

One of the most important things that product managers do is listen to our customers.  But what happens when they tell us something that feels like it’s coming entirely from left field?

It could sound something like:

• They made a decision to go with a competitor, only bringing it up during a regular account review
• They told you they NEEDED this change to the product – only to change their minds
• There’s a widespread criticism of the product that you hadn’t anticipated

Really, there’s a lot of different ways that customers, prospects, and the market can blindside us with feedback — but as product managers, it’s our job to understand, analyze, and respond to these situations, often firing from the hip in a pivotal moment.

Being Open Means Being Blindsided

First and foremost, we need to understand that we’re all going to be taken by surprise at some point in our work — and the more research and work we do with our customers, the more likely it is to happen.  The more you test your UX with actual customers, the more likely you are to find something so utterly broken that it’s practically useless — that you designed.  The more you engage with active customers during quarterly reviews, the more likely you are to have some random request (or demand) come out of left field and challenge you.  Being surprised is a natural byproduct of being open to input and feedback — rather than running from it, as product managers we need to look forward to being surprised.

There are so many possible sources of feedback that can sidetrack us:

• The CEO tells you that there’s a new top priority, and what was important yesterday has to be shelved ASAP
• Your #1 customer decides that they’re only willing to renew their contract with the company if there’s a promise of some functionality that nobody has talked about before today.
• You put your thoroughly-designed, carefully articulated, high-fidelity wireframes in front of a customer, and they have no idea how to use it.
• At an industry conference, you suddenly find out that a competitor is going to release a feature you’ve been planning for months a full six months before you’ll be done with it.

All of these things can send us into a spiral of uncertainty and chaos, if we let them.  But as product managers, we need to look at each of these situations as learning opportunities.  We need to take the input as best we can, dig into the circumstances where we have the capacity, and analyze not just the outcome people want, but the underlying driving forces that brought the situation to bear.  Because if we’re not learning, if we’re not occasionally being taken by surprise, then we’re not growing, evolving, or improving.

If we never hear feedback that we don’t expect, we’re not working hard enough.

Separating the Wheat from the Chaff

It’s important to remember that not all feedback is created equally — even if it comes from the CEO or your #1 customer.  Our job as product managers is to always look below the surface, especially when the feedback that we’re getting isn’t what we expected.  We need to remember that not all feedback is created equally.

For example, let’s say that you’re trying out a new user interface, and the first person that you watch trying to use it in a closed-room UX study just doesn’t get it.  Do you give up and go back to the drawing board?  Absolutely not!  You bring in another person, and another, and another.  We want to confirm that the findings of the first person aren’t just an anomaly; that the outcome is repeatable with different people.  We might even want to change up the profile of the people that we’re studying.  The goal is to determine whether or not the finding is something that we want to act on, or something that we file away for later.

It’s also important to remember the classic adage that the plural of anecdote is not data.  This is particularly important when we’re dealing with internal stakeholders, who may have heard something from “one or two customers” and immediately decide that it applies to all customers, without doing the necessary work to confirm that this is actually the case.  We shouldn’t ever apply something that we hear from one person to our entire customer base; it’s just not a safe move to make.

The recurring theme here is that when we find out something surprising, we need to confirm that discovery with others.  Other customers, other prospects, other stakeholders, other product managers — whatever resource is available to us that will permit us to confirm or challenge that initial finding.  Only by making sure that we take the time and exert the effort to confirm surprising feedback can we separate the kernels of good ideas from the husk of random feedback.

Aligning Your Product to New Information

Sometimes the feedback is so clear, confirmed, and compelling that we really have to take a moment to determine the best way to effect change in our product to address it.

When this happens, it should be viewed as opening a new set of opportunities, rather than a failure of the product or product management in general.  The simple fact is, there are always things that we don’t know about our market, our customers, and sometimes even our product.  It’s how we respond to this type of challenging feedback that makes the difference.

First, if the new information challenges some fundamental aspect of your product, we need to critically assess it as a possible pivot opportunity.  Not many people know that YouTube actually started as a video dating service — the site expected that people would post video profiles of themselves, talking about likes and dislikes, what they were looking for in a date, et cetera.  But, when they looked at what customers were actually doing with the site, they were surprised — people were posting videos of their pets, of their kids, of their everyday life, all of which was completely incongruous with a video dating site.  So, seeing this as a consistent pattern of behavior across the platform, they pivoted the product into a video sharing website…a pivot that seems to have worked out quite well for them.

Of course, even if it is a pivot opportunity or an opportunity for change in the product, we need to step back and determine whether pursuing such an opportunity is really worth it.  What’s the cost of implementing such change — is it merely marketing and positioning, or does it run deeper into the technology stack?  Are there competitors out there in this new direction who are already established, and what kinds of switching costs are customers subject to in that space?  While it would be a great luxury to always be able to adjust to new discoveries and new information about our customers, users, and market — the simple fact is that sometimes we just can’t do that, for any number of reasons.  No matter how compelling the new direction or the change in product might be, we still need to remain aware of the bottom line of the company as a whole.  And sometimes this means discarding “good” ideas so that the revenue keeps coming in — or at least putting them on the shelf until later.

Lastly, perhaps the most important response that we have to take when we’re confronted with unexpected or surprising feedback is to determine why it surprised us in the first place!  Are there blind spots in our process that are leading to us missing crucial information early in the process, when the costs of adjusting our plans or product are far less than later on after things have been built, and plans have been committed?  Sometimes it takes a little root-cause analysis and focus on what led you to the point of missing this information to make the necessary shifts in policy, process, or even understanding that made you miss these tidbits of critical feedback early on.

Being Surprised is Part of the Job

Part of being a product manager is being surprised — by our users, by our market, by our competitors, even by our co-workers.  The best among us not only accept this as a matter of course, but seek out disconfirming information wherever and whenever possible.  They make it part of the product discovery, development, and launch process to challenge their own (and others’) assumptions about what the product is, what the product does, and who the product is for.  They view “certainty” with a necessary dose of skepticism, and seek out data and direct feedback rather than relying on their gut or experience alone.  Surprises should be expected in our role, and it’s more important to focus on how we adjust to these shocks than it is to be worried about them arising in the first place.