The Human Element of Data Driven Decision Making

When we’re feeling confident it’s easy to assume we’re right. Unfortunately, during product strategy meetings, this assumption (and, quite often the reasoning behind it) can lead to conversations that sound like they belong on a middle school bus and not a well-appointed meeting room.

A: Why are we doing this?

‍PM: We just have to do it.

‍A: But why?

‍PM: Because.

‍A: Because why?

‍PM: Because I said so!

The last bit of that conversation may or may not be followed by something along the lines of, "They hired me to own this product, I spend all of my time researching the market, talking to customers, living, breathing and sleeping with this product. I am the in-house expert. It’s my neck on the line and I make sure it has the right features, it supports the right platforms and it is “surprising and delighting” users so they buy it, use it, like it and tweet about it. I obviously know what I’m doing.”And then this might happen:

Not very productive. No matter how right you (think) you are, “because” simply doesn’t cut it in business. It doesn’t matter how long your list of accomplishments is or how many years of experience you have, and it certainly doesn’t matter how accurate your gut instincts have been in the past; you need to bring more to the table than “gut feel” and “I think.” You need data.

“In God we trust, all others must bring data” - W. Edwards Deming

Data is a great equalizer among rational adults and it can even be used to tame some of the less rational ones, too. Data provides reasons for “because”, which can then be used to back up your arguments and convince people. Convincing arguments lead to consensus, and when it comes down to it, that’s really a lot of what your job is all about; getting everyone at the table nodding in agreement.Of course, stating “This is the way it should be” and then bludgeoning your audience with data point after data point is one approach to using data to solidify your position. However, by leading with the “way it should be” bit you’ll likely put everyone who didn’t enter the room on the same page on the defensive, and instead of absorbing and considering the data they’ll be looking to poke holes in it.

So rather than taking the “data as backup” approach to making your case, consider the “let the data speak for itself” method. You’ve done the research and now you are sharing the results. There was a question to be asked, an area to be investigated, the data is in, let’s take a look and see what it tells us.

Data Gives Us License to Be Bold

No one wants to be remembered as the person who greenlit an infamous failure. No one is proudly putting the Microsoft Zune or Nokia N-Gage at the top of their list of professional accomplishments. People want to be associated with success, which in turn makes them tentative to take a risk.

“Data can help fuel the courage needed to make big decisions by providing an extra layer of reinforcement. That’s why, in many ways, business intelligence is in the business of courage,” writes Morten Sandlykke, CEO of TARGIT about the psychology of decision making, “Access to data allows executives to back up their courageous decisions with more facts than ever before. With the right tools at hand, it is not necessary to be a data scientist to dig into data sets and read them like a story.”

When Spotify made the decision to switch from a white background to a black one, they already had results of A/B testing the new background in hand to not only know it wouldn’t backfire, but also that it would improve their bottom line. “Data actually gave them the confidence that something that they thought was going to be a huge risk and big change on their users was actually something that the customers wanted and not only that it was good for the business as well,”  Rochelle King, Global VP of User Experience and Design at Spotify recalled during a Ted Talk on the relationship between data and design. Numerous companies have let customer behavior data lead the way to far more significant changes than color schemes. Pinterest’s initial incarnation, which was called “Tote,” allowed users tag products they wanted to buy when there was a price drop, but the company soon found through usage data that collecting and curating items was more popular than conducting actual transactions, and Tote became Pinterest. Pinterest didn’t become what it is today because someone said “I think we should build a social network where people collect pictures of stuff” but because the data showed what users were actually interested in—data lead the way.

Big Data, Small Data, Just Right Data

Shifting your decision-making process to a data-driven approach doesn’t mean you are handing the keys over to our robot overlords and letting statistics dictate every move your company should make. Data comes in a lot of shapes and sizes, and the best path is using a hybrid approach.“Big Data on its own is often not enough to drive innovation. It can lack context,” writes Pamela Pavliscak in Data-Informed Product Design, “Combining Big Data with deep-dive studies is a way to balance personal data against respect for the individual.” The team at Etsy combines large-scale trends they’re seeing across their platform with structured experiments to explore specific scenarios.

“We try to really avoid having it seem like an either/or decision about do you do qualitative research or market research or do you do data analysis,” Alex Wright, Director of Research at Etsy explains in a recent podcast interview on qualitative and quantitative research, “We see those things as very complimentary. There are points of friction there. Some aspects of data analytics and A/B testing and experimentation have gotten sufficiently sophisticated that they’ve displaced the need for certain kinds of more traditional usability research.” Meanwhile the folks over at AirBnB are also heavy practitioners of experimentation-based decisions, and use small experiments generate data that then fuels the strategic process.

Their advice? Focus on just one metric.“It is good practice to evaluate the success of an experiment based on a single metric of interest. This is to prevent cherry-picking of ‘significant’ results in the midst of a sea of neutral ones,” advises Jan Overgoor, a Data Scientist at AirBnB, “However, by just looking at a single metric you lose a lot of context that could inform your understanding of the effects of an experiment… It is important to consider results in context. Break them down into meaningful cohorts and try to deeply understand the impact of the change you made.”

Related: Need help finding the metrics that matter?

Less is More

Just because the data is available, doesn’t mean you should use all of it. Even the NSA has learned the hard way that too much data can be a very bad thing; they’re so overwhelmed with data that they’re becoming ineffective. While it can be tempting to overwhelm your audience with the avalanche of data you have collected, it will likely do you more harm than good. Research has shown that the human mind is really only capable of concurrently maintaining three-to-five “chunks” of data, if you want your audience to really process and internalize what you are sharing, you want to stick to the highlights and keep the rest as background and backup.

Since you only have a few chances to connect with your audience before their eyes glaze over, focus on presenting the data that leads to a definitive next step: present insights. “This happens when your discoveries begin to coalesce around something of value, when you identify an opportunity to improve your situation,” explains Chris Andrews of POSSIBLE, “Insights are different from observations and discoveries because you can do something about them.”

Data Lets You Use Graphs

Graphs and charts are not unusual in the business world, but they are often left out of the strategic decision making process. After you have gone through all the work of collecting and assembling data to make your case for a certain decision, you should take the opportunity to convey it visually, which will add significant weight to your argument. “The prestige of science appears to grant persuasive power even to such trivial science-related elements as graphs,” write Cornell researchers Aner Tal and Brian Wansink in their study Blinded with Science, “Ostensibly, graphs signal a scientific basis for claims, which grants them greater credibility.”

Data Helps You Make a Case, But it Won’t Tell You Which Case to Make

No matter how much data you have, how statistically significant it is, or how convincing your graphs are, data isn’t actionable without a human element.  Data is information, but information is not a replacement for insight, reason and creativity. “The mistake is thinking data will give you the idea of what to do,” writes William Duggan, a management professor at New York’s Columbia Business School, arguing that data doesn’t equate innovation, “[Data] is not where ideas come from.

”The reliance on data to inform decision making can also lead some down the path of “improving” products just to try and improve the associated data, versus the customer experience or business as a whole.“Experiments should be run to make good decisions about how to improve the product, rather than to aggressively optimize for a metric. Optimizing is not impossible, but often leads to opportunistic decisions for short-term gains,” says Jan Overgoor, Data Scientist at AirBnB, “By focusing on learning about the product you set yourself up for better future decisions.”

Data Lets Them Reach Conclusions Themselves

Data’s most powerful attribute is that - when positioned properly - it can mitigate many of the emotional and personal baggage that everyone (yourself included) is bringing to the table. Data takes the emotion out of things. “Without emotion, we are biologically incapable of making decisions. Logic is often the last step in the process. The conscious intellectual brain steps in to produce a rational backstory to justify impulses generated in the murky corners of the unconscious mind,” says Janet Crawford of Cascadence in a Forbes article on innovation and neuroscience, “When we experience too much stress and threat, the tendency is to retreat into habitual known responses. When we feel sufficiently (but not overly) secure, we venture into new territory.”  

While everyone thinks they are being logical and rational most of the time, most decisions are actually made based on emotions - people “feel” that it’s the right decision. Numbers alone don’t make people feel. “Business decisions are made emotionally and justified rationally,” says Christoph Becker, CEO and CCO of Gyro, “A side effect of the tsunami of digital content is, too often, there is an utter lack of human relevance. That is why if you truly want to connect with business decision makers, you must make them feel. That is why you must focus on the ‘why’ of your business, the pure idea. The overwhelming desire to connect to this essence has been, and always will be, incredibly powerful.” This sets the stage for even the brightest minds relying on their feelings instead of remaining strictly rational when considering a decision. “Whether it's a personal choice or a strategic business decision, emotions often crowd out objectivity.

After all, executives are only human, too,” writes Freek Vermeulen in a HBR piece on removing the bias in strategic business decisions, “Precisely because strategic choices are such important ones, loaded with anxiety and uncertainty ... people start to 'follow their heart,' 'rely on intuition' and 'gut feeling,' overestimate their chances of success, and let their commitment escalate.” Furthermore, while most executives fully acknowledge that they should be using data to drive decisions, most are still not proficient at turning it into something actionable.“Business planning based on data intelligence and predictive analytics is still uncharted territory for many executives, particularly those without IT backgrounds,” says Morten Sandlykke, CEO or TARGIT, “This leads to an ironic challenge: The data is there to reduce the fear in decision making, yet the inability to utilize it and the lack of trust in it perpetuates ‘go with your gut’ as a standard business practice.”  

So while you can position a case as “here is all the data that proves I’m right,” a more effective strategy when dealing with those entering the discussion on the other side is to simply lay out the data in a way that will lead them to reach a similar conclusion on their own.“Create a vision for the other side to bring about discovery and decision on their part. In the end, your opponent will make the decision because he wants to,” says Jim Camp, CEO of The Camp Negotiation Institute in an article on decisions and emotions “You don’t tell your opponent what to think or what’s best. You help them discover for themselves what feels right and best and most advantageous to them. Their ultimate decision is based on self-interest. That’s emotional. I want this. This is good for me and my side.”

The TL;DR: Tips for (Data-Driven) Success

Here are some things to keep in mind while crafting your data-driven narrative:

• Challenge your own assumptions before you start. Is the data showing you something that contradicts your own ideas? Is it maybe less conclusive than you expected? There’s nothing wrong with being wrong, but it’s better to find that out in private instead of when you are putting your reputation on the line.
• Be a storyteller. You’re writing a story where customers and users are the characters and the data is the plot. Present your findings in a logical order and end on the most important item.
• Leave out the noise. If something is not swatting down a popular misconception or supporting your case, don’t include it because there’s only so much people can process. But keep it handy in case someone asks about it.
• Take your time. You’ve been working with this data for a while, and your whole day is spent worrying about your product. This is new stuff for your audience and they may not be as familiar with the nuances of your product’s features and user behavior.
• Use pretty pictures. Graphs and charts will not only make it easier for your audience to digest the data, it will also lend it an air of scientific gravitas that never hurts.
• Don’t gloat. You finally have the proof that you were right and your nemesis in engineering was wrong. This isn’t the time to claim victory, it’s the time to build consensus.
• End with a question… but not the one you want to ask. You’ve laid out all of the evidence, so it can be tempting to assume your audience has reached the same conclusion as you. But instead of asking them “You’ve seen the data, can we do this now?” try asking them “Based on what we have covered, what are your thoughts?”