At last month’s ProductCraft Conference event in San Francisco, Niels Hoven of Cloudflare talked about a particularly widespread problem in product management. As PMs, we should be seeking the truth. After all, only accurate information can help us build valuable and delightful products. However, some product professionals are moving away from the truth and toward what Niels referred to as “truthiness.”
Fans of humorist Stephen Colbert already knew what Niels was talking about, but he took a moment to explain it. Truthiness, he said, was being truthful, but not quite. “Truthy” statements seem like the actual truth. And yet, they don’t encompass the whole truth.
And the whole truth is what product professionals should be after — not truthiness or almost-truth. In his session, entitled “The PM’s Guide to Truthiness in Data,” Niels provides some examples of truthiness in the PM world, his philosophy on metrics, and a few strategies for getting at the actual truth about your product and its performance.
Some common “truthiness” strategies
To show how widespread (and tricky) truthiness can be, Niels chose some examples from the tech world. He didn’t name names, but each instance demonstrated how a major company could use truthiness to appear more successful than they actually were.
No. 1: Pick carefully
According to Niels, pick your metrics carefully is one quick way to get to truthiness. He shared an example from a well-known casual gaming company, which stated they had reached one billion downloads. That’s an impressive number, to be sure. And Niels verified that yes, this count was accurate.
However, total downloads are probably not the most important KPI for a gaming company. A better metric would likely be monthly or daily active users or any measure of true engagement. If people are just downloading the app and playing a game once or twice, the product isn’t really as successful as that “one billion downloads” number would imply.
No. 2: Count carefully
Another truthiness strategy is to “count carefully,” or calculate your metrics accurately while defining them in an unanticipated or misleading way. Once again, Niels cited an example from an unnamed but well-known gaming company. They stated that they had 232 million monthly active users — a great number, right? However, in the fine print, the company explained that any user who played more than one game counted twice as an MAU. So, that number was only half as impressive as it seemed on its face.
No. 3: Get creative
When all else fails, you can get just get creative with your numbers. For this example, Niels selected a popular ride-sharing company, which stated that 91% of drivers drove less than 20 hours per week. That, of course, begged the question: how many of that 91% are actually driving zero hours per week? Are large numbers of users driving “less than 20 hours per week” because they are not driving at all?
“Truthiness” and the metrics fallacy
The danger of truthiness lies in its ability to keep your team from seeing the whole picture of how users are interacting with your product. To reiterate this point, Niels Hoven brought up the parable of the blind men and the elephant. One man grabs a leg and states, “This is a tree.” Another touches the trunk and says it’s a snake. Without a holistic view of their product, PMs may be flying blind when it comes to their customers’ journeys and experiences.
Niels also reminded attendees of an important truth. “What gets measured gets managed,” he said. “And what gets measured also gets gamed.” When data becomes a goal instead of an indicator, that’s a sign of problems to come. This is the heart of what he called the “metrics fallacy.” These two statements are not equivalent:
- “If the product is doing well, then metrics will improve.”
- “If metrics improve, then the product is doing well.”
Your product team should have a mindset based on the first statement, not the second. The focus needs to be on bettering the product, not increasing a potentially arbitrary number.
Holistic product management
Holistic or “whole-picture” product management is what all PMs should strive for. According to Niels, a holistic product management mindset puts:
- Product management over metric management
- Being data-informed above being data-driven
- Narratives over numbers
- Stories over statistics
Don’t put blinders on your product strategy by narrowing your view to a handful of KPIs. While hard data can be very useful, valuable information can also come from qualitative sources, such as customers, competitors, and your own colleagues. The best product strategies are a combination of analytics, user feedback, market data and trends, and your team’s core competencies. By looking at the whole picture rather than just a few KPIs, you’ll have more chances to spot new opportunities for improving your product and keeping your users happy.
