E-BOOK

How to become a data-driven product manager

Scroll to read 

Introduction

In the age of big data, it feels like everything has to be data-driven, a prefix attributed to nearly every function in our professional and personal lives, and often without any action to back it up. Buzzword aside, data is central to a product manager’s (PM’s) job.

Gartner predicts that by 2021, 75% of software providers will rely on insights from embedded software analytics to inform product decisions and measure customer health.

But having data that you can make sense of, have confidence in, connect to the rest of your systems, and access easily and quickly is easier said than done. While this is always going to be a challenge, product managers need to recognize the opportunity in front of them: to leverage data to bolster their decisions, inform new ideas, and ultimately, create a better experience for customers.

To help shed light on how product practitioners of all levels are thinking about this, we interviewed ten experts to hear how they view a product manager’s relationship with data (whether you prefer to call it data-driven, data-informed, or none of the above).

Let’s dive in.

What does it mean to be data-driven?

In Pendo’s 2020 State of Product Leadership report, product managers said they see their decision-making process as more reliant on data than instinct. But like many things in product, everyone (or every company) has their own methods and overall philosophy about data.

When we asked product pros what data-driven product management means to them, the answers varied, but had some clear overlaps.

Greg Bayer, SVP of Product, Nielsen

Product management which follows the philosophy of making customer and market- informed decisions about your product direction, as opposed to gut thinking or listening to just one or two internal points of view.

Greg Bayer /// SVP of Product
Nielsen logo

Sam Benson, Product Operations Specialist, Firefly

It’s essential to always learn and get a better understanding of our users and what is important to them when considering different changes or improvements to the product. To me, the best way to monitor that on a day-to- day basis is through analytics.

Sam Benson /// Product Operations Specialist
Firefly logo

Please fill out the form below to read the rest of this e-book.

Rekha Venkatakrishnan, Senior Manager, Group Product Management

To me, data-driven product management is about leveraging data through multiple venues to make informed decisions as you work through your product hypothesis and throughout the entire product life cycle.

Rekha Venkatakrishnan /// Senior Manager, Group Product Management

Bella Renney, Head of Product, Tray.io

I think being data-driven means using data at the right time, with a heavy emphasis on the richness of qualitative data. It’s less looking at metrics and dashboards every single day, and more that data feeds into everything you do as a product manager.

Bella Renney /// Head of Product

Travis Turney, Senior Data Strategist, Rapid7

I like to think about it as being data-informed, which to me means enabling people with data they can be confident in, that’s timely, and allows them to make decisions in response to customer needs—giving them enough data to be useful, but not so much that it slows them down.

Travis Turney /// Senior Data Strategist
Rapid7 logo

Manosai Eerabathini, Product Manager, Google

The overall philosophy I think about is harmonizing a balance of signals gathered from data to better inform your intuition. I think there’s always an element of interpretation and insight gathering that needs to happen, even with data.

Manosai Eerabathini /// Product Manager
Google logo

Beatrice Fabris, Manager, UX, Content & Development, mimecast

Data helps provide objectiveness, so I think being data-driven is about looking at the data first, while still acknowledging that there’s always a place for gut instinct and experiences to drive decision making.

Beatrice Fabris /// Manager, UX, Content & Development

Viraj Phanse, Senior Product Manager, Amazon Web Services

It is about making informed decisions based on data, and making data from various sources actionable—all geared toward providing the customer with a great experience. Most importantly, it helps connect the dots across the organization by capturing upstream and downstream information and channeling it for customer success.

Viraj Phanse /// Senior Product Manager
Amazon Web Services

From the responses, we can pull out a handful of key themes: using data to make decisions in service of your customers, leveraging data throughout the product life cycle, collecting data from multiple channels, and balancing hard data with your own instincts. We’ll dig into each of these in more detail throughout the rest of the e-book, but let’s start with the elephant in the room: the human side of data analysis.

  • Using data at the right time
  • Balancing signals
  • Using the right amount of data
  • Intuition
  • Entire product life cycle
  • Actionable data
  • Addressing customer needs
  • Understanding users
  • Qualitative and quantitative
  • Data-informed
  • Connecting the organization

There is no data-driven product management without the PM

In his latest book, 21 Lessons for the 21st Century,, historian Yuval Noah Harari writes that “in a world deluged by irrelevant information, clarity is power.” For product managers, that irrelevant information (and hopefully, plenty of relevant information) is data, coming from a multitude of sources. It is then up to the PM to make sense of it.

Beatrice Fabris, Manager, UX, Content & Development, mimecast

For all of the value and“objectivity that comes with the data, there are enough variables or situations where the human component is always going to be part of the experience. Having that objectivity but also bringing in your own insights, experiences, and gut instinct is the better way to go about it.

Beatrice Fabris /// Manager, UX, Content & Development

As we heard in the previous section, “data-driven” doesn’t necessarily mean blindly following the data. In fact, the person tasked with analyzing the data in front of them is equally as important (if not more so) as the data itself. And there are two keys areas where the human element of data analysis can’t be swapped for an algorithm or automated system.

Tatyana Mamut, Head of Product, Nextdoor

The data is what it is, but the data never tells you what a decision should be.

Tatyana Mamut /// Head of Product
Nextdoor logo

1. Making decisions

No matter how informative or eye-opening the data may be, there is always going to be a need for some sort of human judgement. According to Tatyana Mamut, head of product at Nextdoor, the job of a product leader is to interpret the data effectively, understand its meaning, and then move forward depending on whether the situation requires a decision based on empirical observation or values (or both).

Travis Turney, senior data strategist at Rapid7, echoed this sentiment: “You’re never going to have a program where you put in some inputs and it spits out exactly what the solution is. You can’t have data analysis without a human in the loop.”

Sure, there are some questions in product management that can be answered with a single data point—which feature has more usage? What’s our average NPS score? What percentage of X type of users did we retain after three months?—but there are many more that cannot be.

If you’re trying to figure out why users are dropping off at a certain point in the workflow, or how usage has changed for a particular feature, or what version of an in-app guide will drive the most action, there needs to be a human decision- maker involved.

2. Talking to customers

Everything a product manager does is in service of the customer, and the best way to understand what your customers need or want from your product is to talk to them directly. Greg Bayer, SVP of product at Nielsen, says that spending time with customers periodically to get feedback on specific “ features, help with roadmap planning, or even learn how they use complementary technologies “informs a macro point of view for a product manager that is critical for the full vision.”

Even if you think you know what customers want based on how they interact with (i.e. from product usage data) or feel about (i.e. from NPS or CSAT surveys) your product, nothing can replace the insights you gain from face-to-face (or virtual) conversations. Sam Benson said that for the product operations team at Firefly Learning, it’s extremely important to pair quantitative data with qualitative research from user interviews to understand the “why” at every stage of the discovery process.

Sam Benson, Product Operations Specialist, Firefly

We use product analytics in our ‘learn’ phase to find gaps in our product and understand user workflows. We then build on that with the human element and interview users to really empathize with their experience and identify how to improve it.

Sam Benson /// Product Operations Specialist
Firefly logo

Product managers must use data responsibly

The more data product managers use, (ideally) the more value it brings. But something Beatrice Fabris, manager, UX, content, and development at Mimecast, realized quickly is “ that you can often spin the data any way you want. Although quantitative data is thought to be objective, it can still be misused depending on how the human behind the analysis chooses to present it.

Product managers have a responsibility to use data in a way that doesn’t skew or hide the full picture. There will always be a need for product managers to use their own intuition, experiences, and opinions to make the right decisions. But data plays a big part in informing these decisions at every stage of the product development lifecycle.

Beatrice Fabris, Manager, UX, Content & Development, mimecast

That’s something I view as a bit of a concern because people often assume that once you have the data, it’s foolproof. But all the person needs to do is show one part of the frame rather than another part, and it’s an entirely different story.

Beatrice Fabris /// Manager, UX, Content & Development

How product managers use data

While data is always useful for a product manager, it’s useful in different ways. Rekha Venkatakrishnan, a senior manager of group product management, sees data as an ongoing part of the product development life cycle, but the key is to ensure you know the data you want to use (and why) to get to your goals.

Beatrice Fabris from Mimecast echoed this sentiment: “I wouldn’t actually say there’s one area where data is the most valuable. Data can be used throughout the whole process. Making sure we use the data in the right way is where we get the value from.”

She went on to explain that on the UX team, they approach data differently depending on the phase a product is in. Before a new feature release, she looks at where users spend a lot of time in the product to inform how they release complementary in-app guides. For a product that’s already out in the market, they use data to identify areas that need to be improved, and to benchmark in-app guide performance.

All that being said, it’s worth highlighting areas of the product development and customer life cycles where data can play an important role:

Discovery

In the discovery process, product managers can leverage data to identify any gaps in the product to inform what changes or updates to pursue. On the flip side, it’s also useful to look at your most used features to see if there are any ways to improve that experience for your users. If you know a particular feature is associated with key customer outcomes, you should always be looking for new ways to make it better.

Roadmapping

Product usage data is valuable for directly informing the product roadmap (i.e. which features are customers using the most, and therefore require more attention?), but Viraj Phanse, product manager at Amazon Web Services, brought up another scenario. He said that sometimes, a product manager will have an idea for the roadmap but no data available to back it up. In this case, PMs still need to make the call one way or the other and take smart risks. Adopting a customer- centric approach can help as well: focus on your customers’ problems throughout the roadmap journey, and constantly ask yourself what customer problem you’re solving.

Feature launches

For new feature launches, Greg Bayer of Nielsen says his team always tracks adoption and usage first and foremost, including: how many customers are using the new feature, and is it leading to more overall engagement in the product? He emphasized the importance of linking success criteria with your customers because at the end of the day, you want them to be successful because they used your product.

You should also analyze a new feature’s success to understand any usage patterns and what customers might still need from the product. Most importantly, make sure you have tracking in place before going into a new feature launch. Travis Turney of Rapid7 believes the metric itself doesn’t matter as much as having some sort of measurement program so you can learn over time what “good” looks like.

Onboarding

When building an onboarding strategy, product data can help you determine which features to include in your onboarding flow (aka those that are most likely to lead to success in the product) as well as help you measure onboarding effectiveness.

Travis Turney of Rapid7 also called out onboarding as an important time to learn about your customers through the data you collect. Since customers are more likely to feel comfortable providing information about themselves when they first start using your product, Travis says it is your best opportunity to capture data (e.g. their role in the company) that can then allow you to provide a more personalized product experience going forward.

Experimentation and testing

The experimentation process hinges on data as you test, analyze, and learn what works and what doesn’t. Manosai Eerabathini of Google sees data as crucial to experimentation, especially for products that are well established and where testing and implementing small tweaks to the user experience can make a large impact.

Andy Browning, UX writer at Mimecast, explained this from the perspective of the UX team: “We found that when we do user testing, we have our own ideas and instincts going in, but then we actually see the users engaging with the product and get the data, and we might be completely wrong. And you can’t really argue with it—you know where you need to make improvements.”

Retention

Retention is always important, but it has become even more critical for businesses to stay ahead of retention by understanding how customers are using the application. Sam Benson explained how at Firefly Learning, customer success managers (CSMs) need to have an idea of their customers’ user journeys in order to have productive conversations, especially when it comes time for renewals. This way, CSMs can offer the right recommendations tailored to each customer account’s usage.

You can learn the most from customers who don’t stick around

In addition to leveraging data to proactively address retention, Bella Renney, head of product at Tray.io, believes the most valuable information comes from talking to the customers who don’t renew their subscription or continue using your product:

“I’m much more interested in the customers who came to the tool, used it, and then decided that it wasn’t valuable. Why did they churn? I think that’s the most important piece of data you can get as a product manager to make prioritization decisions.”

Data doesn’t start and end with the PM

When used effectively, product data exists as a shared language across the company. Nearly every team has a
reason to care about how the product is performing or how customers are using it, especially at organizations that are “ product led. So, a big part of a product manager’s job is to empower the rest of the organization with the data and insights they’ve collected.

Here are some tips for sharing product data effectively:

  • Ensure all stakeholders have access to product data
  • When sharing product data, deliver the information that’s relevant to each type of stakeholder (e.g. different teams)
  • If you have a north star metric, utilize that as your mechanism for sharing product data
  • Include product data in what’s shared at company-wide all-hands meetings or town halls
  • Provide context as to why the data is important to each stakeholder or team specifically
  • Use Slack (or another communication channel) to share product data and foster collaborative conversations

Greg Bayer, SVP of Product, Nielsen

I make sure all stakeholders have access to broader reporting data if they want, but I always find it most useful to hold a monthly show and tell whereby the product team talks about recent releases and their value to clients.

Greg Bayer /// SVP of Product
Nielsen logo

A product ops use case

When product ops is an established function in an organization, it is often tasked with owning product data—including collecting it, helping analyze it, and making it easily accessible for other teams. Sam Benson of Firefly Learning does just that:

“I work with all of our teams at Firefly to help relieve their workload and enable them to dig deep into user data via product analytics systems like Pendo, to make sure we’re providing a platform that our users love.”

To offer examples, Sam said she analyzes user workflows and customer data, then reports trending issues to cross- functional teams throughout the company. The product ops org also makes sure every team across the business is nudged into utilizing the tools, data, and insights the product team provides. Sam also helps funnel product ideas and feedback from customers back to the product management team.

With all of the different ways product managers use data, it begets the question: which metrics should they use?

Metrics that matter

Using data to build a great product is only half the story. Product managers need to track things like product usage, user sentiment, and customer feedback in order to continuously provide the best product experience and ensure customers’ needs are met. Even Sweetgreen, a fast-casual salad chain, identifies as a technology company, and is shifting to tracking metrics like active users in their app and lifetime customer value rather than just store sales or foot traffic.

Humans naturally gravitate toward information that is organized and when possible, simplified. And one of the most difficult exercises can be determining which product metrics to measure and then, which metrics matter the most. When we asked our interviewees what the most important metrics are for PMs to measure, there was one overwhelming consensus: it all depends.

Here are some of the factors that impact which metrics product managers should track:

  • Whether your company is B2B, B2C, or a hybrid
  • Your industry (including Saas vs. non-Saas, enterprise
    vs. consumer)
  • The stage your product is in
  • Your organization’s goals (e.g. new customer acquisition or retention)

Since there is no one-size-fits-all for product metrics, here are some different ways to approach choosing what to focus on:

Manosai Eerabathini, Product Manager, Google

I think at the end of the day there needs to be a way to connect the dots back to what the overall business objectives are. If you can’t do that, then it probably means you’re not focusing on the right thing.

Manosai Eerabathini /// Product Manager
Google logo

Travis Turney, Senior Data Strategist, Rapid7

You’re going to have a trade off between quantity of data and completeness—do you need a lot of data or do you need fewer data points that are more complete, like written responses from a survey? This is when you might need to employ multiple methods for data collection, for example running a sentiment survey in-app and through email.

Travis Turney /// Senior Data Strategist
Rapid7 logo

Rekha Venkatakrishnan, Senior Manager, Group Product Management

As someone who has worked on B2B and B2C products, it’s all about customers first: acquire, engage, and retain.

Rekha Venkatakrishnan /// Senior Manager, Group Product Management

Greg Bayer, SVP of Product, Nielsen

I’ve worked primarily in B2B where I look for success around feature adoption and activity— how often are clients logging in, or how many individuals are using the product per client. If there is sales data available, I also look at how different KPIs (e.g. ad revenue) are trending based on feature adoption.

Greg Bayer /// SVP of Product
Nielsen logo

Two example metrics frameworks

1. Viraj Phanse of Amazon Web Services shared a metrics framework whereby you categorize metrics by the internal team or external stakeholder they are relevant to. Here are sample metrics for each of his four defined categories:

  • Management: revenue, profitability, Net Promoter Score (NPS), retention, customer lifetime value (CLV)
  • Customer: NPS, customer satisfaction (CSAT) score, customer effort score (CES)
  • Marketing: CLV, conversions, growth in daily and monthly active users
  • Engineering: number of issues, velocity, click-through rate, NPS

2. Tatyana Mamut of Nextdoor shared the metrics framework her team developed, which is broken down into four categories as well:

  • Product value: these are lagging indicators of the product’s success, for example engagement metrics (e.g. DAU/ MAU) and revenue.
  • Product love: these help the product team understand if they are not only building a product that customers use, but one that they love. Ideally, they are leading indicators like week 4 WAU and NPS.
  • Product purpose: most companies have a purpose statement, but don’t necessarily measure their effectiveness at delivering on that purpose. At Nextdoor, the team does this through a Neighborhood Vitality metric, which they track by looking at the number of High Quality Connections that are made (online and offline) and the amount of reported content on their platform.
  • Employee passion: since you can’t create a great product without passionate employees, Nextdoor measures employee passion (via frequent surveys and constant feedback) and elevates it to the top level of their measurement framework.

The north star metric

The idea of a north star metric is a common practice for product teams looking to narrow in on one data point. Since it can be challenging to balance all of the data coming in from multiple places, a north star metric helps teams focus. The only downside being that this could cause product managers to lose sight of other product signals that matter, too.

Manosai Eerabathini of Google wasn’t always a believer in the north star metric, but has come to recognize the value of collapsing your focus and being able to point to one overall metric to track progress. His approach to the north star metric is to build a hierarchy of supporting metrics that ladder up to your north star.

At Tray.io, the team created a custom north star metric called a user score, which aggregates multiple data points (e.g. daily active users, clicks, and other funnel metrics) into one high-level view. Bella Renney says she looks at this metric daily, usually by examining the number of users who are in three different thresholds of scores. This aggregated score also allows her team to drill into underlying data points to understand why a particular customer’s score has increased or decreased.

Her advice for deciding on a north star metric or compilation score of your own? “Take a long, hard look at what success is for your users and build your score around that specifically.”

Manosai Eerabathini, Product Manager, Google

If a PM doesn’t have conviction around their ultimate goal, then it’s going to be difficult to parse out what you’re trying to optimize for. When you have a bunch of different signals you’re either going to run the risk of randomizing yourself, or you will be whip-lashed around with what the data is telling you.

Manosai Eerabathini /// Product Manager
Google logo

Balancing quantitative and qualitative

Another theme that came up in nearly every conversation is that PMs need to balance quantitative and qualitative data. According to Greg Bayer of Nielsen, “a good product manager needs to blend both together equally.”

Here’s what that can look like:

At Mimecast, Beatrice Fabris says they use their product analytics tool to validate whether the problem areas the team hears most often in customer calls match where users are spending the most time in the product. The data complements qualitative information coming from support calls in order to form a more complete picture.

Travis Turney of Rapid7 said the power of quantitative and qualitative data was especially crucial when the team worked to optimize their onboarding flow. By combining information from support tickets and product usage, they were better able to identify areas of the product where they could invest more and provide a better experience for new customers.

Viraj Phanse of Amazon Web Services offered an example related to understanding customers’ problems—he believes product managers need to actually be on calls with customers in order to understand the problems they’re facing.

From there, it’s a matter of digging into the quantitative side of things by looking at NPS survey responses or cohort analysis for the particular feature where the customer is having trouble.
Next, we’ll hear what our experts believe product managers need in order to use data effectively.

Travis Turney, Senior Data Strategist, Rapid7

When you have quantitative data like product usage or survey responses based on a ten- point scale, it’s important to combine that with qualitative data like open-ended text-based surveys or customer feedback. I think that’s where you find the greatest insights.

Travis Turney /// Senior Data Strategist
Rapid7 logo

How to be a better data-driven product manager

To help fellow product leaders continue improving, we asked our interviewees to name the one skill or tool that a data- driven product manager needs to do their job. Here’s what they had to say (PMs, take notes).

Tatyana Mamut, Head of Product, Nextdoor

There are two skills that ten“d to separate the great product leaders from the rest. One is intuition for the customer, built out of spending countless hours with customers and using the product as the customers would use it. The second is courage to make leap-of-faith decisions, even when the decision is unpopular or the data is ambiguous or non-existent.

Tatyana Mamut /// Head of Product
Nextdoor logo

Greg Bayer, SVP of Product, Nielsen

Use Excel wisely. Be able to create pivot tables, trend graphs, and tell a story with a large, raw data set.

Greg Bayer /// SVP of Product
Nielsen logo

Sam Benson, Product Operations Specialist, Firefly

I’d say it’s a product manager’s ability to interpret the data given to them, and how they then present it to others.

Sam Benson /// Product Operations Specialist
Firefly logo

Rekha Venkatakrishnan, Senior Manager, Group Product Management

Invest your time in understanding the data— experiment and play with it, and finally, learn from it. There is no time boxing to this and it is a continuous skill to hone.

Rekha Venkatakrishnan /// Senior Manager, Group Product Management

Beatrice Fabris, Manager, UX, Content & Development, mimecast

You can be the best PM in the world and track everything so diligently and make sure that all of your JIRA tickets are in order. But at the end of the day, if you’re not able to communicate with the designer who is passionate about their own meticulousness, or the vice president who has no time and needs to understand everything in two seconds, then it makes your own job as a PM much harder.

Beatrice Fabris /// Manager, UX, Content & Development

Viraj Phanse, Senior Product Manager, Amazon Web Services

Statistics and data visualization are two skills that all product managers should acquire. They help PMs convert data into meaningful insights, run user experiments, test product hypotheses, and present data to stakeholders in a meaningful way.

Viraj Phanse /// Senior Product Manager
Amazon Web Services

Manosai Eerabathini, Product Manager, Google

Establishing a north star— some high level metric or goal that you can look up at, especially when you’re deep in the weeds and wondering if you’re still on course. I find that really grounding and I would advocate for all product managers to do that.

Manosai Eerabathini /// Product Manager
Google logo

Andy Browning, UX Writer, mimecast

From the UX perspective, I think you really need to deeply understand your users because once you get any data, it’s about how you interpret it to create a better experience for the end user.

Andy Browning /// UX Writer

Travis Turney, Senior Data Strategist, Rapid7

Having data literacy is key. People who really understand data and have that fluency are able to choose goals that are going to help them predict success better than the folks who don’t have that knowledge.

Travis Turney /// Senior Data Strategist
Rapid7 logo

Keep reading