How product builders can prove AI ROI

After weeks of building and testing an AI assistant, it’s a relief to see customers start using it. Then comes the part where you have to explain whether the time and money were well spent.

Product teams have good reason to prepare for that conversation. According to a recent KPMG survey, 49% of organizations reported delaying or scaling back AI agent deployments when costs started outweighing benefits.

Getting a clear answer on ROI can be surprisingly difficult. Your assistant might receive more prompts while users still abandon setup. Customers might like its answers but spend extra time checking them. Even encouraging engagement numbers leave you with work to do before you can put a dollar value on the improvement.

To prove AI ROI, start with a task your users need to complete. Measure whether the AI feature helps them succeed, compare the result with the existing experience, and connect the improvement to a benefit you can value. Include the full cost of building and running the feature.

That gives you a business case you can explain and a basis for deciding what to build next.

Start with the user’s job

Choose one task your AI feature should improve. Helping a new administrator finish workspace setup gives you something concrete to measure: a person with a specific job, a sequence of actions, and a clear result.

Define success before you evaluate the feature. For setup, that might mean completing the required configuration within seven days, without errors that require support.

Then identify who has the opportunity to use the assistant. If it’s available only to administrators starting setup, measure adoption within that group. Using all active customers as the denominator would obscure how well you’re reaching the intended audience.

Follow everyone who starts the workflow, including people who abandon it, encounter an error, or never open the assistant. You’ll need those experiences to understand whether the feature improves setup overall.

Keep quality alongside completion from the beginning. A faster setup process creates little value if customers have to return the next day to fix their configuration.

Establish a credible baseline

Compare the AI workflow with the experience customers would otherwise use. That could be your existing interface, a support-assisted process, or a version of the workflow without the assistant.

Use the same task definition and measurement window for both groups. Account for differences in customer experience, account size, and access to help. A rollout that coincides with a redesigned onboarding flow will require extra care to distinguish the effects.

A randomized experiment can help you attribute an improvement to the AI feature. Choose how to assign participants based on how they work. If administrators in the same account share setup responsibilities, assigning whole accounts may give you a cleaner comparison than assigning individuals.

You can also learn from comparable cohorts or a before-and-after analysis. Be clear about the limits. Customers who choose to use an assistant may already be more motivated or more comfortable with your product. Consider those differences before attributing a higher completion rate to the assistant.

Measure usage alongside task success

Adoption helps you understand whether customers are trying the feature. To evaluate its value, follow what happens after they try it.

For an onboarding assistant, a useful measurement plan might look like this:

Interpret engagement in the context of the job. An assistant that helps someone finish setup in two exchanges may be doing better than one that needs ten. More prompts could reflect growing interest, repeated failed attempts, or a more complicated task. Examine the conversations and completion data before deciding which explanation fits.

Pendo Agent Analytics helps teams inspect conversations, identify use cases and issues, and connect agent interactions to subsequent product behavior. Use that context to investigate where customers succeed and where they get stuck. With Headless Pendo, you can explore agent adoption, retention, and issues directly in connected tools like Claude and ChatGPT via the Pendo MCP, without opening the Pendo app.

Check your measurement coverage, too. If you can’t connect a conversation to the user’s subsequent actions, you may need to address an instrumentation or identity gap before drawing conclusions.

Review a sample of conversations against the corresponding task results. An assistant can give a helpful answer while the customer still struggles to complete setup. Checking both helps you define success in terms that reflect the customer’s experience.

Connect the improvement to business value

Once you’ve measured an improvement, establish what it’s worth.

Faster task completion can free up capacity. Fewer support escalations can reduce the work required to serve customers. Better activation or retention may contribute to revenue.

Each benefit needs an explanation that connects the observed change to its financial effect. If employees save time and use it to handle more work, describe the additional capacity and output. To claim realized savings, identify the spending that changed, such as lower vendor costs, fewer paid hours, or an avoided hire.

Retention requires the same care. Returning users may be finding value in the feature, but you’ll need to examine renewals, expansion, or another relevant commercial outcome to estimate the revenue effect. Label projections and assumptions so readers can distinguish them from observed results.

Match the costs to the same period as the benefits. Include implementation, instrumentation, model-provider charges, infrastructure, human review, and maintenance.

Also check for overlap between benefits. Counting the same avoided support work as both labor savings and additional staff capacity would overstate the return.

Learn from Pendo’s own AI agent

Our experience improving Leo, Pendo’s AI product assistant, shows how this measurement process can inform product decisions.

Using Agent Analytics, the team identified customers asking complex analytics questions that Leo struggled to answer. They examined the frequency and types of requests, then used those findings to prioritize improvements to Leo’s quantitative tooling.

Before expanding the release, the team tested the new tooling with a cohort representing 20% of the user base. The issue rate dropped by 67% for that cohort.

They also saw a 61.3% increase in weekly returning visitors over the preceding 60 days. That broader engagement result covers a different scope from the tooling experiment.

The team had evidence to support a product decision: they understood a recurring customer problem and had tested whether their changes improved the experience. Calculating financial ROI would require the additional cost and benefit analysis described above.

You can follow the same process for your own feature. Identify the problem, prioritize a fix based on its impact, and test the outcome before expanding the rollout.

Use the evidence to decide what comes next

Bring your recommendation into the business case. Explain whom you measured, what you compared, how the outcome changed, and what it cost to deliver that change.

Expand when customers achieve the intended outcome and the benefit supports the investment. Focus the feature on a narrower audience if that’s where you find value. If the results fall short, use the evidence to decide whether to improve the experience, change the use case, or stop investing.

You can make a useful decision before every financial assumption is settled. Be specific about what you’ve observed, what you’ve modeled, and what you still need to measure. That gives product, engineering, and finance a shared basis for the next investment.

Start with one workflow where you need a clearer view of the outcome.

Ready to start proving ROI for your own AI agent? Explore Pendo Agent Analytics to connect agent interactions to user and business outcomes.