Your team has built an AI agent, put it in front of users, and watched the first requests arrive. Now you need to understand whether it is becoming part of their work.
Who discovers it? Who tries it once and returns when the next task comes around? Where do users get stuck? And when the agent helps someone finish, is the improvement worth what it costs to deliver?
Those questions give you a practical framework for measuring AI agent adoption. Follow the journey from discovery through successful work, then use what you learn to improve the experience and measure the return again.
Product teams need to understand what people ask, where interactions break down, and what happens afterward. With Pendo, teams can connect agent interactions to the wider product experience and identify where users need help.
Define the work your agent should help people do
Start with a job your intended users recognize. A reporting assistant might help an administrator create and share a weekly report. Name the job, the audience, and the result that counts as success.
For this assistant, success could mean an accurate report saved and shared within 24 hours of starting. That gives you an outcome to follow across the conversation and the product itself.
Keep people, tasks, and conversations separate. One person may complete several tasks, and one task may span several conversations. If you measure completed tasks, use a consistent task identifier so repeated attempts don’t inflate the count.
The frequency of the job matters, too. Weekly reporting gives people a weekly reason to return. Interpret continued use in the context of how often users need the agent.
For background or proactive agents, start with the event that triggers assistance. Track whether the agent offers help, whether users accept it, and whether the work gets done.
Follow discovery, first use, and repeat use
Start with three questions: can people find the agent, do they try it, and do they return? These measures complement the 10 essential KPIs to measure AI agent performance by showing how the agent becomes part of users’ work.
Discovery: Can users find the agent where they need it? Use Pendo Product Analytics to understand where the relevant work begins. For the reporting assistant, that might be the reporting page. Track how many eligible users are shown its entry point, then use targeted guidance to help more people find it. Being shown an invitation doesn’t guarantee someone noticed it.
First use: Do users make a first request? Build a funnel from the agent’s entry point to a first request, using a segment of people who haven’t used the agent before. Make sure those events are captured. If users drop off, test a clearer invitation or a suggested starting task.
Repeat use: Do users come back? Agent Analytics’ built-in retention metric measures the percentage of visitors who submitted prompts in at least two different weeks within the selected date range, out of all visitors who submitted prompts during that period. Choose a reporting period that gives users a meaningful opportunity to return.
Look at the use cases and accounts behind the average. Pendo Agent Analytics helps you explore what people ask for and the issues they encounter. Use those findings to investigate what brings people back and what discourages them.
Measure completed work and the places people give up
Once people are using the agent, check whether they accomplish the task they came to do.
Consider a hypothetical month in which users start 600 reporting tasks and complete 420 accurate reports, saved and shared within 24 hours. The task completion rate is 70%. Count all task starts, including attempts that fail or stop, and allow each task its full completion window before reporting.
Pendo’s custom success metrics let you define successful conversations. Event metrics track whether conversations lead to specified product actions; hybrid metrics combine conversation success rules with those actions. If you need a task completion rate across several conversations, track and deduplicate the tasks separately.
Keep quality in the definition. A shared report with incorrect data still needs attention. Review a sample of completed work and account for the time users spend checking or repairing it.
Investigate incomplete tasks to understand whether users encountered a problem, needed human help, or finished somewhere your tracking couldn’t observe. Define when you consider a task abandoned, and track transfers to a human separately. A conversation going quiet doesn’t establish that someone gave up.
Use agent issues and product paths to understand where the work stopped. Distinguish unsupported requests from confusing next steps, and keep unknown outcomes visible so missing data doesn’t inflate the success rate.
Use Pendo to address the adoption bottleneck
Use what you learn to choose the next improvement.
If users succeed after trying the agent but few discover it, help more people find it. If people see it and rarely make a first request, explain what it can do. If they try it and disappear, examine the first-task experience and whether the agent supports a job they need to repeat.
Pendo for Agents brings Agent Analytics and Agent Toolkit together. Agent Analytics helps you understand usage, issues, and outcomes. Agent Toolkit, available in open beta, gives agents product and user context so they can offer more relevant help.
For the reporting assistant, that could mean surfacing a relevant Pendo guide when a user struggles to save a report, informed by what they’ve already done. Configure the capabilities your agent needs, then measure whether users complete the task more often. Agent Toolkit and Agent Analytics have separate setup.
Your team can also investigate adoption from its existing AI tools. With Headless Pendo, a PM can ask directly in Claude about agent use cases and issues, then bring product usage and feedback into the investigation.
From there, save an affected audience and draft an in-app guide for the reporting workflow. A person reviews and activates the guide. Afterward, check whether more users succeed and whether the change was worth the effort.
Useful agent work can span conversations, connected tools, and the product interface. Include those experiences when assessing adoption so you can follow the work through to its outcome.
Make the comparison fair
Compare similar users doing similar work over comparable periods, using the same success and quality standards. Give users enough time to finish, and account for other changes that could affect results. Higher completion among agent users shows an association; a randomized rollout or carefully designed controlled comparison can help isolate the agent’s contribution. Include people assigned access who never use it.
Agent Analytics experiments let teams compare configurations, models, tools, and other groups. What you can attribute to a change depends on how participants are assigned and the conditions being compared.
Connect successful adoption to ROI, then improve it
For an internal reporting assistant, time saved is one way to estimate value. Compare the full effort required with and without the agent, including review and rework.
Suppose the 420 completed tasks in our hypothetical month each save five minutes compared with the same work without the agent. That adds up to 35 hours. At an assumed fully loaded labor cost of $60 an hour, the estimated value of time saved is $2,100.
If the agent’s total allocated cost for that month is $1,400, the net estimated value is $700. Divide $700 by $1,400 to get a modeled ROI of 50%.
Include implementation, model and infrastructure charges, maintenance, evaluation, and human oversight in the cost. This example assumes comparable completion rates and quality, with no additional effort on incomplete tasks. Account for those differences if they change.
Time saved creates capacity; it becomes cash savings when spending changes. For a customer-facing agent, establish how improved customer outcomes benefit your business before assigning a financial value. Use actual operating costs and avoid counting the same work twice. Our guide to how product builders can prove AI ROI goes deeper into that business case.
After each change, check whether more users complete the task, whether quality holds, and whether the benefit justifies the cost. Repeat the comparison using the same standards and full costs.
Start with one agent and one improvement
Choose a job your agent should make easier. Define who can use it, how they discover it, what successful work looks like, and when they should have a reason to return.
Find where adoption stalls, use Pendo to understand and improve that experience, and measure the result against the full cost. You’ll have a clearer basis for expanding a successful use case, improving a weak one, or focusing investment where the agent creates the most value.
Explore Pendo for Agents to connect agent adoption, product context, and the outcomes that help you decide what to do next.
