Pushpay empowers mission-based organizations to engage their communities by bringing people together and fostering meaningful connections. Our innovative suite of products helps build cultures of generosity by streamlining donations, enhancing communication, and strengthening relationships.
Software
201-1k
NA
Redmond, Washington
Pushpay introduced a natural language search agent to help church administrators find data more intuitively, which ultimately enables them to connect and engage with their people more effectively. But post-launch, the product team did not have a clear way to measure if it was working. They were relying on engineers to extract prompt logs, piecing together JSON outputs, and struggled to understand what users were asking, or why they were quitting mid-flow.
Pushpay implemented Pendo Agent Analytics during the beta phase of the agent rollout, giving the team early visibility into prompt behavior, friction points, and emergent use cases. They tracked user frustration, surfaced unexpected query patterns, and used replays to understand exactly where and why users dropped off.
The team gained full visibility into user prompts and behaviors, identified prompt fatigue thresholds, and uncovered emergent user needs. This drove prompt tuning, filter enhancements, experience redesigns, and user education. With clear data and tighter feedback loops, Pushpay evolved the agent into a scalable, trusted experience that helps users find information in under ten seconds (down from one to two minutes).
Pushpay builds donor engagement and church management software for faith-based organizations. Its primary users are administrative church staff who manage giving, events, and member records. In 2025, the team launched a natural language search agent that lets users query their people data using conversational language instead of navigating nearly 200 filters.
The vision was clear: empower every user, not just power users. But once live, it wasn’t clear what was working or failing. “We were using logs. Engineers were showing us JSON,” explained Paul Frank, staff product manager at Pushpay. “We were trying to decode what people were asking. As a PM, that’s tough.”
Before Agent Analytics, the team was relying on patchwork telemetry and gut instinct. After launching Agent Analytics, everything changed.
The first pattern was hard to ignore: a large group of users submitted three or four prompts, then stopped using the feature. Agent Analytics made this drop-off obvious. It helped the team identify the sticking point and refine the experience to get users to value faster.
Within days, they also saw unexpected queries that revealed new needs, and a clear gap between power users (with 50–100+ prompts) and everyone else.
"We had a huge cohort of users that hit that three or four prompt ceiling and then quit. Agent Analytics helped us identify this issue and determine how to improve guidance or refine prompts."
Paul Frank, Staff Product Manager, Pushpay
They used this insight to finetune prompt responses, improve result transparency, and rethink how they guided users through initial queries.
Prompt categorization flipped Pushpay’s early expectations. Initially, the team thought they had a clear idea of what users would ask. Instead, Agent Analytics showed a wide range of queries, with many being valid, but unsupported.
“Every day is a surprise,” said Frank. “We’re seeing our users ask the agent questions we never thought to support, but in many cases, we can. That’s been huge for roadmap prioritization.”
In some cases, users asked questions that were fully supported by the platform, but the filters weren’t easily surfaced. This insight led the team to redesign prompt inferences, improve how filters were labeled, and rethink how results were presented.
Using both emergent and tracked use cases, the team has continuously optimized their agent to align better with how users actually search.
Today, Pendo Agent Analytics is central to Pushpay’s agent lifecycle. The team reviews prompt-level metrics, rage prompts, and replay data daily to understand friction, refine prompt engineering, and validate what success looks like.
They monitor rage prompts daily to catch breakdowns early, track prompt themes to identify trending use cases, and flag off-script behavior that signals where the agent needs tuning.
As Frank puts it, “It’s not about the answer. It’s about the question. That’s what Agent Analytics helps us see—what people are trying to do, and where to go next.”
Pushpay also connected Pendo Guides to thumbs up/down events to capture in-the-moment feedback and educate users proactively.
“Prompt data tells you what they want. Replays show you what they did. Together, they’re insanely powerful,” explained Frank. “You see exactly what the user did, what the output was, and whether it matched their intent. It helps us define success—and fix things that aren’t working.”
With real user voice and behavior now embedded into their decision-making, Pushpay has transformed its search agent from a risky pilot to a repeatable, trusted product pattern, and built a stronger foundation for every future agent.
“It’s almost like going for a walk with your data. Agent Analytics keeps us grounded in what’s actually happening—what users are doing, what they’re asking. That habit of checking in daily is now core to our process.”
Paul Frank, Staff Product Manager, Pushpay
👉 Get started with Agent Analytics for free, or get a demo today.
