AI quality
How to read the AI quality dashboard: what each figure means, what it is compared against, and how to open the conversations behind it.
The AI quality page answers one question: is your AI agent actually helping your customers, and is it getting better or worse? Every figure on it is shown next to the same figure for the period before, because a number on its own cannot be acted on — an 18% resolve rate is a crisis if it was 40% last month and a success if it was 6%.
Find it under AI & Automation → AI quality.
The four headline figures#
Resolved by AI
The share of finished conversations the AI closed on its own, with no member of staff writing anything. This is the number the product is built to move.
Satisfaction
The average rating customers gave after an AI-resolved conversation, with the number of ratings it is based on. Treat a high score from three ratings with suspicion.
Asked for a person
How often customers asked to be put through to a human — whether they tapped the button or simply typed "operator". This measures demand for your team, not a failure of the knowledge base.
Left waiting
Customers who asked for a person and never got one within a working day. This is the figure worth acting on first: those conversations were neither answered by the AI nor picked up by anyone.
Select any figure to open the conversations behind it. "Left waiting: 171" becomes a list of 171 customers you can still go and reply to.
How conversations are counted#
Every conversation that reaches a settled state is placed in exactly one of five buckets.
| Outcome | What happened |
|---|---|
| Resolved by AI | The AI answered and the chat closed with no human involved. |
| Escalated to ticket | A support ticket was opened. Not a failure — it is the agent's designed second option — but the work moved to a queue. |
| Handled by an operator | Somebody on your team wrote in the chat, whether they were asked for or simply took over. |
| Left waiting for an operator | The customer asked for a person and nobody ever replied. |
| Abandoned | The customer stopped replying without resolving or escalating. |
"Asked for a person" is deliberately not one of the buckets. A customer can ask for a human and still be helped by the AI, and an operator can step in without being asked. Keeping the two separate means neither figure quietly distorts the other.
Abandoned conversations are reported on their own rather than counted as failures. Some of them are satisfied customers who simply left, and there is no way to tell which.
Why the channel breakdown matters most#
A single overall resolve rate almost always hides two different stories. One tenant's page showed a healthy-looking 22% overall — while one widget resolved 48% of its conversations and its replacement resolved 0.3%. Nothing in the headline number said so.
Check this table whenever the overall figure moves without an obvious cause. Deleted channels keep their names here, so months of history stay readable.
Safety guards#
The last card counts replies that were caught and rewritten before a customer ever saw them, broken down by what was wrong:
- Promised to follow up — the AI said it would check and come back, which it cannot do unless a background lookup was genuinely scheduled.
- Claimed to transfer — the AI told the customer it was passing them to someone. It has no such ability.
- Leaked internal narration — the AI's working-out escaped into the reply.
- Returned nothing — the model produced an empty reply.
A rising count here is not a sign the guards are failing; it is a sign they are working. It does mean the underlying model or prompt is drifting, and a sustained rise is worth investigating.
When the numbers appear#
Conversations that end on a button press are counted immediately. The ones that simply stop — the majority — are settled by an hourly sweep, and a conversation is only called abandoned after two days of silence, or left waiting after a full working day. Today's figures are therefore always slightly incomplete; yesterday's are not.