The four policy errors any AI tool scoring customer service conversations should catch are a wrong refund amount quoted, incorrect eligibility criteria, a missed required regulatory disclosure, and a wrong escalation path. All four can sound polite and confident, so a generic QA scorecard that grades tone and clarity will often pass them. Revelir AI built RevelirQA, an AutoQA engine that performs QA to the standard of a human quality analyst, calibrated with each customer until the customer's human QA analysts review and validate a sample of its evaluations and judge more than 95 out of 100 correct. Over 95% accuracy is achieved per customer through this calibration process. Scoring every conversation also matters, but most auto QA tools already claim 100% coverage as table stakes. Human-level judgment is what catches these four errors.
TL;DR
- RevelirQA performs QA to the standard of a human quality analyst, calibrated per customer to over 95% accuracy, meaning the customer's own QA analysts judge more than 95 of every 100 sampled evaluations correct. Full coverage of 100% of conversations is table stakes most auto QA tools already claim; judgment is the differentiator.
- The four errors that matter most: wrong refund amount, incorrect eligibility criteria, missed regulatory disclosure and wrong escalation path.
- Generic scorecards grade how an agent spoke. Catching policy errors requires checking what they said against the actual policy document.
- Rare errors slip through small manual samples, so full coverage is necessary. It is still table stakes, not the differentiator.
- Every flagged error should show its evidence: the quoted line, the SOP passage and reasoning a QA lead can override.
About the Author: This article comes from the team at Revelir AI, maker of RevelirQA, an AI quality assurance platform that Xendit and Tiket.com run in production on thousands of tickets per week. It scores human and AI chatbot conversations against each company's own policies.
What is a policy error, and why do generic QA scorecards miss it?
The central problem with these four errors is that they are failures of substance, not style. A policy error is a response that contradicts a company's documented rules about what customers are owed, who qualifies, what must be disclosed or where a case must go.
Generic scorecards ask whether the agent was empathetic and clear. Those questions are worth asking, but a reply can score full marks on both while quoting the wrong refund.
Think of a generic scorecard as a spellchecker. It confirms every word is real and the grammar holds, so "Your refund of $50 will arrive in three days" passes whether the correct amount is $50 or $35. Catching the mistake needs a fact-checker that opens the source document and compares. That is the mechanism behind retrieval-augmented generation (RAG) in automated quality assurance. Before scoring, the system retrieves the policy for the ticket's contact reason. It then evaluates the meaning of what the agent said against that policy instead of matching keywords.
What are the four policy errors an AI QA tool should catch?
Building on that distinction, four error types carry most of the risk that tone-focused scoring leaves behind.
| Policy error | What it looks like | What the AI must check against | Why it matters |
|---|---|---|---|
| Wrong refund amount quoted | Promising a full refund where policy allows a partial one | Refund and compensation policy | Financial loss, or a broken promise the customer escalates later |
| Incorrect eligibility criteria | Telling a customer they qualify for a waiver or rebooking they do not | Eligibility rules by product or booking type | Inconsistent treatment and an unwanted precedent |
| Missed required regulatory disclosure | Skipping a mandatory statement before a regulated action | Compliance documentation | Regulatory exposure that is hard to prove after the fact |
| Wrong escalation path | Closing a case at first line that belonged to a specialist team | Escalation procedures | CSAT risk and compliance exposure |
Each one hides in a different way:
- Refund errors look like good service. The customer is happy, so CSAT rises while margin leaks.
- Eligibility errors depend on context outside the transcript, such as the customer's plan. The scorer must retrieve the right rule for that case.
- Disclosure errors are omissions. Nothing in the transcript looks wrong. The scorer has to know what should have been said.
- Escalation errors often end in a "resolved" ticket. The mistake is the route, not the outcome.
AI chatbots make the same four errors, with one difference: a chatbot repeats a wrong policy at scale, on every conversation where that policy comes up. Scoring chatbot conversations against the same policies as human ones is part of that.
Why does manual QA sampling rarely surface these errors?
The traits that make these errors hard to spot in one transcript also make them hard to find statistically. Manual QA sampling most commonly misses low-frequency, high-impact errors such as skipped mandatory disclosures.
Scoring every conversation fixes the sampling problem. It does not fix the judgment problem. An auto QA tool that scores 100% of tickets with a spellchecker's understanding of your policies simply passes more confident mistakes. Coverage tells you the AI looked. Accuracy tells you it understood what it saw.
How can you tell whether an AI QA tool judged a policy error correctly?
With coverage now standard, the harder question for any buyer is whether the AI's judgment holds up under review. Use these tests:
- Does it cite the source? Each judgment should name the SOP passage and version it measured against.
- Does it quote the evidence? The flagged agent line should appear next to the failed metric.
- Can a human overrule it? QA leads should be able to override grades, and agents should be able to contest them.
- Was accuracy measured on your data? A vendor benchmark tells you little. Accuracy should be validated by your own analysts on your own tickets.
RevelirQA records the model used, the exact prompt, the documents retrieved and the reasoning behind each criterion on every evaluation, not just flagged ones. At Xendit, which operates under regulated requirements in Indonesian fintech, this level of auditability is a compliance requirement. Rendy D. of tiket.com put the practical result this way: "We've manually reviewed tickets for years. Revelir is the first product that has made AI ticket review at scale actually usable."
How do you build a QA scorecard that catches all four errors?
Putting this into practice starts with the QA scorecard, because an AI can only flag what the scorecard defines as a failure.
- Separate critical errors from soft skills. Make each of the four errors its own pass/fail metric so a warm tone cannot average it away.
- Map each metric to one source document. Refund policy, eligibility rules, compliance documentation and escalation procedures should each be clearly owned and versioned.
- Weight by contact reason. A disclosure check matters on payment tickets and is noise on password resets.
- Calibrate against tickets your analysts already graded. Agree up front on what "correct" means before trusting any accuracy number.
- Read clusters, not single failures. If many agents quote the same wrong refund, the SOP is probably unclear. The fix may be a knowledge base update, a chatbot prompt change or a new escalation rule.
Frequently Asked Questions
What is policy-aware AutoQA?
It is automated quality assurance that retrieves a company's own policies before scoring each conversation, then checks what the agent said against them instead of grading against generic criteria.
Can AI detect a disclosure that was never said?
Yes, if the scorecard defines the disclosure as required for that contact reason. The AI checks the conversation against the compliance document and flags the omission.
Do the same checks apply to AI chatbots?
They should. RevelirQA applies the same QA scorecard to human and chatbot conversations, for any chatbot platform that produces a conversation log via API.
What does "over 95% accuracy" mean for RevelirQA?
Revelir calibrates the AI with each customer until the customer's own QA analysts, reviewing a sample of evaluations, judge more than 95 out of 100 correct. It is reached per customer, not out of the box.
Does a flagged error trigger automatic discipline?
Not with RevelirQA. It takes no automatic action. Results go to the QA lead, who decides what happens next.
How can a team test this before committing?
Revelir runs a six-week paid pilot that grades an agreed subset of 10,000 conversations against accuracy targets set with your QA leads in week one. The $10,000 pilot fee is credited in full against the deployment fee if you proceed.
About Revelir AI
Revelir AI is a Singapore-headquartered company building AI customer service QA software for global enterprise teams. Its scoring engine, RevelirQA, grades human and AI chatbot conversations against each customer's own SOPs at the standard of a human quality analyst, with a full audit trail behind every score. It connects to any helpdesk via API, including Zendesk and Salesforce, and runs as SaaS or in a dedicated tenant. Xendit and Tiket.com use it in production in English and Indonesian, and every ticket is also enriched with sentiment, contact reason and recurring issue type to show where product and operations problems repeat.
If your team needs to know whether agents and chatbots are quoting the right refunds, applying the right eligibility rules, reading the required disclosures and routing cases correctly, talk to Revelir AI at https://www.revelir.ai/.
