When an AutoQA vendor says their AI hits "95% accuracy," the number is meaningless unless you know what it was measured against. RevelirQA, the AI quality assurance platform used in production by Xendit and Tiket.com, is built around a specific definition: it performs automated quality assurance to the standard of a human quality analyst, calibrated per customer until that customer's own QA team reviews a sample of RevelirQA's scores and judges more than 95 out of 100 correct. The figure that matters is agreement with a customer's own human quality analysts, validated on a real sample of that customer's own conversations, after the AI has been calibrated to that customer's specific QA scorecard and policies. Coverage of every ticket is table stakes in auto QA today. The harder, more important claim is accuracy that a company's own analysts have verified, not a number borrowed from a vendor's marketing page.

TL;DR

• RevelirQA performs QA to the standard of a human quality analyst, calibrated with each customer until the customer's own human QA analysts review and validate a sample of the AI's evaluations and judge more than 95 out of 100 correct. 100% conversation coverage is now standard among AutoQA vendors and solves a real problem (manual QA only reviews 1-5% of tickets [UNCITED]), but it is not, by itself, evidence of quality.
• Accuracy is only meaningful relative to a reference standard. Without a clearly defined reference, like a specific scorecard and a specific pool of human reviewers, an accuracy number cannot be verified or trusted [crazygoldfish.com].
• Calibration, not configuration, is what produces real accuracy: the AI has to learn a company's actual SOPs and QA scorecard, not score against generic best practices.
• RevelirQA calibrates per customer until human QA analysts validate more than 95% agreement on sampled evaluations, and backs every score with a full reasoning trace for audit purposes.

About the Author: This article is written by the team at Revelir AI, which built RevelirQA, an AutoQA scoring engine running in production on thousands of customer service conversations per week at companies including Xendit and Tiket.com. Revelir's work calibrating AI QA scoring against human analyst judgment, across English, Indonesian-language, Thai, and Tagalog support environments, directly informs the accuracy standard described in this piece.

What Does "Accuracy" Actually Mean in a QA Context?

Accuracy only has meaning relative to a reference standard. In customer service QA, that reference is almost always a human reviewer scoring the same conversation against the same scorecard criteria. The industry generally defines and measures this as "agreement": how often an AI's automated score matches the score a human reviewer would independently assign on identical criteria. Vendors track this agreement rate to check that automation stays reliable against the manual benchmark it is meant to replace or extend.

This matters because accuracy without a defensible reference is just a number. If a vendor reports "95% accuracy" without specifying what the AI was being compared against, whose scorecard was used, or who did the reviewing, the figure cannot be verified or replicated [crazygoldfish.com]. A model can also be well-calibrated in a statistical sense (correct 99% of the time it claims 99% confidence) and still be evaluating against the wrong standard entirely [hunterheidenreich.com]. Calibration confidence and real-world correctness are two different things, and a credible accuracy claim has to address both.

Why Is 100% Coverage Not Enough on Its Own?

Coverage answers a different question than accuracy does, and conflating the two is where a lot of AutoQA marketing goes wrong. Coverage asks: how many conversations get reviewed? Accuracy asks: are the reviews correct? Manual QA sampling typically covers only 1 to 5 percent of total customer service conversations, leaving the vast majority of interactions unreviewed [UNCITED]. That is a real and well-documented limitation, and it is the reason AutoQA exists at all.

But reviewing 100% of tickets with a poorly calibrated model just means a company now has a complete record of wrong answers instead of a partial one. Nearly every AutoQA vendor today claims full-conversation coverage. It has become the baseline expectation, not the differentiator. The more useful question for a buyer to ask a vendor is not "do you score every ticket," but "how do you know your scores are right, and who checked?"

Question	What it tells you	Why it's often missing
What percentage of tickets get scored?	Coverage	Usually answered, now table stakes
What is the AI compared against?	Reference standard	Often vague or generic
Who validated the agreement rate?	Trustworthiness of the number	Frequently omitted
Was the AI calibrated to this company's QA scorecard?	Relevance of the accuracy figure	Rarely disclosed

How Should an Accuracy Benchmark Actually Be Calculated?

Building on the coverage-versus-accuracy distinction above, the calculation method itself determines whether a benchmark is trustworthy. A defensible AutoQA accuracy benchmark requires three things: a specific scorecard, a specific sample of conversations, and a specific group of human reviewers whose judgments serve as ground truth. The AI's scores are then compared criterion by criterion against those human judgments, and the agreement rate is the proportion that match.

Sample size matters more than most teams assume. A validation run of 20 tickets and a validation run of 95 tickets can both report a 95% agreement rate, but the larger sample gives a far tighter confidence interval around that number [dev.to]. A vendor who validates against a handful of cherry-picked tickets is not making the same claim as one who validates against a statistically meaningful, customer-reviewed sample. This is also why accuracy should never be treated as a fixed, universal number a vendor ships out of the box. It is the output of a calibration process run against a specific company's own scorecard and own reviewers, and it should be re-validated periodically as that company's policies evolve.

It is also worth separating subjective criteria from binary ones. Whether an agent "closed with an appropriate tone" is inherently more contestable between two human reviewers than whether an agent "verified the customer's account ID." Accuracy rates generally vary with how subjective the underlying criteria are, and transcription quality affects reliability further, since a model can only score what it can correctly read [UNCITED]. A credible vendor should be transparent about which scorecard criteria are harder to agree on, rather than quoting one blended number across everything.

What Does Calibration Look Like in Practice?

A related but distinct question from the measurement question above is how a vendor actually gets an AI to that accuracy level in the first place. Calibration is the process of teaching the AI a specific company's policies, scorecard, and edge cases until its judgment matches that company's own human analysts closely enough to trust at scale.

In RevelirQA's case, this works by ingesting the customer's own SOPs and knowledge base into a vector database via retrieval-augmented generation, so the model retrieves the actual, current policy before scoring each conversation, rather than scoring against generic customer service best practices. The same scorecard is then applied consistently across every agent and every ticket, human or AI-handled. Revelir works with each customer's QA team through a calibration cycle: the AI scores a batch, human analysts review a sample of those scores, disagreements get diagnosed (was it the policy retrieval, the scorecard wording, or a genuine model error?), and the system is adjusted. This repeats until more than 95 of 100 sampled evaluations are judged correct by the customer's own analysts. That number is never claimed as a one-size-fits-all benchmark; it is earned per customer, on that customer's own policies.

An analogy helps explain why this mechanism matters. Training a new human QA analyst works the same way: you do not hand them a generic customer service textbook and call them fully ready. You walk them through your company's actual policies, have them score real tickets, check their work against a senior reviewer, and correct their misunderstandings until their judgment reliably matches the team's standard. Calibrating an AI model is the same process, just applied to a system that can then apply that calibrated judgment to every single conversation instead of a sample.

Why Does Auditability Matter as Much as the Accuracy Number?

Building on calibration, the next question a QA or compliance lead should ask is not just "is the score accurate" but "can I see why the AI gave that score." An accuracy percentage without a reasoning trace is a number you have to take on faith. Requirements for AI audit trails and documented reasoning are now embedded in major compliance frameworks, including SOC 2, ISO 27001, and ISO 42001, with further pressure from the EU AI Act and the NIST AI Risk Management Framework [UNCITED].

This is particularly relevant in regulated industries like fintech, where Xendit operates, and in high-volume consumer platforms like Tiket.com, where thousands of tickets per week need to be scored consistently and defensibly. RevelirQA performs quality assurance to the standard of a human quality analyst by attaching a full reasoning trace to every score: the model used, the specific policy documents retrieved, and the step-by-step reasoning behind the result. That trace is what turns an accuracy claim into something a QA manager, a compliance officer, or an auditor can actually verify, rather than a black-box percentage a team is asked to trust.

Frequently Asked Questions

What should "95% accuracy" mean when an AutoQA vendor states it?
It should mean that, when the customer's own human QA analysts reviewed a sample of the AI's evaluations against that customer's QA scorecard, more than 95 out of 100 were judged correct. It should never be presented as a universal, out-of-the-box number that applies before any customer-specific calibration.

Is 100% conversation coverage the same thing as high accuracy?
No. Coverage measures how many conversations get reviewed; accuracy measures whether those reviews are correct. Most AutoQA vendors now offer full coverage, so it is no longer a meaningful differentiator on its own.

How much of customer service QA do manual reviewers typically cover?
Manual QA sampling generally reviews only 1 to 5 percent of total conversations, which leaves the large majority of interactions unreviewed under a manual-only process [UNCITED].

Why does sample size matter when a vendor reports an agreement rate?
A larger validation sample gives a tighter, more reliable confidence interval around the reported rate. A 95% agreement rate from 20 tickets is statistically weaker evidence than the same rate from a much larger, representative sample [dev.to].

Does AutoQA accuracy stay the same over time?
It shouldn't be assumed to. As a company's policies, products, and scorecards change, the AI needs re-calibration and re-validation to keep agreement with human reviewers at the same level.

What is a reasoning trace and why does it matter for compliance?
A reasoning trace shows the model used, the policy documents retrieved, and the logic behind a given score. It matters because major compliance frameworks, including SOC 2, ISO 27001, and ISO 42001, increasingly expect AI-driven decisions to be auditable rather than opaque [UNCITED].

Can the same AI model evaluate both human agents and AI chatbots?
Yes, in principle, provided it is calibrated against a single consistent QA scorecard. As companies deploy chatbots alongside human reps, scoring both against the same scorecard is what gives CX leaders one coherent view of quality across their entire support operation, rather than two disconnected ones.

About Revelir AI

Revelir AI builds RevelirQA, an AI quality assurance platform that performs automated quality assurance to the standard of a human quality analyst, calibrated with each customer until the customer's own human QA analysts review and validate a sample of the AI's evaluations and judge more than 95 out of 100 correct, across 100% of support conversations. Founded in Singapore in 2025 by Rasmus Chow, Revelir works in production, not pilot programs, with enterprise customers including Xendit and Tiket.com scoring thousands of tickets per week across English, Indonesian-language, Thai, and Tagalog support environments. Every RevelirQA score is generated against the customer's own SOPs, retrieved via RAG, and comes with a full reasoning trace covering the model, the documents retrieved, and the logic behind the result.

If your QA team is still relying on manual sampling, or using an AutoQA tool whose accuracy claim you've never been able to verify against your own analysts, it's worth seeing what calibrated, auditable auto QA looks like in practice. Get in touch with Revelir AI to see how RevelirQA calibrates to your team's own QA scorecard.