RevelirQA, Revelir AI's AutoQA engine, performs QA to the standard of a human quality analyst, calibrated with each customer to over 95% accuracy as validated by that customer's own QA analysts on a sample of evaluations. Scoring 100% of conversations is table stakes that most auto QA tools already claim; the real differentiator is human-level judgment. To validate sentiment and contact reason labels yourself, follow three steps:

• Have your analysts blind-label a stratified sample of conversations.
• Compare their labels with the tool's label by label, not as one overall score.
• Fix any category your analysts disagree on among themselves before you blame the model.

The real test is whether the labels hold up when your own analysts review them, and whether the AI's judgment matches the standard of a human quality analyst on your own taxonomy.

TL;DR

• RevelirQA performs QA to the standard of a human quality analyst. Revelir calibrates it with each customer to over 95% accuracy, meaning that customer's own QA analysts judge more than 95 of every 100 sampled evaluations correct. Coverage of 100% of conversations is table stakes that most AutoQA tools already claim.
• Validate labels with a blind, stratified sample that your analysts label. A vendor benchmark cannot do this job.
• Report accuracy per label and per stage of the conversation. An overall rate hides weak categories.
• If your analysts disagree with each other on a label, rewrite the definition first.
• Re-validate whenever your taxonomy, channels, languages or policies change.

About the Author: Revelir AI builds RevelirQA, an AI quality assurance platform headquartered in Singapore. Xendit and Tiket.com run RevelirQA in production on thousands of tickets a week. Its ticket enrichment adds sentiment and contact reason to every ticket, and it is calibrated with each customer until that customer's QA analysts judge more than 95 out of 100 sampled evaluations correct.

Why do sentiment and contact reason labels need human validation?

A label is only useful if it matches how your own analysts would classify the same conversation, and no vendor benchmark can tell you that. Sentiment analysis captures how customers feel across calls, chats, emails and social messages [roicallcentersolutions.com][callrail.com]. Contact reason tells you why they got in touch. Both drive staffing, product fixes and escalation decisions, so a mislabeled trend sends the wrong team after the wrong problem.

• Benchmarks test someone else's taxonomy. Published sentiment accuracy figures are useful context, but the boundary between your "billing dispute" and "refund request" labels was never in that test.
• Ad hoc prompting is not a metric. Asking a general chatbot about sentiment works for exploration. There is a clear point where you outgrow it for a real metric [enterpret.com].
• Nobody spot-checks labels by accident. Manual QA typically reviews only a small share of conversations. Labels on the rest go unchecked unless you sample them on purpose.

How should you build the validation sample?

The goal is to compare the tool against your analysts, so the sample must keep analyst judgment independent and give every label enough examples to measure.

1. Freeze definitions. Give each label a one-sentence definition plus one tricky example.
2. Stratify. Pull conversations per label, per channel and per language, not one random batch.
3. Include the edges. Add tickets labeled "Other," multi-issue tickets and conversations in code-switched language.
4. Label blind. Analysts must not see the tool's label. Once a reviewer sees a suggested answer, they tend to confirm it.
5. Double-label a subset. Have two analysts label the same tickets so you can measure human agreement.
6. Adjudicate. A QA lead resolves each disagreement and records the reason.

Here is why stratification matters. A random sample mirrors your volume, so rare labels get only a handful of examples. Think of a bathroom scale that reads correctly at 70 kg but 5 kg light at 100 kg. If you only test it on average adults, it looks accurate. Your biggest contact reasons are the average adults. The small, costly ones, like fraud reports or regulatory complaints, are where the scale reads wrong.

How do you measure whether the labels are correct?

Once you have the sample, the measurement that matters is agreement per label, not one headline percentage.

QA metric	Question it answers	Warning sign
Analyst-to-analyst agreement	Can humans apply this label consistently?	Your two analysts often split on the same ticket
Per-label precision	When the tool says "refund request," how often is it right?	Small labels score low while the overall rate looks fine
Per-label recall	Of the tickets analysts call "refund request," how many did the tool catch?	Tickets leak into "Other"
Confusion pairs	Which labels get swapped with each other?	The same two labels keep trading tickets
Sentiment at start vs. end	Does the tool capture how the customer's mood changed?	Angry openings get labeled positive because the chat ended with "thanks"

Sample size per label sets how much you can trust each number. With 20 tickets in a label, a single disagreement moves that label's accuracy by five percentage points.

Why do labels usually fail?

When the numbers come back, the next job is diagnosis. Most label errors trace to a few causes, and several of them are taxonomy problems rather than model problems.

Failure	Likely cause	Fix
Polite complaints labeled positive	Sentiment read from surface cues instead of meaning	Add polite-but-unhappy examples to the definition
Multi-issue tickets misclassified	Taxonomy allows only one reason	Use a primary and a secondary reason
"Other" keeps growing	Taxonomy lags behind new products or issues	Review "Other" regularly and promote recurring themes to labels
Weak results in one language	Sample and calibration dominated by one language	Validate each language separately

How often should you re-validate, and where should disagreements go?

Treat validation as a recurring control that runs on a schedule and on triggers. Labels drift as your business changes, even when the model does not. Re-validate when:

• you add or rename contact reasons
• you launch a new channel, language or product
• you change policies or SOPs
• your vendor changes the underlying model

Disagreements only improve accuracy if they flow back to whoever tunes the system. In RevelirQA, every evaluation shows the reasoning, the quote it came from and the document it checked against. QA leads can override a grade, agents can contest scores, and disagreements feed back into calibration. Revelir's paid pilot starts the same way. Accuracy definitions and targets are agreed with the customer's QA leads in week one. The AI then grades an agreed subset of 10,000 conversations over six weeks, and the $10,000 pilot fee is credited in full against the deployment fee if the company proceeds.

Frequently Asked Questions

Is overall accuracy enough to trust a tool?
No. High-volume labels dominate the overall rate. Always check precision and recall per label.

Should analysts see the AI's label while reviewing?
No. Seeing the label pulls reviewers toward agreeing with it, which inflates accuracy.

Can we rely on the vendor's published accuracy?
Treat it as a starting point. Published figures are measured on other companies' taxonomies, not yours.

What if our own analysts disagree?
Rewrite the label definition and re-label before you judge the tool. A label humans cannot apply consistently cannot be measured.

Do we need to document validation results?
Yes, especially in regulated industries. Documented results show auditors and stakeholders exactly how labels and scores were checked.

What does "over 95% accuracy" mean for RevelirQA?
Revelir calibrates the AI with each customer until that customer's QA analysts judge more than 95 out of 100 sampled evaluations correct. It is reached per customer, not out of the box.

About Revelir AI

Revelir AI builds RevelirQA, an AI quality assurance platform that performs automated quality assurance to the standard of a human quality analyst. Revelir calibrates it with each customer to over 95% accuracy, as validated by that customer's own QA analysts on a sample of evaluations. RevelirQA scores 100% of conversations against each customer's own SOPs and QA scorecard, and every score carries an auditable reasoning trace. It also enriches every ticket with sentiment, contact reason and recurring issue type, and it runs in production at Xendit and Tiket.com and is built for enterprise teams worldwide.

Want to see how your analysts' judgment compares with an AI calibrated to your own definitions? Learn more or talk to the team at https://www.revelir.ai/.